DEV Community

Viktor
Viktor

Posted on

How to Scrape Twitch with Go: Chat Logs over IRC WebSockets, Moderation Events, Full-Text Search, and Chat Stats (2026)

People search "twitch chat logs" about 6,600 times a month in the US, and most of them hit the same wall: Twitch has no API that returns chat from the past. Helix doesn't have one, and neither does EventSub. If a message isn't captured while it scrolls by, it's gone for you. So a chat log is something you build going forward: connect, listen, store.

This post builds twitchlog, a small Go program that does exactly that:

  1. log connects to Twitch chat over its IRC WebSocket, joins channels, and writes every message to SQLite, along with subs, raids, timeouts, bans, and room settings
  2. search runs full-text queries (FTS5) across everything you've logged
  3. export prints a plain-text log for one channel and day, or for one chatter
  4. stats summarizes a channel: messages per minute, unique chatters, subscriber share, moderation, top Twitch emotes, events
  5. import rebuilds the database from the raw IRC log, so the raw log is your source of truth
  6. live (optional) checks which channels are live before you join them

It's five Go files and two dependencies: github.com/coder/websocket for the WebSocket and modernc.org/sqlite, a pure-Go SQLite (no cgo). Everything below ran on November 20, 2026 with Go 1.27.3 on a Linux VM. I logged #xqc and #kaicenat for about 24 minutes in total (12,443 messages). Output blocks are pasted from those runs, except that I replaced chatters' usernames with viewer_NN placeholders.

Read this first. Twitch's terms prohibit scraping. The Twitch Terms of Service (last modified 08/12/2026) say you may not "access any website, server, software application, or other computer resource owned, used, and/or licensed by Twitch, including but not limited to the Twitch Services, by means of any robot, spider, scraper, crawler, or other automated means for any purpose", and the license section excludes "use of any data mining, robots, or similar data gathering or extraction methods". Read those clauses and think about the legal risk for your use case before you run anything here. Prefer the official routes: Twitch documents IRC and EventSub as its chat interfaces, and for new projects it recommends EventSub (channel.chat.message) with a user access token that has the user:read:chat scope. The anonymous justinfan login used below is not documented. Chat is also personal data: usernames next to what people said. Keep logs only as long as you need them, don't publish logs of individual people, and check privacy law (GDPR and similar) if you store them. The code redacts messages that moderators delete or that belong to timed-out and banned users, because Twitch removes them from view too.

What you can and can't get

What How Notes
Live chat, from now on IRC over WebSocket (wss://irc-ws.chat.twitch.tv:443) or EventSub This post uses IRC; EventSub needs a user token
Chat from before you connected No API Helix and EventSub only deliver new messages
Chat replay on VODs Only through the website's private GraphQL Not covered here
Who is in chat Helix Get Chatters, needs a moderator token Anonymous IRC only lists you in NAMES
Timeouts, bans, deleted messages CLEARCHAT and CLEARMSG events You only see them if you're connected at the time
7TV, BTTV, FFZ emotes Not in the IRC tags They arrive as plain words

Before writing a parser, I tested which transport the VM could reach. The raw IRC ports didn't work from that network: TLS on port 6697 ended with unexpected eof, and a plain connection to 6667 got HTTP/1.1 400 Bad Request from an HTTP proxy. The WebSocket upgrade on port 443 worked first try:

HTTP/1.1 101 Switching Protocols
Connection: upgrade
Upgrade: websocket
Enter fullscreen mode Exit fullscreen mode

That's the main reason to use WebSockets: it's ordinary HTTPS traffic, so it gets through proxies and firewalls that block IRC ports. Twitch shut down the unencrypted ws:// endpoint on port 80 in August 2025, so use wss://.

Setup

mkdir twitchlog && cd twitchlog
go mod init twitchlog
go get github.com/coder/[email protected] modernc.org/[email protected]
Enter fullscreen mode Exit fullscreen mode

The program is split into irc.go (parsing), conn.go (the connection), store.go (SQLite), main.go (commands), and stats.go (the report and the live check). All five are below in full, so you can copy them into one folder and go build.

Step 1: Parse Twitch's IRC lines

Twitch sends IRC with IRCv3 tags in front. A chat message looks like this (shortened, username replaced):

@badge-info=;badges=bronze-chain/1;color=#8A2BE2;display-name=viewer_01;emotes=;first-msg=0;id=00000000-0000-4000-8000-000000000001;mod=0;returning-chatter=0;room-id=71092938;subscriber=0;tmi-sent-ts=1795147987206;user-id=100000001 :viewer_01!viewer_01@viewer_01.tmi.twitch.tv PRIVMSG #xqc :SCHIZO
Enter fullscreen mode Exit fullscreen mode

The tags carry everything useful: a stable message id, the numeric user-id (logins can change, IDs can't), badge-info with subscription months, emotes with character positions, bits, first-msg, reply-parent-msg-id, and the server timestamp tmi-sent-ts in milliseconds. Tag values are escaped (\s is a space, \: is a semicolon), so unescape them. System messages are full of escapes: a raid arrives as system-msg=1\sraiders\sfrom\s...\shave\sjoined!.

irc.go:

package main

import (
    "strings"
)

// Message is one parsed IRC line from Twitch, IRCv3 tags included.
type Message struct {
    Raw     string
    Tags    map[string]string
    Prefix  string // "nick!user@host" or "tmi.twitch.tv"
    Command string // PRIVMSG, USERNOTICE, CLEARMSG, PING, 001, ...
    Params  []string
}

// Nick returns the login part of the prefix ("" for server messages).
func (m Message) Nick() string {
    if i := strings.IndexByte(m.Prefix, '!'); i > 0 {
        return m.Prefix[:i]
    }
    return ""
}

// Channel returns the first param without '#', if it is a channel.
func (m Message) Channel() string {
    if len(m.Params) > 0 && strings.HasPrefix(m.Params[0], "#") {
        return m.Params[0][1:]
    }
    return ""
}

// Trailing returns the last param (the chat text for PRIVMSG).
func (m Message) Trailing() string {
    if len(m.Params) == 0 {
        return ""
    }
    return m.Params[len(m.Params)-1]
}

// Parse turns one line into a Message.
// Format: [@tags ][:prefix ]COMMAND [params] [:trailing]
func Parse(line string) Message {
    m := Message{Raw: line, Tags: map[string]string{}}
    rest := line
    if strings.HasPrefix(rest, "@") {
        end := strings.IndexByte(rest, ' ')
        if end < 0 {
            return m
        }
        for _, kv := range strings.Split(rest[1:end], ";") {
            k, v, _ := strings.Cut(kv, "=")
            m.Tags[k] = unescapeTag(v)
        }
        rest = rest[end+1:]
    }
    if strings.HasPrefix(rest, ":") {
        end := strings.IndexByte(rest, ' ')
        if end < 0 {
            return m
        }
        m.Prefix = rest[1:end]
        rest = rest[end+1:]
    }
    for rest != "" {
        if strings.HasPrefix(rest, ":") {
            m.Params = append(m.Params, rest[1:])
            break
        }
        word, after, found := strings.Cut(rest, " ")
        if m.Command == "" {
            m.Command = word
        } else {
            m.Params = append(m.Params, word)
        }
        if !found {
            break
        }
        rest = after
    }
    return m
}

// IRCv3 tag values escape ';' ' ' '\' CR and LF.
func unescapeTag(v string) string {
    if !strings.Contains(v, `\`) {
        return v
    }
    var b strings.Builder
    for i := 0; i < len(v); i++ {
        if v[i] != '\\' || i == len(v)-1 {
            if v[i] != '\\' {
                b.WriteByte(v[i])
            }
            continue
        }
        i++
        switch v[i] {
        case ':':
            b.WriteByte(';')
        case 's':
            b.WriteByte(' ')
        case 'r':
            b.WriteByte('\r')
        case 'n':
            b.WriteByte('\n')
        default: // "\\" and unknown escapes
            b.WriteByte(v[i])
        }
    }
    return b.String()
}
Enter fullscreen mode Exit fullscreen mode

Step 2: The connection

The connection has more rules than the parser:

  • Login. Request the twitch.tv/tags and twitch.tv/commands capabilities, then PASS and NICK. For read-only access, any justinfan plus digits nick works with any password. With an OAuth token you'd send PASS oauth:<token> and your own login instead.
  • One frame, several lines. Twitch's docs warn that a single WebSocket message can hold several IRC lines separated by \r\n. Split every frame, or you'll parse a batch as one message and drop the rest.
  • PING/PONG. Twitch sends PING :tmi.twitch.tv about every five minutes and drops you if you don't answer. In my run they arrived at 11:28:37 and 11:33:36.
  • RECONNECT. Twitch sends this before restarting a server. Reconnect right away.
  • Join limits. Twitch's docs say 20 join attempts per 10 seconds for a normal account, and 100 channels joined at the same time. The code waits 11 seconds after every 15 joins.
  • Silent failures. Joining a channel that doesn't exist returns nothing at all: no error, no NOTICE. I joined thisuserdoesnotexist_zz9 along with the two real channels and only got JOIN confirmations for the real ones. Check for the JOIN echo.

conn.go:

package main

import (
    "context"
    "errors"
    "fmt"
    "log"
    "math/rand/v2"
    "strings"
    "time"

    "github.com/coder/websocket"
)

var ircURL = "wss://irc-ws.chat.twitch.tv:443" // var so tests can point it elsewhere

// errReconnect means Twitch asked us to move to another server.
var errReconnect = errors.New("server sent RECONNECT")

// session runs one WebSocket connection until it drops or ctx ends.
// Every parsed line goes to out.
func session(ctx context.Context, channels []string, out chan<- Message) error {
    dialCtx, cancel := context.WithTimeout(ctx, 15*time.Second)
    defer cancel()
    c, _, err := websocket.Dial(dialCtx, ircURL, nil)
    if err != nil {
        return fmt.Errorf("dial: %w", err)
    }
    defer c.CloseNow()
    c.SetReadLimit(1 << 20) // a burst of long messages can exceed the 32 KB default

    send := func(line string) error {
        wctx, cancel := context.WithTimeout(ctx, 10*time.Second)
        defer cancel()
        return c.Write(wctx, websocket.MessageText, []byte(line+"\r\n"))
    }

    // Anonymous, read-only login: any justinfan<digits> nick, any PASS.
    nick := fmt.Sprintf("justinfan%d", 10000+rand.IntN(80000))
    for _, l := range []string{
        "CAP REQ :twitch.tv/tags twitch.tv/commands",
        "PASS SCHMOOPIIE",
        "NICK " + nick,
    } {
        if err := send(l); err != nil {
            return err
        }
    }

    // JOINs are limited (20 per 10 s for normal users), so pace them.
    go func() {
        for i, ch := range channels {
            if i > 0 && i%15 == 0 {
                select {
                case <-time.After(11 * time.Second):
                case <-ctx.Done():
                    return
                }
            }
            if err := send("JOIN #" + strings.ToLower(ch)); err != nil {
                return
            }
        }
    }()

    for {
        // Twitch PINGs about every 5 minutes; 6 minutes of silence = dead link.
        rctx, cancel := context.WithTimeout(ctx, 6*time.Minute)
        _, data, err := c.Read(rctx)
        cancel()
        if err != nil {
            return fmt.Errorf("read: %w", err)
        }
        // One WebSocket frame can carry several IRC lines.
        for _, line := range strings.Split(strings.TrimRight(string(data), "\r\n"), "\r\n") {
            if line == "" {
                continue
            }
            m := Parse(line)
            switch m.Command {
            case "PING":
                if err := send("PONG :" + m.Trailing()); err != nil {
                    return err
                }
            case "RECONNECT":
                return errReconnect
            case "NOTICE":
                log.Printf("NOTICE %s: %s (msg-id=%s)", m.Channel(), m.Trailing(), m.Tags["msg-id"])
            }
            select {
            case out <- m:
            case <-ctx.Done():
                return ctx.Err()
            }
        }
    }
}

// run keeps a session alive with exponential backoff until ctx ends.
func run(ctx context.Context, channels []string, out chan<- Message) {
    backoff := time.Second
    for ctx.Err() == nil {
        start := time.Now()
        err := session(ctx, channels, out)
        if ctx.Err() != nil {
            return
        }
        if time.Since(start) > time.Minute || errors.Is(err, errReconnect) {
            backoff = time.Second // it was healthy; reconnect quickly
        }
        log.Printf("connection ended: %v; reconnecting in %s", err, backoff)
        select {
        case <-time.After(backoff):
        case <-ctx.Done():
            return
        }
        backoff = min(backoff*2, 2*time.Minute)
    }
}
Enter fullscreen mode Exit fullscreen mode

ircURL is a var so you can point a test build somewhere unreachable and watch the backoff work. I built one with -ldflags "-X main.ircURL=wss://127.0.0.1:9":

2026/11/20 11:23:37 connection ended: dial: failed to WebSocket dial: failed to send handshake request: Get "https://127.0.0.1:9": dial tcp 127.0.0.1:9: connect: connection refused; reconnecting in 1s
2026/11/20 11:23:38 connection ended: dial: failed to WebSocket dial: failed to send handshake request: Get "https://127.0.0.1:9": dial tcp 127.0.0.1:9: connect: connection refused; reconnecting in 2s
2026/11/20 11:23:40 connection ended: dial: failed to WebSocket dial: failed to send handshake request: Get "https://127.0.0.1:9": dial tcp 127.0.0.1:9: connect: connection refused; reconnecting in 4s
2026/11/20 11:23:44 connection ended: dial: failed to WebSocket dial: failed to send handshake request: Get "https://127.0.0.1:9": dial tcp 127.0.0.1:9: connect: connection refused; reconnecting in 8s
2026/11/20 11:23:52 connection ended: dial: failed to WebSocket dial: failed to send handshake request: Get "https://127.0.0.1:9": dial tcp 127.0.0.1:9: connect: connection refused; reconnecting in 16s
2026/11/20 11:23:57 done: 0 lines, 0 chat messages stored
Enter fullscreen mode Exit fullscreen mode

The backoff resets after a connection that lasted more than a minute, so a normal RECONNECT doesn't leave you waiting two minutes.

Step 3: Store messages, and honor deletions

This is where most chat loggers are sloppy. Three things deserve care.

Moderation. CLEARMSG means a moderator deleted one message (target-msg-id). CLEARCHAT with a target-user-id is a timeout (when ban-duration is present) or a ban, and Twitch hides that user's recent messages. CLEARCHAT with no target clears the whole chat. The code keeps the row (time, user, channel) but empties the text and records why. In 22 minutes of logging I saw 74 CLEARCHAT events and no CLEARMSG at all. Timeouts are how these chats get moderated: 375 of 7,360 #xqc messages (5.1%) ended up redacted.

Emote positions are code points, not bytes. The emotes tag says 425618:2-4 and means characters 2 to 4. With an emoji earlier in the message, slicing the UTF-8 bytes gives you garbage. Slicing a []rune is correct. I checked all 1,702 emote ranges from my first two runs: 3 of them would have come out wrong with byte slicing, all in one message that started with ✌🏾. That's rare, and it's exactly the kind of bug that slips through testing.

Invisible characters. Twitch rejects the same message sent twice in a row, so chat clients append an invisible character to get around it. In my first two runs, 219 of 11,662 messages ended in U+034F (combining grapheme joiner) and 22 in U+E0000. Strip them before storing, or "KEKW KEKW" and "KEKW KEKW\u034f" count as different messages and searches miss.

Twitch's docs also say the server "may also send a message multiple times" if it thinks you didn't get it, so the message id is the primary key and inserts use INSERT OR IGNORE.

store.go:

package main

import (
    "database/sql"
    "strconv"
    "strings"
    "time"

    _ "modernc.org/sqlite"
)

const schema = `
PRAGMA journal_mode=WAL;
CREATE TABLE IF NOT EXISTS messages (
  id TEXT PRIMARY KEY, channel TEXT, room_id TEXT, user_id TEXT, login TEXT,
  display_name TEXT, text TEXT, sent_at INTEGER, badges TEXT,
  sub_months INTEGER, bits INTEGER, first_msg INTEGER, reply_to TEXT,
  emotes TEXT, deleted TEXT);
CREATE INDEX IF NOT EXISTS messages_chan_time ON messages(channel, sent_at);
CREATE INDEX IF NOT EXISTS messages_user ON messages(user_id);
CREATE VIRTUAL TABLE IF NOT EXISTS messages_fts USING fts5(
  text, content='messages', content_rowid='rowid');
CREATE TRIGGER IF NOT EXISTS messages_ai AFTER INSERT ON messages BEGIN
  INSERT INTO messages_fts(rowid, text) VALUES (new.rowid, new.text);
END;
CREATE TRIGGER IF NOT EXISTS messages_au AFTER UPDATE OF text ON messages BEGIN
  INSERT INTO messages_fts(messages_fts, rowid, text) VALUES ('delete', old.rowid, old.text);
  INSERT INTO messages_fts(rowid, text) VALUES (new.rowid, new.text);
END;
CREATE TABLE IF NOT EXISTS events (
  id TEXT PRIMARY KEY, channel TEXT, kind TEXT, login TEXT, system_msg TEXT, text TEXT,
  sent_at INTEGER, months INTEGER);
CREATE TABLE IF NOT EXISTS roomstate (
  channel TEXT PRIMARY KEY, room_id TEXT, slow INTEGER, followers_only INTEGER,
  emote_only INTEGER, subs_only INTEGER, updated_at INTEGER);`

func openDB(path string) (*sql.DB, error) {
    db, err := sql.Open("sqlite", path)
    if err != nil {
        return nil, err
    }
    db.SetMaxOpenConns(1)
    if _, err := db.Exec(schema); err != nil {
        return nil, err
    }
    return db, nil
}

func atoi(s string) int { n, _ := strconv.Atoi(s); return n }

func sentAt(m Message) int64 {
    if ts, err := strconv.ParseInt(m.Tags["tmi-sent-ts"], 10, 64); err == nil {
        return ts
    }
    return time.Now().UnixMilli()
}

// subMonths reads "subscriber/14" out of badge-info.
func subMonths(badgeInfo string) int {
    for _, b := range strings.Split(badgeInfo, ",") {
        if k, v, ok := strings.Cut(b, "/"); ok && (k == "subscriber" || k == "founder") {
            return atoi(v)
        }
    }
    return 0
}

// emoteNames maps the emotes tag ("25:0-4,6-10/1902:12-16") to names.
// Positions count Unicode code points, not bytes, so index into []rune.
func emoteNames(tag, text string) []string {
    if tag == "" {
        return nil
    }
    runes := []rune(text)
    var names []string
    for _, e := range strings.Split(tag, "/") {
        _, ranges, _ := strings.Cut(e, ":")
        for _, r := range strings.Split(ranges, ",") {
            a, b, ok := strings.Cut(r, "-")
            s, t := atoi(a), atoi(b)
            if !ok || s < 0 || t >= len(runes) || s > t {
                continue
            }
            names = append(names, string(runes[s:t+1]))
        }
    }
    return names
}

// clean drops the invisible characters chat clients append to get past
// Twitch's "identical message" check (U+034F, U+E0000), plus trailing spaces.
func clean(s string) string {
    s = strings.NewReplacer("\u034f", "", "\U000E0000", "").Replace(s)
    return strings.TrimSpace(s)
}

// apply writes one IRC message into the database inside tx.
// Returns true if it stored a chat message.
func apply(tx *sql.Tx, m Message) (bool, error) {
    switch m.Command {
    case "PRIVMSG":
        text := clean(m.Trailing())
        // /me messages arrive wrapped in CTCP ACTION.
        if strings.HasPrefix(text, "\x01ACTION ") {
            text = "/me " + strings.TrimSuffix(strings.TrimPrefix(text, "\x01ACTION "), "\x01")
        }
        res, err := tx.Exec(`INSERT OR IGNORE INTO messages VALUES
            (?,?,?,?,?,?,?,?,?,?,?,?,?,?,NULL)`,
            m.Tags["id"], m.Channel(), m.Tags["room-id"], m.Tags["user-id"], m.Nick(),
            m.Tags["display-name"], text, sentAt(m), m.Tags["badges"],
            subMonths(m.Tags["badge-info"]), atoi(m.Tags["bits"]), atoi(m.Tags["first-msg"]),
            m.Tags["reply-parent-msg-id"], strings.Join(emoteNames(m.Tags["emotes"], m.Trailing()), " "))
        if err != nil {
            return false, err
        }
        n, _ := res.RowsAffected() // 0 when Twitch re-sent a message we already have
        return n == 1, nil

    case "USERNOTICE": // sub, resub, subgift, raid, announcement, ...
        months := atoi(m.Tags["msg-param-cumulative-months"])
        text := ""
        if len(m.Params) > 1 {
            text = m.Trailing()
        }
        _, err := tx.Exec(`INSERT OR IGNORE INTO events VALUES (?,?,?,?,?,?,?,?)`, m.Tags["id"], m.Channel(),
            m.Tags["msg-id"], m.Tags["login"], m.Tags["system-msg"], text, sentAt(m), months)
        return false, err

    case "CLEARMSG": // a moderator deleted one message
        _, err := tx.Exec(`UPDATE messages SET text='', emotes='', deleted='deleted'
            WHERE id=?`, m.Tags["target-msg-id"])
        return false, err

    case "CLEARCHAT":
        reason := "ban"
        if m.Tags["ban-duration"] != "" {
            reason = "timeout"
        }
        if uid := m.Tags["target-user-id"]; uid != "" {
            // Twitch hides a timed-out or banned user's recent messages; do the same.
            _, err := tx.Exec(`UPDATE messages SET text='', emotes='', deleted=?
                WHERE user_id=? AND channel=? AND sent_at<=? AND deleted IS NULL`,
                reason, uid, m.Channel(), sentAt(m))
            return false, err
        }
        _, err := tx.Exec(`UPDATE messages SET text='', emotes='', deleted='clear'
            WHERE channel=? AND sent_at<=? AND deleted IS NULL`, m.Channel(), sentAt(m))
        return false, err

    case "ROOMSTATE": // full state on JOIN, partial updates later
        _, err := tx.Exec(`INSERT INTO roomstate VALUES (?,?,?,?,?,?,?)
            ON CONFLICT(channel) DO UPDATE SET
              slow=COALESCE(excluded.slow, slow),
              followers_only=COALESCE(excluded.followers_only, followers_only),
              emote_only=COALESCE(excluded.emote_only, emote_only),
              subs_only=COALESCE(excluded.subs_only, subs_only),
              updated_at=excluded.updated_at`,
            m.Channel(), m.Tags["room-id"], optInt(m, "slow"), optInt(m, "followers-only"),
            optInt(m, "emote-only"), optInt(m, "subs-only"), time.Now().UnixMilli())
        return false, err
    }
    return false, nil
}

// optInt returns nil when the tag is absent, so partial ROOMSTATE updates
// don't overwrite the other settings.
func optInt(m Message, k string) any {
    v, ok := m.Tags[k]
    if !ok {
        return nil
    }
    return atoi(v)
}
Enter fullscreen mode Exit fullscreen mode

The FTS5 table uses content='messages', so the text isn't stored twice. The update trigger keeps the index in sync when a message is redacted, so a deleted message stops showing up in search.

Step 4: The commands

main.go wires it together. The logger buffers messages and writes them in one transaction per second, or every 1,000 messages. A commit per message adds a disk sync per message, which gets expensive in a busy chat: #kaicenat peaked at 814 messages in one minute during my run.

With -raw, every IRC line also goes to a text file. That file is worth keeping. import replays it into a fresh database, so you can change the schema or fix a bug and rebuild everything. Because of INSERT OR IGNORE, importing the same file twice adds nothing.

main.go:

// twitchlog: log Twitch chat to SQLite over IRC WebSockets, then search,
// summarize and export it. Usage:
//
//  twitchlog log    -db chat.db -channels xqc,caedrel [-for 30m] [-raw raw.log]
//  twitchlog search -db chat.db -q 'NEAR(gta mall, 10)' [-channel xqc] [-user name]
//  twitchlog stats  -db chat.db -channel xqc [-since 60m]
//  twitchlog export -db chat.db -channel xqc|-user name [-day 2026-11-20] [-tz Asia/Saigon]
//  twitchlog import -db chat.db -raw raw.log   (rebuild from a raw log)
//  twitchlog live   -channels xqc,caedrel   (needs SCRAPEATLAS_API_KEY)
package main

import (
    "bufio"
    "context"
    "database/sql"
    "flag"
    "fmt"
    "log"
    "os"
    "os/signal"
    "strings"
    "syscall"
    "time"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "usage: twitchlog log|search|stats|export|live [flags]")
        os.Exit(2)
    }
    cmd, args := os.Args[1], os.Args[2:]
    fs := flag.NewFlagSet(cmd, flag.ExitOnError)
    dbPath := fs.String("db", "chat.db", "SQLite file")
    channels := fs.String("channels", "", "comma-separated channel logins")
    channel := fs.String("channel", "", "one channel login")
    dur := fs.Duration("for", 0, "stop after this long (0 = until Ctrl-C)")
    raw := fs.String("raw", "", "also append raw IRC lines to this file")
    q := fs.String("q", "", "FTS5 query")
    user := fs.String("user", "", "filter by login")
    since := fs.Duration("since", time.Hour, "stats window")
    day := fs.String("day", "", "export one day (YYYY-MM-DD, in -tz)")
    tz := fs.String("tz", "UTC", "time zone for export/search output")
    fs.Parse(args)

    loc, err := time.LoadLocation(*tz)
    if err != nil {
        log.Fatal(err)
    }
    if cmd == "live" {
        if err := cmdLive(split(*channels)); err != nil {
            log.Fatal(err)
        }
        return
    }
    db, err := openDB(*dbPath)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    switch cmd {
    case "log":
        err = cmdLog(db, split(*channels), *dur, *raw)
    case "import":
        err = cmdImport(db, *raw)
    case "search":
        err = cmdSearch(db, *q, *channel, *user, loc)
    case "stats":
        err = cmdStats(db, *channel, *since)
    case "export":
        err = cmdExport(db, *channel, *user, *day, loc)
    default:
        err = fmt.Errorf("unknown command %q", cmd)
    }
    if err != nil {
        log.Fatal(err)
    }
}

func split(s string) []string {
    var out []string
    for _, c := range strings.Split(s, ",") {
        if c = strings.TrimPrefix(strings.ToLower(strings.TrimSpace(c)), "#"); c != "" {
            out = append(out, c)
        }
    }
    return out
}

func cmdLog(db *sql.DB, channels []string, dur time.Duration, rawPath string) error {
    if len(channels) == 0 {
        return fmt.Errorf("-channels is required")
    }
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()
    if dur > 0 {
        var cancel context.CancelFunc
        ctx, cancel = context.WithTimeout(ctx, dur)
        defer cancel()
    }
    var raw *os.File
    if rawPath != "" {
        f, err := os.OpenFile(rawPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
        if err != nil {
            return err
        }
        defer f.Close()
        raw = f
    }

    msgs := make(chan Message, 4096)
    go func() { run(ctx, channels, msgs); close(msgs) }()

    // Batch writes: one transaction per second instead of one per message.
    tick := time.NewTicker(time.Second)
    defer tick.Stop()
    var batch []Message
    var stored, total int
    flush := func() error {
        if len(batch) == 0 {
            return nil
        }
        tx, err := db.Begin()
        if err != nil {
            return err
        }
        for _, m := range batch {
            ok, err := apply(tx, m)
            if err != nil {
                tx.Rollback()
                return fmt.Errorf("%s: %w", m.Command, err)
            }
            if ok {
                stored++
            }
        }
        batch = batch[:0]
        return tx.Commit()
    }
    report := time.Now()
    for {
        select {
        case m, ok := <-msgs:
            if !ok {
                err := flush()
                log.Printf("done: %d lines, %d chat messages stored", total, stored)
                return err
            }
            total++
            if raw != nil {
                fmt.Fprintln(raw, m.Raw)
            }
            switch m.Command {
            case "JOIN":
                log.Printf("joined #%s", m.Channel())
            case "PING":
                log.Printf("PING from server, answered with PONG")
            case "PRIVMSG", "USERNOTICE", "CLEARMSG", "CLEARCHAT", "ROOMSTATE":
                batch = append(batch, m)
            }
            if len(batch) >= 1000 {
                if err := flush(); err != nil {
                    return err
                }
            }
        case <-tick.C:
            if err := flush(); err != nil {
                return err
            }
            if time.Since(report) > time.Minute {
                log.Printf("%d chat messages stored so far", stored)
                report = time.Now()
            }
        }
    }
}

// cmdImport replays a raw IRC log into the database. INSERT OR IGNORE on the
// message id makes it safe to run twice or over overlapping logs.
func cmdImport(db *sql.DB, rawPath string) error {
    f, err := os.Open(rawPath)
    if err != nil {
        return err
    }
    defer f.Close()
    tx, err := db.Begin()
    if err != nil {
        return err
    }
    sc := bufio.NewScanner(f)
    sc.Buffer(make([]byte, 64*1024), 1<<20)
    lines, stored := 0, 0
    for sc.Scan() {
        lines++
        ok, err := apply(tx, Parse(sc.Text()))
        if err != nil {
            tx.Rollback()
            return fmt.Errorf("line %d: %w", lines, err)
        }
        if ok {
            stored++
        }
    }
    if err := sc.Err(); err != nil {
        tx.Rollback()
        return err
    }
    log.Printf("imported %d lines, %d new chat messages", lines, stored)
    return tx.Commit()
}

func cmdSearch(db *sql.DB, q, channel, user string, loc *time.Location) error {
    if q == "" {
        return fmt.Errorf("-q is required")
    }
    rows, err := db.Query(`
        SELECT m.channel, m.login, m.sent_at,
               snippet(messages_fts, 0, '[', ']', '…', 12)
        FROM messages_fts JOIN messages m ON m.rowid = messages_fts.rowid
        WHERE messages_fts MATCH ? AND (?='' OR m.channel=?) AND (?='' OR m.login=?)
        ORDER BY m.sent_at DESC LIMIT 50`, q, channel, channel, user, user)
    if err != nil {
        return err
    }
    defer rows.Close()
    n := 0
    for rows.Next() {
        var ch, login, snip string
        var ts int64
        if err := rows.Scan(&ch, &login, &ts, &snip); err != nil {
            return err
        }
        fmt.Printf("%s #%s %s: %s\n", time.UnixMilli(ts).In(loc).Format("2006-01-02 15:04:05"), ch, login, snip)
        n++
    }
    fmt.Printf("%d matches (newest first, max 50)\n", n)
    return rows.Err()
}

func cmdExport(db *sql.DB, channel, user, day string, loc *time.Location) error {
    if channel == "" && user == "" {
        return fmt.Errorf("-channel or -user is required")
    }
    from, to := int64(0), int64(1<<62)
    if day != "" {
        d, err := time.ParseInLocation("2006-01-02", day, loc)
        if err != nil {
            return err
        }
        from, to = d.UnixMilli(), d.AddDate(0, 0, 1).UnixMilli()
    }
    rows, err := db.Query(`SELECT sent_at, channel, login, text, deleted FROM messages
        WHERE (?='' OR channel=?) AND (?='' OR login=?) AND sent_at>=? AND sent_at<?
        ORDER BY sent_at`, channel, channel, user, user, from, to)
    if err != nil {
        return err
    }
    defer rows.Close()
    for rows.Next() {
        var ts int64
        var ch, login, text string
        var deleted sql.NullString
        if err := rows.Scan(&ts, &ch, &login, &text, &deleted); err != nil {
            return err
        }
        if deleted.Valid {
            text = "<message " + deleted.String + ">"
        }
        fmt.Printf("[%s] #%s %s: %s\n", time.UnixMilli(ts).In(loc).Format("2006-01-02 15:04:05"), ch, login, text)
    }
    return rows.Err()
}
Enter fullscreen mode Exit fullscreen mode

Step 5: Stats, and checking who's live

stats.go has the channel report and the optional live command. Emote counts come from the emotes tag, so only Twitch-native emotes are counted. 7TV, BTTV, and FFZ emotes (OMEGALUL, KEKW, ICANT, and friends) are plain words in the text. You could count them by matching against those services' emote lists, but that's a separate project.

Joining channels that aren't live wastes join slots and fills your database with nothing. IRC can't tell you whether a channel is live, and Helix needs an app token for that. I used ScrapeAtlas* for this check. It's a scraping API that covers 50 platforms, and its GET /v1/twitch/profile/basic?handle=... returns isLive and currentViewersCount. If you already have a Helix app token, Get Streams does the same job.

stats.go:

package main

import (
    "database/sql"
    "encoding/json"
    "fmt"
    "net/http"
    "net/url"
    "os"
    "sort"
    "strings"
    "time"
)

func cmdStats(db *sql.DB, channel string, since time.Duration) error {
    if channel == "" {
        return fmt.Errorf("-channel is required")
    }
    from := time.Now().Add(-since).UnixMilli()
    var n, chatters, subs, firsts, bits int
    var t0, t1 sql.NullInt64
    err := db.QueryRow(`SELECT COUNT(*), COUNT(DISTINCT user_id),
        COALESCE(SUM(sub_months>0),0), COALESCE(SUM(first_msg),0), COALESCE(SUM(bits),0),
        MIN(sent_at), MAX(sent_at)
        FROM messages WHERE channel=? AND sent_at>=?`, channel, from).
        Scan(&n, &chatters, &subs, &firsts, &bits, &t0, &t1)
    if err != nil {
        return err
    }
    if n == 0 {
        return fmt.Errorf("no messages for #%s in the last %s", channel, since)
    }
    mins := max(float64(t1.Int64-t0.Int64)/60000, 1)
    fmt.Printf("#%s, last %s: %d messages from %d chatters over %.1f min (%.1f msg/min)\n",
        channel, since, n, chatters, mins, float64(n)/mins)
    fmt.Printf("  from subscribers: %.1f%%   first-time chatters: %d   bits cheered: %d\n",
        100*float64(subs)/float64(n), firsts, bits)

    var peakMin int64
    var peak int
    db.QueryRow(`SELECT sent_at/60000 AS m, COUNT(*) c FROM messages
        WHERE channel=? AND sent_at>=? GROUP BY m ORDER BY c DESC LIMIT 1`, channel, from).Scan(&peakMin, &peak)
    fmt.Printf("  busiest minute: %s UTC with %d messages\n",
        time.UnixMilli(peakMin*60000).UTC().Format("15:04"), peak)

    rows, err := db.Query(`SELECT COALESCE(deleted,'kept'), COUNT(*) FROM messages
        WHERE channel=? AND sent_at>=? GROUP BY 1 ORDER BY 2 DESC`, channel, from)
    if err != nil {
        return err
    }
    var parts []string
    for rows.Next() {
        var k string
        var c int
        rows.Scan(&k, &c)
        parts = append(parts, fmt.Sprintf("%s=%d", k, c))
    }
    rows.Close()
    fmt.Println("  moderation:", strings.Join(parts, " "))

    // Top chatters.
    rows, err = db.Query(`SELECT login, COUNT(*) c FROM messages
        WHERE channel=? AND sent_at>=? GROUP BY user_id ORDER BY c DESC LIMIT 5`, channel, from)
    if err != nil {
        return err
    }
    parts = parts[:0]
    for rows.Next() {
        var login string
        var c int
        rows.Scan(&login, &c)
        parts = append(parts, fmt.Sprintf("%s (%d)", login, c))
    }
    rows.Close()
    fmt.Println("  top chatters:", strings.Join(parts, ", "))

    // Top Twitch emotes (from the emotes tag, so third-party emotes are not counted).
    counts := map[string]int{}
    rows, err = db.Query(`SELECT emotes FROM messages
        WHERE channel=? AND sent_at>=? AND emotes<>''`, channel, from)
    if err != nil {
        return err
    }
    for rows.Next() {
        var e string
        rows.Scan(&e)
        for _, name := range strings.Fields(e) {
            counts[name]++
        }
    }
    rows.Close()
    type kv struct {
        k string
        v int
    }
    var top []kv
    for k, v := range counts {
        top = append(top, kv{k, v})
    }
    sort.Slice(top, func(i, j int) bool {
        if top[i].v != top[j].v {
            return top[i].v > top[j].v
        }
        return top[i].k < top[j].k
    })
    parts = parts[:0]
    for i := 0; i < len(top) && i < 8; i++ {
        parts = append(parts, fmt.Sprintf("%s %d", top[i].k, top[i].v))
    }
    fmt.Println("  top Twitch emotes:", strings.Join(parts, ", "))

    // Events (subs, gifts, raids...).
    rows, err = db.Query(`SELECT kind, COUNT(*) FROM events
        WHERE channel=? AND sent_at>=? GROUP BY kind ORDER BY 2 DESC`, channel, from)
    if err != nil {
        return err
    }
    parts = parts[:0]
    for rows.Next() {
        var k string
        var c int
        rows.Scan(&k, &c)
        parts = append(parts, fmt.Sprintf("%s=%d", k, c))
    }
    rows.Close()
    fmt.Println("  events:", strings.Join(parts, " "))
    return nil
}

// cmdLive asks ScrapeAtlas which channels are live right now and prints
// them comma-separated, so you can feed them to `twitchlog log -channels`.
func cmdLive(channels []string) error {
    key := os.Getenv("SCRAPEATLAS_API_KEY")
    if key == "" {
        return fmt.Errorf("set SCRAPEATLAS_API_KEY")
    }
    var live []string
    for _, ch := range channels {
        req, _ := http.NewRequest("GET",
            "https://api.scrapeatlas.com/v1/twitch/profile/basic?handle="+url.QueryEscape(ch), nil)
        req.Header.Set("x-api-key", key)
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        var p struct {
            DisplayName         string `json:"displayName"`
            IsLive              bool   `json:"isLive"`
            CurrentViewersCount *int   `json:"currentViewersCount"`
            Failure             *struct {
                Code string `json:"code"`
            } `json:"failure"`
        }
        err = json.NewDecoder(res.Body).Decode(&p)
        res.Body.Close()
        switch {
        case err != nil:
            return fmt.Errorf("%s: %w", ch, err)
        case p.Failure != nil:
            fmt.Fprintf(os.Stderr, "%-16s %s\n", ch, p.Failure.Code)
        case p.IsLive:
            fmt.Fprintf(os.Stderr, "%-16s live, %d viewers\n", ch, *p.CurrentViewersCount)
            live = append(live, ch)
        default:
            fmt.Fprintf(os.Stderr, "%-16s offline\n", ch)
        }
    }
    fmt.Println(strings.Join(live, ","))
    return nil
}
Enter fullscreen mode Exit fullscreen mode

Running it

Build it and check who's live:

$ go build -o twitchlog .
$ export SCRAPEATLAS_API_KEY=...
$ ./twitchlog live -channels xqc,caedrel,kaicenat,zackrawrr,shroud,jynxzi,thisuserdoesnotexist_zz9
xqc              live, 32822 viewers
caedrel          offline
kaicenat         live, 30218 viewers
zackrawrr        offline
shroud           offline
jynxzi           offline
thisuserdoesnotexist_zz9 not_found
xqc,kaicenat
Enter fullscreen mode Exit fullscreen mode

Status goes to stderr and the live list goes to stdout, so you can pass it straight in: ./twitchlog log -channels "$(./twitchlog live -channels ...)".

Log for 12 minutes:

$ ./twitchlog log -db chat.db -channels xqc,kaicenat -for 12m -raw raw2.log
2026/11/20 11:24:07 joined #xqc
2026/11/20 11:24:07 joined #kaicenat
2026/11/20 11:25:05 250 chat messages stored so far
2026/11/20 11:26:05 755 chat messages stored so far
2026/11/20 11:27:05 1174 chat messages stored so far
2026/11/20 11:28:05 1711 chat messages stored so far
2026/11/20 11:28:37 PING from server, answered with PONG
2026/11/20 11:29:06 2411 chat messages stored so far
2026/11/20 11:30:07 3106 chat messages stored so far
2026/11/20 11:31:08 3704 chat messages stored so far
2026/11/20 11:32:09 4427 chat messages stored so far
2026/11/20 11:33:09 4994 chat messages stored so far
2026/11/20 11:33:36 PING from server, answered with PONG
2026/11/20 11:34:09 5398 chat messages stored so far
2026/11/20 11:35:09 6106 chat messages stored so far
2026/11/20 11:36:05 done: 7279 lines, 7188 chat messages stored
Enter fullscreen mode Exit fullscreen mode

The first run lasted 10 minutes (4,474 messages) and the second one 12 minutes (7,188). Notice the gap: the first run ended at 11:23:05 and the second one joined at 11:24:07, so about a minute of chat is missing from my logs for good. Nothing can fill it in later. For a real logger, run it as a service and leave it running.

The stats report, with chatters' usernames replaced:

$ ./twitchlog stats -db chat.db -channel xqc -since 3h
#xqc, last 3h0m0s: 7360 messages from 1374 chatters over 26.4 min (279.2 msg/min)
  from subscribers: 36.7%   first-time chatters: 22   bits cheered: 800
  busiest minute: 04:31 UTC with 574 messages
  moderation: kept=6982 timeout=375 ban=3
  top chatters: viewer_02 (94), viewer_03 (78), viewer_04 (78), viewer_05 (75), viewer_06 (68)
  top Twitch emotes: LUL 71, xqcL 40, BatChest 32, DinoDance 9, xqcEZ_TK 9, DinoBusted 8, Kreygasm 8, JinxLUL 7
  events: resub=7 viewermilestone=2 sub=2
$ ./twitchlog stats -db chat.db -channel kaicenat -since 3h
#kaicenat, last 3h0m0s: 5083 messages from 1536 chatters over 26.3 min (193.1 msg/min)
  from subscribers: 25.8%   first-time chatters: 135   bits cheered: 1000
  busiest minute: 04:35 UTC with 814 messages
  moderation: kept=4952 timeout=128 ban=3
  top chatters: viewer_07 (107), viewer_08 (65), viewer_09 (52), viewer_10 (49), viewer_11 (48)
  top Twitch emotes: kaicFireMGM 448, <3 62, kaicQM 53, NeedHeal 42, BleedPurpleHD 36, kaicMGM 35, kaicHeart 27, WutFace 24
  events: resub=9 sub=8 raid=5 submysterygift=2 subgift=2 viewermilestone=1
Enter fullscreen mode Exit fullscreen mode

The two chats differ in clear ways. #xqc is busier and more of its messages come from subscribers (36.7% vs 25.8%), and moderators hand out a lot of timeouts. #kaicenat has six times as many first-time chatters, more raids, and one channel emote, kaicFireMGM, accounts for about a third of all Twitch emotes used there (448 of 1,370). The "busiest minute" is in UTC (04:35 UTC is 11:35 in Asia/Saigon, where I ran this).

Full-text search uses FTS5 syntax: phrases in quotes, * for prefixes, NEAR(a b, 10) for words close together.

$ ./twitchlog search -db chat.db -q '"gta 6"' -channel xqc -tz Asia/Saigon | tail -6
2026-11-20 11:21:00 #xqc viewer_12: [gta 6] and 2000s youtube in one stream, nice
2026-11-20 11:20:25 #xqc viewer_13: have you seen that mall in [gta 6] yet
2026-11-20 11:17:08 #xqc viewer_14: the [gta 6] has that gta 5 driving wobble screen thing, i…
2026-11-20 11:14:12 #xqc viewer_15: [GTA 6] is kinda mid compared to gta 5
2026-11-20 11:13:19 #xqc viewer_14: why is forsen not streaming [gta 6] yet :(
12 matches (newest first, max 50)
$ ./twitchlog search -db chat.db -q 'NEAR(gta mall, 10)' -tz Asia/Saigon
2026-11-20 11:20:25 #xqc viewer_13: have you seen that [mall] in [gta] 6 yet
1 matches (newest first, max 50)
Enter fullscreen mode Exit fullscreen mode

Pass user input through as-is and FTS5 will choke on punctuation:

$ ./twitchlog search -db chat.db -q 'gta 6?'
2026/11/20 11:42:19 SQL logic error: fts5: syntax error near "?" (1)
Enter fullscreen mode Exit fullscreen mode

If the query comes from users, wrap each word in double quotes before you pass it on.

Exports print one line per message. Redacted messages keep their place in the timeline, so the log stays honest about what happened without repeating what was removed:

$ ./twitchlog export -db chat.db -channel xqc -day 2026-11-20 -tz Asia/Saigon
[2026-11-20 11:20:53] #xqc viewer_16: glass in front
[2026-11-20 11:20:53] #xqc viewer_17: jewelry
[2026-11-20 11:20:53] #xqc viewer_18: All that for 300
[2026-11-20 11:20:53] #xqc viewer_19: <message timeout>
[2026-11-20 11:20:54] #xqc viewer_20: Glass
[2026-11-20 11:20:54] #xqc viewer_21: jewlery
[2026-11-20 11:20:54] #xqc viewer_22: meow
[2026-11-20 11:20:55] #xqc viewer_23: jewelry💎!!
[2026-11-20 11:20:55] #xqc viewer_24: EZ
[2026-11-20 11:20:55] #xqc viewer_25: Omds break the front
[2026-11-20 11:20:55] #xqc viewer_26: smash the glasses
[2026-11-20 11:20:55] #xqc viewer_27: Merch drop #3 is out https://xqc.store/ EZ
Enter fullscreen mode Exit fullscreen mode

Storage is small. The raw logs for all 12,443 messages came to 4.9 MB, and the SQLite file, FTS index included, to 3.4 MB. That's about 400 bytes of raw IRC per message, and a chat at 280 messages a minute would produce roughly 160 MB of raw log per 24 hours of streaming. Compress old raw logs with gzip.

Pitfalls, in one list

  • There's no backfill. Every minute your logger is down is lost. Neither IRC nor EventSub has history.
  • Raw IRC ports may be blocked. Port 443 WebSockets got through where 6667 and 6697 didn't.
  • Split frames on \r\n. One WebSocket message can carry several IRC lines.
  • Answer PING, or you get dropped. It comes about every 5 minutes, and a read deadline a bit longer than that catches dead connections.
  • A nonexistent channel fails silently. Confirm each JOIN echo.
  • Emote ranges are code points. Slice []rune, not bytes.
  • Strip U+034F and U+E0000 before storing or comparing messages.
  • Deduplicate on the message id. Twitch can send the same message twice, and replays will too.
  • Use user-id, not the login, to group messages by person. Logins can be renamed.
  • Timeouts redact more than one message. Store the reason, and keep the CLEARCHAT timestamp so a replay doesn't redact messages sent after it.
  • Batch your writes. A busy chat broke 800 messages a minute.
  • Third-party emotes are invisible to the tags.
  • Join limits: 20 joins per 10 seconds, 100 channels at once on a normal account.

Wrapping up

  • Twitch chat logs are forward-only. You build them by being connected; no API gives you yesterday's chat.
  • IRC over wss:// on port 443 is the simplest way to read chat, and the anonymous login needs no app registration. EventSub is the documented route Twitch recommends.
  • Treat moderation as data. Timeouts and deletions arrive as events. Honoring them keeps your log in line with what Twitch shows, and it's the decent thing to do.
  • Keep the raw log. With import, it's a full rebuild whenever your schema changes.

What would you track in a chat? Emote trends across a season, moderation load per stream, how raids change a chat? If you've built a Twitch chat logger with EventSub instead, I'd like to hear how it went.


*I build it.

Top comments (0)