Avoid bufio.Scanner token limits by switching to Reader

10981 views

                  package ingest

import (
  "bufio"
  "bytes"
  "io"
)

func ReadLines(r io.Reader, fn func([]byte) error) error {
  br := bufio.NewReader(r)
  for {
    line, err := br.ReadBytes('
')
    if len(bytes.TrimSpace(line)) > 0 {
      if e := fn(bytes.TrimRight(line, "
")); e != nil {
        return e
      }
    }
    if err == io.EOF {
      return nil
    }
    if err != nil {
      return err
    }
  }
}

bufio.Scanner is great until it isn’t: by default it refuses tokens larger than 64K, which makes it a bad fit for long log lines or large JSON records. The failure mode is subtle—scan stops and Err() returns a token-too-long error. For production log processing, I prefer bufio.Reader.ReadBytes(' ') or ReadString so there’s no hard token limit. The example below reads lines in a loop, handles io.EOF correctly, and leaves you in control of memory use. If you truly need a limit, you can enforce one explicitly by checking len(line) and rejecting oversized inputs. This pattern is a small change, but it eliminates a class of ingestion outages where a single large line breaks an entire pipeline.

Leah Thompson

More from Leah Thompson

Avoid bufio.Scanner token limits by switching to Reader

0 Comments

More from Leah Thompson