codesnips
go 27 lines · 1 tab

Avoid bufio.Scanner token limits by switching to Reader

Leah Thompson Jan 2026
1 tab 
package ingest

import (
  "bufio"
  "bytes"
  "io"
)

func ReadLines(r io.Reader, fn func([]byte) error) error {
  br := bufio.NewReader(r)
  for {
    line, err := br.ReadBytes('
')
    if len(bytes.TrimSpace(line)) > 0 {
      if e := fn(bytes.TrimRight(line, "
")); e != nil {
        return e
      }
    }
    if err == io.EOF {
      return nil
    }
    if err != nil {
      return err
    }
  }
}
1 file · go Explain with highlit

bufio.Scanner is great until it isn’t: by default it refuses tokens larger than 64K, which makes it a bad fit for long log lines or large JSON records. The failure mode is subtle—scan stops and Err() returns a token-too-long error. For production log processing, I prefer bufio.Reader.ReadBytes(' ') or ReadString so there’s no hard token limit. The example below reads lines in a loop, handles io.EOF correctly, and leaves you in control of memory use. If you truly need a limit, you can enforce one explicitly by checking len(line) and rejecting oversized inputs. This pattern is a small change, but it eliminates a class of ingestion outages where a single large line breaks an entire pipeline.

Related snips

go 
package api

import (
  "net/http"
  "runtime/debug"
)

Expose build metadata for debugging deploys

go observability build
by Leah Thompson 1 tab
typescript 
export interface RetryOptions {
  retries: number;
  baseMs: number;
  maxMs: number;
  signal?: AbortSignal;
  onRetry?: (attempt: number, delay: number, err: unknown) => void;

Exponential backoff with jitter for retries

typescript reliability retry
by codesnips 2 tabs
go 
package dbutil

import (
  "context"

  "github.com/jackc/pgconn"

Retry Postgres serialization failures with bounded attempts

go postgres transactions
by Leah Thompson 1 tab
go 
package files

import (
  "context"
  "time"

Presigned S3 upload URLs (AWS SDK v2)

go aws s3
by Leah Thompson 1 tab
go 
package deps

import (
  "net"
  "net/http"
  "time"

HTTP client tuned for production: timeouts, transport, and connection reuse

go http client
by Leah Thompson 1 tab
go 
package api

import (
  "io"
  "net/http"
  "os"

Safe multipart uploads using temp files (bounded memory)

go http uploads
by Leah Thompson 1 tab

Share this code

Here's the card — post it anywhere.

Avoid bufio.Scanner token limits by switching to Reader — share card
Link copied