codesnips
go 55 lines · 1 tab

Streaming CSV import with batched inserts

Leah Thompson Jan 2026
1 tab 
package importers

import (
  "context"
  "encoding/csv"
  "io"

  "github.com/jackc/pgx/v5/pgxpool"
)

type Row struct {
  Email string
  Name  string
}

func ImportUsers(ctx context.Context, db *pgxpool.Pool, r io.Reader) error {
  cr := csv.NewReader(r)
  batch := make([]Row, 0, 500)

  flush := func() error {
    if len(batch) == 0 {
      return nil
    }
    b := &pgxpool.Batch{}
    for _, row := range batch {
      b.Queue(`INSERT INTO users(email, name) VALUES($1, $2) ON CONFLICT (email) DO UPDATE SET name=EXCLUDED.name`, row.Email, row.Name)
    }
    br := db.SendBatch(ctx, b)
    defer br.Close()
    if _, err := br.Exec(); err != nil {
      return err
    }
    batch = batch[:0]
    return nil
  }

  for {
    rec, err := cr.Read()
    if err == io.EOF {
      return flush()
    }
    if err != nil {
      return err
    }
    if len(rec) < 2 {
      continue
    }
    batch = append(batch, Row{Email: rec[0], Name: rec[1]})
    if len(batch) >= 500 {
      if err := flush(); err != nil {
        return err
      }
    }
  }
}
1 file · go Explain with highlit

CSV imports are where memory usage quietly explodes if you read everything at once. I stream rows with encoding/csv, validate each record, and batch inserts to keep DB overhead low. The important detail is backpressure: if the database is slow, the importer must naturally slow down rather than buffering millions of rows. I usually set a maximum batch size and flush on EOF. For validation, I keep a counter of rejected rows and return a summary so operators can see what happened without scanning logs. In production, I pair this with idempotency (import IDs) and a staging table so the main tables aren’t left half-updated on failures. The snippet below focuses on the core loop: read, validate, batch, flush. It’s simple, fast, and safe for large datasets.

Related snips

go 
package api

import (
  "net/http"
  "runtime/debug"
)

Expose build metadata for debugging deploys

go observability build
by Leah Thompson 1 tab
go 
package dbutil

import (
  "context"

  "github.com/jackc/pgconn"

Retry Postgres serialization failures with bounded attempts

go postgres transactions
by Leah Thompson 1 tab
ruby 
require "csv"

class PeopleCsvStream
  include Enumerable

  HEADERS = %w[id full_name email signed_up_at plan].freeze

Resilient CSV Export as a Streamed Response

rails performance streaming
by codesnips 3 tabs
ruby 
class AddSettingsToAccounts < ActiveRecord::Migration[7.1]
  disable_ddl_transaction!

  def change
    add_column :accounts, :settings, :jsonb, null: false, default: {}

Postgres JSONB Partial Index for Feature Flags

rails postgres jsonb
by codesnips 3 tabs
go 
package files

import (
  "context"
  "time"

Presigned S3 upload URLs (AWS SDK v2)

go aws s3
by Leah Thompson 1 tab
go 
package deps

import (
  "net"
  "net/http"
  "time"

HTTP client tuned for production: timeouts, transport, and connection reuse

go http client
by Leah Thompson 1 tab

Share this code

Here's the card — post it anywhere.

Streaming CSV import with batched inserts — share card
Link copied