go 107 lines · 3 tabs

Stream and Decode Newline-Delimited JSON (NDJSON) from an HTTP Response Body in Go

Shared by codesnips Jul 2026

3 tabs

package feed

import "time"

type Event struct {
	ID        string    `json:"id"`
	Type      string    `json:"type"`
	Actor     string    `json:"actor"`
	CreatedAt time.Time `json:"created_at"`
}

package feed

import (
	"bufio"
	"context"
	"encoding/json"
	"io"
)

const maxLineBytes = 4 << 20 // 4MB per record

func StreamEvents(ctx context.Context, body io.ReadCloser) (<-chan Event, <-chan error) {
	events := make(chan Event)
	errs := make(chan error, 1)

	go func() {
		defer body.Close()
		defer close(events)
		defer close(errs)

		scanner := bufio.NewScanner(body)
		scanner.Buffer(make([]byte, 0, 64*1024), maxLineBytes)

		for scanner.Scan() {
			line := scanner.Bytes()
			if len(line) == 0 {
				continue
			}

			var ev Event
			if err := json.Unmarshal(line, &ev); err != nil {
				errs <- err
				return
			}

			select {
			case events <- ev:
			case <-ctx.Done():
				errs <- ctx.Err()
				return
			}
		}

		if err := scanner.Err(); err != nil {
			errs <- err
		}
	}()

	return events, errs
}

package feed

import (
	"context"
	"fmt"
	"net/http"
)

type Client struct {
	HTTP    *http.Client
	BaseURL string
}

func (c *Client) FetchEvents(ctx context.Context, org string) error {
	url := fmt.Sprintf("%s/orgs/%s/events", c.BaseURL, org)
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
		return err
	}
	req.Header.Set("Accept", "application/x-ndjson")

	resp, err := c.HTTP.Do(req)
	if err != nil {
		return err
	}
	if resp.StatusCode != http.StatusOK {
		resp.Body.Close()
		return fmt.Errorf("feed: unexpected status %d", resp.StatusCode)
	}

	events, errs := StreamEvents(ctx, resp.Body)
	for {
		select {
		case ev, ok := <-events:
			if !ok {
				return nil // stream drained cleanly
			}
			fmt.Printf("%s %s by %s\n", ev.CreatedAt.Format("15:04:05"), ev.Type, ev.Actor)
		case err := <-errs:
			if err != nil {
				return err
			}
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

3 files · go Explain with highlit

Newline-delimited JSON (NDJSON) is a wire format where each line is a self-contained JSON object, which makes it ideal for streaming large result sets over HTTP without buffering the entire payload in memory. This snippet shows how to consume such a stream incrementally in Go, decoding one record at a time as bytes arrive off the socket.

The event.go tab defines the Event value type that each line decodes into. Nothing exotic happens here — it exists so the decoder has a concrete target and so the consumer code reads clearly. Keeping the domain type small and separate is deliberate: the streaming logic should not care what shape the records take.

The ndjson.go tab holds the core reader. StreamEvents wraps the response body in a bufio.Scanner and raises the token buffer with scanner.Buffer so a single oversized line does not trip the default 64KB limit. Rather than returning a slice, it returns a receive-only channel and pushes each decoded Event through it; this gives the caller natural backpressure, since the goroutine blocks on the send until the consumer is ready. The context.Context is checked on every iteration so a cancelled request stops the scan promptly instead of draining the whole body. Blank lines are skipped, JSON errors are surfaced through a second error channel, and defer body.Close() guarantees the socket is released even on early exit. Returning two channels keeps data and failure paths distinct without forcing a sentinel value into the stream.

The client.go tab ties it together. FetchEvents builds a request with the caller's context.Context, sets an Accept header advertising application/x-ndjson, and hands the live resp.Body straight to StreamEvents — the body is never read into memory whole. The consumer uses a select over the two channels so it can react to either a record or an error, and the ok check on the event channel signals clean end-of-stream.

The main trade-off is that a bufio.Scanner-based approach assumes newline framing is reliable and that no single record exceeds the configured buffer; for adversarial or unbounded input a json.Decoder reading tokens directly is safer. For well-behaved server streams, though, this pattern is simple, allocation-light, and cancellable.

Related snips

package api

import (
  "net/http"
  "runtime/debug"
)

Expose build metadata for debugging deploys

go observability build

by Leah Thompson 1 tab

typescript

export interface RetryOptions {
  retries: number;
  baseMs: number;
  maxMs: number;
  signal?: AbortSignal;
  onRetry?: (attempt: number, delay: number, err: unknown) => void;

Exponential backoff with jitter for retries

typescript reliability retry

by codesnips 2 tabs

package dbutil

import (
  "context"

  "github.com/jackc/pgconn"

Retry Postgres serialization failures with bounded attempts

go postgres transactions

by Leah Thompson 1 tab

ruby

require "csv"

class PeopleCsvStream
  include Enumerable

  HEADERS = %w[id full_name email signed_up_at plan].freeze

Resilient CSV Export as a Streamed Response

rails performance streaming

by codesnips 3 tabs

rust

use crossbeam::channel::unbounded;
use std::thread;

fn main() {
    let (tx, rx) = unbounded();

Crossbeam for advanced concurrent data structures

rust concurrency lock-free

by Marcus Chen 1 tab

javascript

// Creating a Promise
const myPromise = new Promise((resolve, reject) => {
  const success = true;

  setTimeout(() => {
    if (success) {

Promises and async/await patterns for asynchronous JavaScript

javascript promises async-await

by Alex Chang 1 tab

Share this code

Here's the card — post it anywhere.

Stream and Decode Newline-Delimited JSON (NDJSON) from an HTTP Response Body in Go — share card

Link copied