codesnips

etl

sql 
-- Import CSV with COPY (fastest method)
COPY users (username, email, age, created_at)
FROM '/path/to/users.csv'
WITH (
  FORMAT csv,
  HEADER true,

Efficient data import and export strategies

database import export
by Maria Garcia 2 tabs
typescript 
import { Writable } from "node:stream";
import type { Pool } from "pg";

interface Row {
  email: string;
  name: string;

Streaming CSV import (Node streams)

streams postgres nodejs
by codesnips 3 tabs
java 
package com.example.demo.batch;

import com.example.demo.model.User;
import com.example.demo.model.UserDTO;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;

Batch processing with Spring Batch

java spring-batch batch-processing
by David Kumar 2 tabs
python 
from __future__ import annotations

from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Dict, List

Parsing a Nested JSON Feed into Normalized Dataclasses in Python

dataclasses json parsing
by codesnips 3 tabs
python 
import hashlib
from datetime import datetime, timezone

from sqlalchemy import BigInteger, Column, DateTime, String, UniqueConstraint
from sqlalchemy.orm import declarative_base

Efficient Bulk Insert in SQLAlchemy with a Reusable Chunking Helper

sqlalchemy postgres bulk-insert
by codesnips 3 tabs
javascript 
'use strict';

const { Transform } = require('stream');

class LineSplitter extends Transform {
  constructor(options = {}) {

Split a Large Log File Into Date-Based Chunks With a Node.js Transform Stream

nodejs streams backpressure
by codesnips 3 tabs
python 
from datetime import date, datetime
from decimal import Decimal, InvalidOperation
from enum import Enum

from pydantic import BaseModel, field_validator

Parsing and Normalizing a Transactions CSV into Typed Records with Pydantic

csv pydantic data-parsing
by codesnips 2 tabs
sql 
CREATE TABLE events (
    id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    source      text        NOT NULL,
    external_id text        NOT NULL,
    payload     jsonb       NOT NULL,
    occurred_at timestamptz NOT NULL,

Batched writes with COPY (conceptual)

postgres performance sql
by codesnips 3 tabs
javascript 
const fs = require('fs');
const readline = require('readline');

async function* streamCsvRows(filePath) {
  const stream = fs.createReadStream(filePath, { encoding: 'utf8' });
  const rl = readline.createInterface({ input: stream, crlfDelay: Infinity });

Stream and Import a Large CSV File Line-by-Line with Node.js readline

nodejs streams readline
by codesnips 3 tabs