typescript 113 lines · 3 tabs

BullMQ job idempotency via dedupe id

Shared by codesnips Jan 2026
3 tabs
import { Queue } from "bullmq";
import { createHash } from "crypto";

export const connection = { host: "127.0.0.1", port: 6379 };

export interface ChargePayload {
  paymentIntentId: string;
  amountCents: number;
  currency: string;
}

export const chargeQueue = new Queue<ChargePayload>("charges", { connection });

function dedupeId(intentId: string): string {
  return "charge:" + createHash("sha256").update(intentId).digest("hex").slice(0, 24);
}

export async function enqueueChargeCustomer(payload: ChargePayload) {
  const jobId = dedupeId(payload.paymentIntentId);

  return chargeQueue.add("charge-customer", payload, {
    jobId,
    attempts: 5,
    backoff: { type: "exponential", delay: 2000 },
    // Keep completed jobs so a re-add with the same jobId is ignored.
    removeOnComplete: { count: 5000 },
    removeOnFail: { age: 24 * 3600 },
  });
}
3 files · typescript Explain with highlit

This snippet shows how to make BullMQ jobs safely retryable and deduplicated so that the same logical event never produces two side effects, even when a producer enqueues it multiple times or a worker crashes mid-run. The core idea is two layers of protection: BullMQ's own jobId for enqueue-time dedupe, and a Redis-backed result marker for processing-time idempotency.

In queue.ts, the Queue is created and enqueueChargeCustomer derives a deterministic jobId from a stable business key (here a payment intent id) via dedupeId. BullMQ guarantees that adding a job with an existing jobId is a no-op while that job still exists, so two rapid webhook deliveries for the same intent collapse into one job. The removeOnComplete window matters: as long as the completed job is retained, re-adds are ignored, so the retention count effectively defines the dedupe horizon. That trade-off is important — a very short retention window reopens the door to duplicate enqueues, while a long one costs Redis memory.

Because jobId dedupe only helps at enqueue time, worker.ts adds a second guard for the case where a job is retried after it already succeeded partway. The IdempotencyStore in idempotency.ts uses a Redis SET ... NX lock plus a persisted result key. Before doing real work the worker calls begin, which either returns a cached result (short-circuiting duplicate processing) or acquires a lock. After the side effect runs, complete stores the result and releases the lock; abort releases the lock on failure so a retry can proceed.

The SET key value NX PX pattern gives an atomic "claim this key if unclaimed" with an expiry, preventing a dead worker from holding the lock forever. The persisted result: key is what makes a genuine retry idempotent: chargeCustomer runs once, and any later attempt reads the stored charge id instead of calling the payment provider again. Note the ordering in the worker — the external charge happens inside the lock, and only then is the result recorded, so a crash between the two leaves the lock to expire and the job to retry cleanly. This layering is the standard approach when exactly-once delivery is impossible but effectively-once processing is required.


Related snips

Share this code

Here's the card — post it anywhere.

BullMQ job idempotency via dedupe id — share card
Link copied