A payment gateway has a 400 ms hiccup. You have 10,000 clients in flight.
Every one of them times out in the same instant, catches the same error, and enters the same retry loop. Your backoff is 1s, 2s, 4s, so all 10,000 requests come back at t=1000ms. Together. The gateway, which was just starting to recover, receives the load it had before — in a single burst — and falls over again.
You didn't add retries to make things worse. You added them because the docs said to. But a retry loop without jitter isn't error handling. It's a traffic amplifier with a delay.
Three things fix this, and they fix different problems: jitter breaks the correlation between clients, a circuit breaker stops sending traffic to something already dead, and an idempotency key makes the retries safe in the first place. Most codebases have one of the three.
Everything below runs — no dependencies, Node 24+, tests included.
Failures are correlated because your clients are synchronized
A transient failure — a dropped connection, a 429, a 504 from the acquirer — is not random noise. It hits every client at the same time, because they're all talking to the same degraded dependency.
Those clients then compute a backoff delay from the same formula, with the same attempt number, starting from the same instant:
delay = base * 2^(attempt - 1)
Deterministic input, deterministic output. You didn't build a backoff. You built a scheduled stampede, and you scheduled it for every client simultaneously.
That's why gateway outages so often have a second, worse peak a few seconds after the first one. The retries cause it, not the original fault.
One more number worth internalizing: failure probability compounds across hops. Ten hops at 99.9% each gives 0.999^10 ≈ 99.0% end to end. At 10,000 requests per second, that "three nines" infrastructure is dropping 100 requests every second. Retries aren't optional at that volume. Naive retries are fatal.
Fix 1: full jitter
The delay stays exponential. The offset becomes random, so each client lands at a different point on the timeline.
export const sleep = (ms: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, ms));
/** Uniform random in [0, min(maxDelay, base * 2^(attempt-1))) */
export function fullJitter(attempt: number, baseDelayMs: number, maxDelayMs: number): number {
const ceiling = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));
return Math.random() * ceiling;
}
export type RetryPolicy = {
maxAttempts: number;
baseDelayMs: number;
maxDelayMs: number;
isRetryable?: (error: unknown) => boolean;
onRetry?: (info: { attempt: number; delayMs: number; error: unknown }) => void;
};
export function isTransientError(error: unknown): boolean {
const e = error as { status?: number; code?: string };
if (typeof e.status === 'number') {
return e.status === 408 || e.status === 429 || e.status >= 500;
}
return e.code === 'ECONNRESET' || e.code === 'ETIMEDOUT' || e.code === 'EAI_AGAIN';
}
export async function withRetry<T>(fn: () => Promise<T>, policy: RetryPolicy): Promise<T> {
const isRetryable = policy.isRetryable ?? isTransientError;
let lastError: unknown;
for (let attempt = 1; attempt <= policy.maxAttempts; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error;
// Never retry a 400. A declined card will be declined forever.
if (attempt === policy.maxAttempts || !isRetryable(error)) throw error;
const delayMs = fullJitter(attempt, policy.baseDelayMs, policy.maxDelayMs);
policy.onRetry?.({ attempt, delayMs, error });
await sleep(delayMs);
}
}
throw lastError;
}
Two details are load-bearing:
-
await sleep(...)inside anasyncfunction does not block the event loop. A busy-wait loop does, and it stalls every other request in the process. -
isRetryableis a hard boundary, not a nicety. Retrying acard_declinedburns quota, delays the user, and can trip fraud heuristics.
What that actually buys you
Same 10,000 clients, retrying once. Here's the whole simulation:
function histogram(delays: number[], bucketMs = 100): { at: number; count: number }[] {
const buckets = new Map<number, number>();
for (const d of delays) {
const b = Math.floor(d / bucketMs) * bucketMs;
buckets.set(b, (buckets.get(b) ?? 0) + 1);
}
return [...buckets.entries()].sort((a, b) => a[0] - b[0]).map(([at, count]) => ({ at, count }));
}
/** Renders on a scale shared by both runs, so the blocks are comparable. */
function render(rows: { at: number; count: number }[], scaleMax: number, width = 50): string {
return rows
.map(({ at, count }) => {
const bar = '█'.repeat(Math.max(1, Math.round((count / scaleMax) * width)));
return ` ${String(at).padStart(6)}ms ${String(count).padStart(7)} ${bar}`;
})
.join('\n');
}
const CLIENTS = 10_000;
const fixed = Array.from({ length: CLIENTS }, () => 1_000); // naive: everybody at 1000ms
const jittered = Array.from({ length: CLIENTS }, () => fullJitter(1, 1_000, 30_000));
console.log(' naive — fixed 1000ms delay');
console.log(render(histogram(fixed), CLIENTS));
console.log('\n full jitter — random in [0, 1000ms)');
console.log(render(histogram(jittered), CLIENTS));
Real output, one run:
naive — fixed 1000ms delay
when clients
1000ms 10000 ██████████████████████████████████████████████████
full jitter — random in [0, 1000ms)
when clients
0ms 1003 █████
100ms 970 █████
200ms 980 █████
300ms 1000 █████
400ms 954 █████
500ms 1048 █████
600ms 1044 █████
700ms 984 █████
800ms 991 █████
900ms 1026 █████
summary
strategy peak/100ms retries
fixed 1000ms 10000 10000
full jitter 1048 10000
9.5x lower peak, same number of retries
Same 10,000 retries, same total load: the peak per 100 ms window drops from 10,000 to 1,048 for four lines of arithmetic. (Your numbers will differ by a few percent; it's Math.random().)
Which jitter? Measured over 20,000 samples at attempt 4 (base 1000 ms, cap 30 s):
| Variant | min | median | max |
|---|---|---|---|
| full | 1 ms | 3,973 ms | 7,999 ms |
| equal | 4,000 ms | 5,988 ms | 8,000 ms |
| decorrelated | 1,001 ms | 12,411 ms | 24,000 ms |
Full jitter has no floor — a few clients retry almost immediately, which feels wrong but is exactly what flattens the curve. If you need a guaranteed minimum gap, use equal jitter (ceiling/2 + random(0, ceiling/2)). Decorrelated jitter derives the next delay from the previous one (min(cap, random(base, previous * 3))), so it drifts well above the exponential ceiling and recovers more slowly — useful when you want successive failures to back off harder than a fixed schedule would.
The simulation, the breaker, and the tests below are in JSTypescriptProgrammingSeries / 30_fintech / 14_chapter / retry-sim. Two commands, no install: node src/sim.ts and node --test "src/*.test.ts".
Fix 2: a circuit breaker that fails fast
Jitter handles failures that resolve in milliseconds. It does nothing for a provider that's down for 40 minutes. In that case every worker exhausts its retries, waits out its backoff, and keeps knocking. You get slow failures, saturated connection pools, and a queue backing up into your own API.
A circuit breaker is a stateful wrapper that stops calling a dependency it has already proven broken.
export class CircuitOpenError extends Error {
constructor(message = 'Circuit is open: downstream is failing fast') {
super(message);
this.name = 'CircuitOpenError';
}
}
export type BreakerState = 'closed' | 'open' | 'half-open';
export type BreakerOptions = {
windowSize: number; // outcomes to consider
failureRateThreshold: number; // 0..1
openDurationMs: number; // cooldown before probing
probesToClose: number; // consecutive successes to close
};
export class CircuitBreaker {
private state: BreakerState = 'closed';
private window: boolean[] = []; // true = success
private openedAt = 0;
private probeSuccesses = 0;
private readonly opts: BreakerOptions;
constructor(opts: BreakerOptions) {
this.opts = opts;
}
get currentState(): BreakerState {
if (this.state === 'open' && Date.now() - this.openedAt >= this.opts.openDurationMs) {
this.state = 'half-open';
this.probeSuccesses = 0;
}
return this.state;
}
async execute<T>(fn: () => Promise<T>, fallback?: () => Promise<T>): Promise<T> {
if (this.currentState === 'open') {
if (fallback) return fallback();
throw new CircuitOpenError();
}
try {
const result = await fn();
this.record(true);
return result;
} catch (error) {
this.record(false);
throw error;
}
}
private record(success: boolean): void {
if (this.state === 'half-open') {
if (!success) return this.trip();
if (++this.probeSuccesses >= this.opts.probesToClose) this.reset();
return;
}
this.window.push(success);
if (this.window.length > this.opts.windowSize) this.window.shift();
// Only judge once the window is full, otherwise the first error trips it.
if (this.window.length === this.opts.windowSize) {
const failureRate = this.window.filter((ok) => !ok).length / this.window.length;
if (failureRate >= this.opts.failureRateThreshold) this.trip();
}
}
private trip(): void {
this.state = 'open';
this.openedAt = Date.now();
this.window = [];
this.probeSuccesses = 0;
}
private reset(): void {
this.state = 'closed';
this.window = [];
this.probeSuccesses = 0;
}
}
The state machine is the part reviewers get wrong:
- Closed — traffic flows. It keeps a rolling window of the last N outcomes and trips when the rate crosses the threshold. A rate, not a count: five failures out of five requests is a dead provider, five out of 10,000 is a Tuesday.
-
Open —
CircuitOpenErrorimmediately, zero sockets opened. That's the entire point: you fail in microseconds instead of after 30 seconds of timeouts. - Half-open — after the cooldown, one request is allowed through as a probe. Success closes the circuit; a single failure re-trips it. Don't let 500 requests be the probe, or the recovering provider gets the herd again.
If you run more than one pod, this in-memory breaker is wrong. Each replica learns about the outage independently, so you get 12 partial breakers instead of one. Share the state in Redis, or use a service mesh that does it for you.
Fix 3: idempotency — the part that makes retries legal
Retries are only safe if the gateway can recognize a duplicate. That's what the idempotency key is for: a client-generated, deterministic identifier that travels with every attempt of the same logical charge.
Note the word deterministic. crypto.randomUUID() inside your retry loop generates a new key on every attempt and defeats the whole mechanism. The key belongs to the charge, generated once at the API boundary — the tested pipeline below asserts that the key never changes between attempts.
import { isTransientError } from './retry.ts';
type IdempotencyRecord<T> =
| { status: 'in-flight'; startedAt: number }
| { status: 'done'; result: T }
| { status: 'unknown'; error: string; startedAt: number };
export class IdempotencyStore<T> {
private readonly records = new Map<string, IdempotencyRecord<T>>();
async run(key: string, fn: () => Promise<T>): Promise<T> {
const existing = this.records.get(key);
if (existing?.status === 'done') return existing.result;
if (existing?.status === 'in-flight') {
throw new Error(`Duplicate request for ${key} is already in flight`);
}
if (existing?.status === 'unknown') {
throw new Error(`Outcome for ${key} is unknown — reconcile before retrying`);
}
this.records.set(key, { status: 'in-flight', startedAt: Date.now() });
try {
const result = await fn();
this.records.set(key, { status: 'done', result });
return result;
} catch (error) {
// The distinction most implementations miss:
if (isTransientError(error)) {
// We do NOT know whether the money moved. Keep the key, block blind retries.
this.records.set(key, {
status: 'unknown',
error: String(error),
startedAt: Date.now(),
});
} else {
// Definitive rejection (400, declined). The charge never happened.
this.records.delete(key);
}
throw error;
}
}
/** Exposed so the tests can inspect the stored outcome. */
peek(key: string): IdempotencyRecord<T> | undefined {
return this.records.get(key);
}
}
That unknown state is the whole reason reconciliation exists as a subsystem. A timeout is not a failure — it's an absence of information. If a charge times out after the gateway debited the account, and you delete the key and let the user retry with a fresh one, you have charged them twice. The ledger's zero-sum invariant doesn't save you: both entries are internally balanced, the customer is just out $49.99.
The correct move is to park the charge in unknown, keep the key, and let a reconciliation job match it against the provider's settlement feed later. When the outcome is unknown, the answer is not another retry.
Putting the three layers together
Order matters: idempotency outside, breaker in the middle, retry inside.
import type { CircuitBreaker } from './breaker.ts';
import type { IdempotencyStore } from './idempotency.ts';
import { withRetry, type RetryPolicy } from './retry.ts';
export type Charge = {
idempotencyKey: string;
amountCents: number;
currency: string;
};
export type ChargeResult = {
reference: string;
amountCents: number;
};
export type ChargeDeps = {
store: IdempotencyStore<ChargeResult>;
breaker: CircuitBreaker;
gateway: (charge: Charge) => Promise<ChargeResult>;
policy: RetryPolicy;
};
export async function chargeCustomer(charge: Charge, deps: ChargeDeps): Promise<ChargeResult> {
return deps.store.run(charge.idempotencyKey, () =>
deps.breaker.execute(() => withRetry(() => deps.gateway(charge), deps.policy)),
);
}
Read it inside out:
-
IdempotencyStoreguarantees one logical charge, however many attempts happen underneath. -
CircuitBreakerrefuses the call entirely if the gateway is known-dead, so retries never even start. -
withRetryabsorbs the transient noise that the breaker hasn't tripped on. -
gatewayreceives the same idempotency key on every attempt.
If the breaker opens mid-flight, pick one behavior explicitly: return 503 and let the user choose another method, or fail over to a secondary acquirer using the same idempotency key, so a duplicate can't slip through the failover path. That's what the fallback argument on execute is for.
It runs, and it's tested
14 tests, no network, no flaky timers — node:test against the exact code above:
✔ stays closed while the failure rate is below threshold
✔ trips open when the rate crosses the threshold
✔ open circuit fails fast without touching the network
✔ half-open probe success closes the circuit
✔ failed probe re-opens immediately
✔ fallback is used instead of throwing
✔ same key executes the charge once and replays the result
✔ a concurrent duplicate is rejected, not replayed
✔ a timeout leaves the key in unknown, blocking blind retries
✔ a definitive rejection frees the key for a legitimate retry
✔ a transient blip is absorbed and every attempt carries the same key
✔ a repeated request is replayed, not charged twice
✔ an unresolved timeout blocks a second attempt with the same key
✔ the breaker stops new keys once the gateway is known-dead
ℹ tests 14
ℹ pass 14
ℹ fail 0
git clone https://github.com/edgarmilvus/JSTypescriptProgrammingSeries
cd JSTypescriptProgrammingSeries/30_fintech/14_chapter/retry-sim
node src/sim.ts # the 10,000-client chart
node --test "src/*.test.ts" # 14/14
Watch the 12-second demo or read the code — same simulation, same numbers.
Five pitfalls that survive code review
1. Config mutated at runtime. Building the retry config as a plain object and tweaking maxRetries per request is a race condition waiting for load. Freeze it at construction.
2. A random key inside the retry loop. Covered above, still the most common production incident in this category.
3. Breaker state in process memory at scale. Fine on one pod, actively harmful on twelve.
4. Retrying non-retryable errors. 429 and 504 are transient. card_declined and invalid_api_key are not. Retrying the second category delays the user and can trip the provider's fraud heuristics.
5. Swallowing errors inside the loop. A catch {} with no re-throw, or a missing await, produces unhandled rejections that kill the worker mid-batch — and now you have charges in an unknown state with nobody to reconcile them.
The takeaway
Retries, breakers, and idempotency keys are usually taught as three separate patterns. In a payment pipeline they're one mechanism with three jobs: jitter makes retries survivable, the breaker makes failure fast, and the key makes the whole thing reversible.
If you take one thing away: a timeout is not a failure, it's an unknown. Treat it as one and the rest of the design falls into place.
The ledger model underneath all of this — bank-grade decimal math, SELECT ... FOR UPDATE concurrency control, webhook replay protection, Plaid/Teller reconciliation, and a capstone core ledger — is what **FinTech Architecture in TypeScript: Precision Math, Double-Entry Ledgers, and High-Reliability Payment Pipelines* covers end to end. Get the book. The nine-volume Enterprise TypeScript Architect bundle includes it at a fraction of the separate price.*
Top comments (0)