Every Node.js developer has written this at some point:
async function fetchWithRetry(url, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
return await fetch(url);
} catch (err) {
if (i === retries - 1) throw err;
}
}
}
It looks reasonable. It also has three separate bugs that only show up under real load: it retries instantly with no delay, it retries every failing request at the exact same rate, and it keeps hammering a dependency that is already down. In a system with more than a handful of concurrent callers, this pattern doesn't add resilience — it adds a self-inflicted denial-of-service attack on your own backend the moment that backend has a bad five minutes.
This article builds retry logic that actually holds up in production: exponential backoff with jitter, a circuit breaker that stops retrying once a dependency is clearly down, and a retry budget that caps the damage a retry storm can do. All examples are plain TypeScript with no dependencies, so you can drop them into any Node service.
Why naive retries make outages worse
Imagine a database that hiccups for 10 seconds under load. A thousand concurrent requests each retry three times with no delay. Instead of 1,000 requests hitting the database, it now sees up to 4,000 in the same window — right as it's trying to recover. This is a retry storm, and it's one of the most common ways a brief blip turns into a multi-minute outage.
Two further problems compound it:
- Synchronized retries. If every client uses the same fixed delay (say, exactly 1 second), all the retries land in the same instant, recreating the thundering herd on every attempt.
- No circuit breaking. If the dependency is fully down, not just slow, retrying doesn't help — it just adds load to a system that's already failing, and keeps your own service tied up waiting on timeouts instead of failing fast.
The fix is three independent mechanisms, each solving one of these problems.
1. Exponential backoff with full jitter
The delay between retries should grow exponentially, and it should be randomized so concurrent clients don't retry in lockstep. AWS's architecture blog popularized the term "full jitter" for the approach that performs best under load: pick a random delay between 0 and the exponential cap, rather than adding a small random offset to a fixed delay.
function backoffDelay(attempt: number, baseMs = 200, capMs = 10_000): number {
const exp = Math.min(capMs, baseMs * 2 ** attempt);
return Math.floor(Math.random() * exp);
}
async function sleep(ms: number) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
With baseMs = 200, attempt 0 waits up to 200ms, attempt 3 waits up to 1.6s, attempt 6 waits up to 10s (the cap). Because each client independently samples a random value in that range, retries spread out across the window instead of clustering.
2. A circuit breaker that actually resets
Backoff slows down retries against a struggling dependency, but it doesn't stop them against a dependency that's completely down. For that you need a circuit breaker: after enough consecutive failures, stop calling the dependency entirely for a cooldown period, then let a single "probe" request through to check if it has recovered.
type State = "closed" | "open" | "half-open";
class CircuitBreaker {
private state: State = "closed";
private failures = 0;
private nextAttempt = 0;
constructor(
private readonly failureThreshold = 5,
private readonly cooldownMs = 30_000
) {}
async exec<T>(fn: () => Promise<T>): Promise<T> {
if (this.state === "open") {
if (Date.now() < this.nextAttempt) {
throw new Error("circuit open: dependency assumed down");
}
this.state = "half-open";
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (err) {
this.onFailure();
throw err;
}
}
private onSuccess() {
this.failures = 0;
this.state = "closed";
}
private onFailure() {
this.failures++;
if (this.state === "half-open" || this.failures >= this.failureThreshold) {
this.state = "open";
this.nextAttempt = Date.now() + this.cooldownMs;
}
}
}
The key detail people get wrong: a failure while half-open must immediately reopen the circuit rather than incrementing a counter that has to climb back up to the threshold. Otherwise a dependency that's flapping (up for one request, down for the next) never actually gets protected — you keep letting probes through and counting failures from zero each time.
One circuit breaker instance per downstream dependency, not one global instance — a slow payment provider shouldn't trip the breaker for your unrelated email service.
3. Retry budgets: capping the blast radius
Backoff and circuit breakers protect a single dependency from a single client's retries. A retry budget protects the system by capping the fraction of total traffic that's allowed to be retries, independent of how many individual clients are backing off correctly. This matters because backoff alone doesn't help if you have 50,000 concurrent clients — even spread-out retries from that many callers can still overwhelm a recovering service.
class RetryBudget {
private tokens: number;
constructor(private readonly capacity = 100, private readonly refillPerSec = 10) {
this.tokens = capacity;
setInterval(() => {
this.tokens = Math.min(capacity, this.tokens + refillPerSec);
}, 1000);
}
tryConsume(): boolean {
if (this.tokens < 1) return false;
this.tokens -= 1;
return true;
}
}
This is a simple token bucket: every retry attempt (not the original request) consumes a token, and tokens refill at a fixed rate. When the bucket is empty, retries are skipped and the call fails fast instead of adding load. A budget of "10% of request volume may be retries" is a common starting point — tune capacity and refillPerSec against your actual request rate.
Putting it together
const breaker = new CircuitBreaker();
const budget = new RetryBudget();
async function resilientFetch(
url: string,
maxAttempts = 4
): Promise<Response> {
let lastErr: unknown;
for (let attempt = 0; attempt < maxAttempts; attempt++) {
if (attempt > 0 && !budget.tryConsume()) {
throw new Error("retry budget exhausted, failing fast");
}
if (attempt > 0) {
await sleep(backoffDelay(attempt));
}
try {
return await breaker.exec(() => fetch(url));
} catch (err) {
lastErr = err;
if (!isRetryable(err)) throw err;
}
}
throw lastErr;
}
function isRetryable(err: unknown): boolean {
// Retry network errors and 5xx/429. Never retry 4xx other than 429 —
// a malformed request will fail identically on every attempt.
if (err instanceof Response) {
return err.status === 429 || err.status >= 500;
}
return true;
}
Note that the first attempt never touches the budget or sleeps — only retries do. This keeps the happy path at zero added latency.
The idempotency trap
All of the above assumes the request is safe to run twice. Retrying a GET is free. Retrying a POST /charge-card without protection can double-charge a customer. Before wiring retry logic onto any write operation, make sure the endpoint either is naturally idempotent (a PUT that sets absolute state) or accepts an idempotency key that lets the server recognize and discard a duplicate:
await fetch("/charges", {
method: "POST",
headers: { "Idempotency-Key": requestId },
body: JSON.stringify(payload),
});
Generate requestId once per logical operation, outside the retry loop, and reuse it across attempts — a new key per attempt defeats the purpose entirely.
Testing it without waiting real minutes
Because the backoff cap can reach 10+ seconds, tests need fake timers rather than real setTimeout delays:
import { describe, it, expect, vi } from "vitest";
it("opens the circuit after threshold failures", async () => {
vi.useFakeTimers();
const breaker = new CircuitBreaker(3, 5000);
const failing = () => Promise.reject(new Error("down"));
for (let i = 0; i < 3; i++) {
await expect(breaker.exec(failing)).rejects.toThrow("down");
}
await expect(breaker.exec(failing)).rejects.toThrow("circuit open");
vi.advanceTimersByTime(5001);
await expect(breaker.exec(() => Promise.resolve("ok"))).resolves.toBe("ok");
});
vi.advanceTimersByTime lets you assert the cooldown-then-recovery behavior in milliseconds of real test time instead of actually sleeping 5 seconds per test case.
Common mistakes to avoid
- Retrying 4xx errors. A 400 or 404 will fail identically forever; retrying just adds latency and load for no benefit. Only retry 429 and 5xx.
- Fixed-delay retries. Even a "random-ish" delay without the full-jitter approach tends to re-synchronize under sustained load. Use the algorithm, not an approximation of it.
- One circuit breaker for everything. A shared breaker means an unrelated slow dependency trips protection for a healthy one. Scope breakers per downstream target.
-
No cap on total retry time. A request that retries 8 times with exponential backoff can take over a minute to finally fail — make sure
maxAttemptsand the backoff cap keep worst-case latency inside whatever SLA the caller expects. - Retrying non-idempotent writes without an idempotency key. This is the one that turns a resilience feature into a data-integrity bug.
None of these three mechanisms — backoff, circuit breaking, retry budgets — is a replacement for the others. Backoff spreads out load from a single client, circuit breakers stop calling a dependency that's clearly down, and retry budgets cap the aggregate cost across every client combined. Used together, they turn "retry logic" from a liability that amplifies outages into what it's supposed to be: a way to absorb the brief, ordinary blips that distributed systems produce constantly, without making the bad ones worse.
Top comments (0)