Headline: Naive retries turn a slow dependency into a self-inflicted outage, because every caller retrying on the same schedule multiplies load on a service that is already failing. The combination that works is a hard per-attempt timeout, exponential backoff with full jitter under a total retry budget, an
Idempotency-Keyheader on every non-idempotent write, and a circuit breaker that stops calling a dependency that is already down.
A background worker I maintain calls a third-party payments API. During a partial outage on their side, requests started taking 30 seconds and then failing. My retry wrapper did exactly what I had told it to do: try again immediately, three times. Throughput collapsed, the queue backed up, and when the dependency started recovering my workers were still hammering it in lockstep. The retries were not helping the recovery; they were part of what was preventing it. These are my notes from rebuilding that call path.
Key takeaways
- Retry amplification is real arithmetic. Three retries per request means a dependency at its breaking point receives up to four times its normal load at the exact moment it has the least capacity.
- Only retry failures that are safe to repeat. Connection errors, HTTP 502/503/504, and HTTP 429 are retryable; HTTP 400, 401, 403, and 422 are not, because repeating a request the server understood and rejected will never produce a different answer.
-
An idempotency key is a client-generated unique string sent in the
Idempotency-Keyheader so the server can recognise a retry of a write it already processed and return the original response instead of performing the work twice. -
Full jitter beats fixed backoff. Sleeping a random duration in
[0, base * 2^attempt]spreads a fleet of retrying clients across the window instead of synchronising them into repeated thundering herds. - A circuit breaker fails fast on a dependency that is already down, converting a 30-second timeout into an immediate error and giving the dependency room to recover. Per-dependency, never global.
Why do retries make an outage worse instead of better?
Retries multiply load precisely when a dependency has the least capacity to absorb it. A service that normally handles 1,000 requests per second, configured with three retries across every caller, can see 4,000 requests per second the moment it starts failing. The failure causes the load, and the load sustains the failure. This feedback loop is called retry amplification, and it is the single most common way a partial degradation becomes a total outage.
Synchronisation makes it worse. If every worker retries after exactly one second, then two, then four, the whole fleet arrives at the dependency at the same three instants. The recovering service gets a spike, tips over again, and resets everyone's retry clock to the same value — the classic thundering herd. Fixed backoff does not spread load; it schedules a collision.
The third cost is the one I underestimated: retries consume my own capacity too. A worker sitting in a retry sleep is a worker not processing the queue. With no per-attempt timeout, four attempts at 30 seconds each means a single message occupies a worker for two minutes. My throughput fell before the dependency ever fully failed.
Which failures are actually safe to retry?
Retry only when the request either never reached the server or definitely failed without side effects. In practice that means classifying by status code and error type, not by wrapping everything in a generic try/catch.
Retryable: TCP connection errors (ECONNRESET, ECONNREFUSED, ETIMEDOUT), DNS failures, HTTP 502, 503, and 504, and HTTP 429 with the delay taken from the Retry-After header. These indicate the server was unavailable or explicitly asked you to wait.
Not retryable: HTTP 400, 401, 403, 404, and 422. The server received the request, understood it, and rejected it. Retrying a malformed payload produces the same rejection with extra load, and retrying a 401 can trip an authentication provider's own rate limiting.
The genuinely hard case is HTTP 500 and the request timeout. Both are ambiguous: the write may have committed before the response was lost. This is exactly the situation idempotency keys exist to resolve. Without a key, retrying an ambiguous POST /charges risks a duplicate charge, and not retrying it risks losing a legitimate payment. With a key, the ambiguity disappears and the retry is safe.
What is an idempotency key, and how do I implement one server-side?
An idempotency key is a client-generated unique string, typically a UUID v4, sent in the Idempotency-Key HTTP header so that repeating a request produces the original result rather than a second side effect. Stripe popularised the header, and it is now the de facto convention for payment and provisioning APIs.
The critical detail is that the key must be generated once per logical operation, not once per HTTP attempt. Generating a fresh UUID inside the retry loop defeats the entire mechanism, because each attempt then looks like a brand-new operation to the server. I generate the key when the job is enqueued and store it on the job record, so every retry — including a retry after a process crash and restart — reuses the same value.
async function chargeCustomer(job: ChargeJob) {
return callWithRetry(() =>
fetch('https://api.example.com/v1/charges', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
// Generated once when the job row was created, never inside the loop.
'Idempotency-Key': job.idempotencyKey,
},
body: JSON.stringify({ amount: job.amount, customer: job.customerId }),
signal: AbortSignal.timeout(10_000),
}),
);
}
On the receiving side — and I own APIs that other people retry against — the implementation is a table with a unique constraint doing the concurrency control, not application code checking before inserting.
CREATE TABLE idempotency_records (
key text PRIMARY KEY,
request_hash text NOT NULL,
status text NOT NULL, -- 'in_progress' | 'completed'
response_status int,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now()
);
The handler inserts the key with status in_progress inside the same transaction as the business write. A unique-violation error on insert means another attempt is already handling this key: if the stored record is completed, replay response_status and response_body verbatim; if it is still in_progress, return HTTP 409 so the client backs off rather than racing. Relying on a SELECT followed by an INSERT reintroduces the race the key was meant to close, because two concurrent retries can both read "not found".
Store a hash of the request body alongside the key. If the same key arrives with a different payload, that is a client bug, and returning HTTP 422 surfaces it immediately instead of silently replaying an unrelated response. Expire records after a fixed window — 24 hours is a common choice — because keys are a retry-safety mechanism, not permanent storage.
How should I configure backoff, jitter, and timeouts?
Use exponential backoff with full jitter, cap the delay, and bound the whole operation with a deadline. Full jitter means sleeping a uniformly random duration in [0, min(cap, base * 2^attempt)] rather than the deterministic value. The randomness is the point: it decorrelates a fleet of clients so the recovering dependency sees a smooth arrival rate instead of synchronised spikes.
const RETRYABLE_STATUS = new Set([429, 502, 503, 504]);
async function callWithRetry(
fn: () => Promise<Response>,
{ retries = 3, base = 200, cap = 10_000 } = {},
): Promise<Response> {
for (let attempt = 0; ; attempt++) {
const res = await fn().catch((err) => err as Error);
const isRetryable =
res instanceof Error
? res.name !== 'AbortError'
: RETRYABLE_STATUS.has(res.status);
if (!isRetryable) {
if (res instanceof Error) throw res;
return res;
}
if (attempt >= retries) {
if (res instanceof Error) throw res;
return res;
}
const retryAfter =
res instanceof Response ? Number(res.headers.get('retry-after')) * 1000 : NaN;
const ceiling = Math.min(cap, base * 2 ** attempt);
const delay = Number.isFinite(retryAfter) && retryAfter > 0
? retryAfter // honour the server's instruction
: Math.random() * ceiling; // full jitter
await new Promise((r) => setTimeout(r, delay));
}
}
Three configuration rules I now treat as non-negotiable. First, every attempt gets its own timeout: AbortSignal.timeout(10_000) is one line and it is what stops a hung socket from holding a worker indefinitely. Second, honour Retry-After when the server sends it, because a 429 with an explicit delay is the dependency telling you exactly what it needs. Third, keep retries shallow — two or three — and never retry inside a layer that is itself being retried, or the counts multiply: three retries at each of three layers is 27 requests from one logical call.
Set a deadline for the whole operation, not just a retry count. If the caller is an HTTP request that will time out in 30 seconds anyway, a retry schedule that can run for 60 seconds is pure waste: the client has already given up and the work is discarded.
When does a circuit breaker help, and when is it overkill?
A circuit breaker is a wrapper that tracks recent failures for one dependency and, past a threshold, short-circuits subsequent calls by failing immediately instead of attempting them. It has three states: closed (calls pass through), open (calls fail instantly for a cooldown period), and half-open (a small number of trial calls decide whether to close again or re-open).
The breaker solves a problem retries cannot. When a dependency is fully down, the correct behaviour is to stop calling it — both to let it recover and to stop burning your own latency budget on calls that are certain to fail. A 10-second timeout becomes an instant error, which means your API returns a fast, honest degraded response instead of queueing requests behind dead connections.
Scope the breaker per dependency, and ideally per endpoint group. One global breaker means a broken analytics endpoint can block your checkout calls. Choose thresholds from a failure rate over a rolling window rather than a raw count, so low-traffic services do not trip on two unlucky requests. And make sure the fallback path is genuinely useful: serving stale cached data or queueing the work for later is worth building a breaker for, while an error page you would have shown anyway is not.
Overkill looks like this: a single-instance internal service with predictable traffic, or a dependency whose failures are already rare and fast. A breaker adds state, configuration, and a new way to be wrong — I have seen a misconfigured breaker stay open long after the dependency recovered. Timeouts and bounded retries come first; add the breaker when a dependency has actually demonstrated that it fails hard.
Which mechanism fixes which failure?
These four mechanisms are not alternatives. Each addresses a different failure, and using one without the others leaves a hole.
| Mechanism | Problem it solves | What it does not solve |
|---|---|---|
| Per-attempt timeout | A hung request holding a worker or connection forever | Nothing about whether the work succeeded |
| Bounded retry with jitter | Transient failures; synchronised client herds | Duplicate side effects on ambiguous writes |
| Idempotency key | Duplicate writes when a retry follows a lost response | Load amplification; a dependency that is fully down |
| Circuit breaker | Wasted calls and latency against a dependency that is down | Individual transient blips; correctness of a single write |
Ordered by return on effort: add per-attempt timeouts first, because they are one line and they bound your worst case. Add idempotency keys to every non-idempotent write next, because without them you cannot retry ambiguous failures safely at all. Fix the retry schedule with jitter and a budget third. Add breakers last, on the dependencies that have earned one.
FAQ
Q: Can I use a hash of the request body as the idempotency key?
A: No, because two legitimate identical requests — a customer intentionally buying the same item twice — would collide and the second one would be silently swallowed. Generate a UUID per logical operation and store the body hash separately to detect key reuse with a different payload.
Q: Is it safe to retry a GET request without an idempotency key?
A: Yes. GET, HEAD, PUT, and DELETE are defined as idempotent in the HTTP specification, so repeating them produces the same server state. POST and PATCH are the methods that need a key, and a PUT implemented with side effects such as counters needs one in practice too.
Q: How many retries should I configure?
A: Two or three at a single layer. More attempts rarely convert a failure into a success, and retries at multiple nested layers multiply: three retries in an HTTP client wrapped by three retries in a job queue is up to sixteen requests from one logical call.
Q: Does full jitter make retries slower on average?
A: Slightly, for a single client, since the expected delay is half the ceiling rather than the full value. Across a fleet it is faster overall, because decorrelated arrivals let the dependency recover instead of being knocked over by each synchronised wave.
Q: Where should the idempotency key live for a queued background job?
A: On the job record, written when the job is enqueued. Generating it at call time means a crashed and re-delivered job produces a new key, which is exactly the duplicate-write scenario the key was supposed to prevent.
Originally published on devya.dev. Also on eng-ahmed.com. Built by Devya Solutions.
Top comments (1)
The 'generate the key when the job is enqueued, not inside the retry loop' is the detail that most tutorials miss. I've seen production bugs where someone put crypto.randomUUID() inside the fetch call and couldn't figure out why they were double-charging customers. This article should be required reading.