When a downstream database or microservice experiences a brief latency spike, your application's retry logic is supposed to mask the glitch. Instead, in most production architectures, retries are the exact mechanism that turns a 5-second hiccup into a 45-minute cascading outage.
As documented in the AWS Builders' Library, retries are inherently selfish: when a client retries a failed call, it asserts that its request is important enough to demand more resources from a dependency that is already struggling to survive.
If your HTTP client uses fixed delays, standard exponential backoff without jitter, or retries across multiple layers of your stack, you haven't built resilienceβyou've built a synchronized load amplifier.
The Problem: Synchronized Herds and Layered Multiplication
Most engineers know that immediate retries (delay = 0) are dangerous, so they configure standard exponential backoff: wait 1s, then 2s, then 4s.
However, as Marc Brooker demonstrated in the AWS Architecture Blog research on Exponential Backoff and Jitter, standard exponential backoff only reduces how often a single client retriesβit does not break synchronization across clients.
When a downstream service stalls for 500ms, 1,000 concurrent requests fail at the exact same millisecond:
- All 1,000 clients sleep for
1000ms(2^0 * 1000). - Exactly one second later, all 1,000 clients wake up in lockstep and hammer the recovering server simultaneously, alongside normal incoming traffic.
- The server collapses again, and the synchronized wave repeats at
2000msand4000ms.
The Common Mistake: Un-Jittered Backoff and Blind Error Retries
Consider a typical production wrapper around an HTTP or RPC call that passes most code reviews without question:
// β Looks resilient, but causes synchronized retry storms under load
export async function fetchOrderDetails(orderId: string): Promise<Order> {
const maxRetries = 3;
const baseDelayMs = 1000;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
const response = await httpClient.get(`/orders/${orderId}`, {
timeout: 2000,
});
return response.data;
} catch (error: any) {
if (attempt === maxRetries) {
throw error;
}
// 1. Retries ALL errors (including 400 Bad Request or 401 Unauthorized)
// 2. Deterministic exponential delay keeps failing clients synchronized
// 3. No global cap on how many retries the process is firing right now
const delay = baseDelayMs * Math.pow(2, attempt);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
throw new Error("Unreachable");
}
The Consequence in Production
When a deployment or database lock causes latency to cross the 2000ms timeout threshold:
- Calling instances keep open sockets waiting, then fire synchronized retry waves at
+1s,+2s, and+4s. - Even if you restart the overloaded downstream pods, the backlog of in-flight retries immediately crushes the new instances before their caches warm upβa classic positive feedback loop described in Google's SRE Book on Addressing Cascading Failures.
The Improved Approach: Full Jitter + Retry Budgets
To make retries safe in production, you need three controls working together:
-
Full Jitter: Randomize the delay between
0and the exponential cap (sleep = random(0, min(cap, base * 2^attempt))) to flatten synchronized spikes into a smooth, constant arrival rate. -
Strict Retryability Filtering: Only retry transient transport errors or explicit
429/503statuses (respectingRetry-Afterheaders when present). - A Local Retry Budget (Token Bucket): Cap retries at a maximum percentage (e.g., 10%) of total outbound traffic. If the dependency is completely down, the budget drains and clients fail fast instead of amplifying the outage.
// β
Production-safe: Full Jitter, Retry Budget (10% cap), and transient filtering
class RetryBudget {
private tokens = 100;
private readonly maxTokens = 100;
private readonly retryCost = 10; // 1 retry costs 10x what a normal request deposits
recordRequest(): void {
this.tokens = Math.min(this.maxTokens, this.tokens + 1);
}
tryAcquireRetry(): boolean {
if (this.tokens >= this.retryCost) {
this.tokens -= this.retryCost;
return true;
}
return false;
}
}
const orderServiceBudget = new RetryBudget();
function isTransientError(status?: number, code?: string): boolean {
if (code === "ECONNRESET" || code === "ETIMEDOUT") return true;
return status === 429 || status === 502 || status === 503 || status === 504;
}
export async function fetchOrderDetailsSafe(orderId: string): Promise<Order> {
const maxRetries = 3;
const baseDelayMs = 200;
const maxCapMs = 5000;
orderServiceBudget.recordRequest();
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
const response = await httpClient.get(`/orders/${orderId}`, {
timeout: 2000,
});
return response.data;
} catch (error: any) {
const status = error.response?.status;
const canRetry =
attempt < maxRetries &&
isTransientError(status, error.code) &&
orderServiceBudget.tryAcquireRetry();
if (!canRetry) {
throw error; // Fail fast if permanent error or retry budget is exhausted
}
// Full Jitter formula: random(0, min(cap, base * 2^attempt))
const expCap = Math.min(maxCapMs, baseDelayMs * Math.pow(2, attempt));
const fullJitterDelay = Math.floor(Math.random() * expCap);
await new Promise((resolve) => setTimeout(resolve, fullJitterDelay));
}
}
throw new Error("Unreachable");
}
Click to compare Full Jitter vs. Equal Jitter vs. Decorrelated Jitter
In AWS's simulation of 100 contending clients, Full Jitter reduced total server calls by more than 50% compared to standard exponential backoff while finishing faster. Here is when to pick each variant:
random(0, min(cap, base * 2^attempt))): Best default choice. Lowest total server work and fastest completion time under high contention.temp / 2 + random(0, temp / 2)): Guarantees a minimum sleep window (temp / 2) so a client never retries at 0ms, while still de-synchronizing the herd.min(cap, random(base, prevSleep * 3))): Stateless-free alternative that scales sleep based on the previous random wait time rather than a strict power of two.
Production Resilience Checklist
Before shipping any client SDK, queue consumer, or API gateway rule, verify your retry policy against these five rules:
- [ ] Retry at a single layer in the stack: Disable automatic retries in inner microservice hops and ORM drivers if your API gateway or top-level caller already retries; let lower layers fail fast.
- [ ] Always apply Full Jitter: Never deploy fixed-interval or pure
2^nbackoff in systems with more than one concurrent caller. - [ ] Enforce a 10% Retry Budget: Use a token bucket so that if more than 10% of requests are failing, clients stop retrying and shed load immediately.
- [ ] Require Idempotency Keys for mutations: Never enable automatic retries on
POSTorPATCHendpoints unless an idempotency key is passed and enforced atomically at the storage layer. - [ ] Propagate deadlines across hops: If an incoming request has a
2000mstimeout and already spent1800msin upstream processing, don't fire a retry that takes another1500msafter the user has already disconnected.
The Takeaway
Retries are only effective when failures are transient, isolated, and rare. The moment a failure becomes systemicβsuch as a saturated connection pool or a degraded database replicaβnaive retries turn your own microservices into an internal botnet. Replace deterministic backoff with Full Jitter, restrict retries to a single architectural layer, and enforce a Retry Budget so your system knows when to stop digging.
Have you ever experienced a self-inflicted retry storm in production, and do you enforce retry budgets at the service mesh/gateway level or inside application code?
Sources & Further Reading
- Exponential Backoff And Jitter β AWS Architecture Blog (Marc Brooker)
- Timeouts, retries, and backoff with jitter β Amazon Builders' Library
- Addressing Cascading Failures β Google Site Reliability Engineering (SRE) Book
Connect with Me
If you found this article helpful, let's connect!
- π» GitHub: johnnylemonny
Top comments (24)
Really good article πΈ
The point about retries multiplying across layers stood out to me. Itβs easy to think βmore retries = more resilience,β but under failure it can become the opposite very quickly.
The retry budget idea was especially interesting! π
Thanks! Thatβs exactly the intuitive trap - retries feel like free safety nets until a real incident turns them into a synchronized load amplifier. Glad the retry budget concept resonated with you!
Thanks John. And thanks for actually answering people in your own thread, that's rarer than it should be.
One thing I left out above that your post pushed me on: the counter is only half the fix, because the counter gets muted too. Somebody adds "succeeded on attempt >1" to a dashboard, it sits at 3%, nobody knows what 3% means, and six months later it reads 11% and still nobody knows. A number with no baseline is just decoration. It needs a memory: recorded at deploy time, compared to last week, and owned by whoever owns the dependency. Then "non-zero in steady state" is a claim you can actually argue with.
So, since you clearly collect these: what's the retry bug in your own stack that the counter stayed flat on? The one where the metric looked fine and the failure was real anyway. Those are the cases I'm after.
The "baseline decoration" problem is so true - if nobody owns the baseline, a metric drifting from 3% to 11% just becomes accepted background noise until a major outage hits.
As for the retry bug where the counter stayed completely flat: our HTTP-level retry wrapper was monitoring standard transport failures and 5xx/429 status codes. Everything looked 100% green and the retry counter was dead zero.
The catch? A downstream service was returning HTTP 200 OK with logical error payloads (a classic GraphQL/RPC pattern). An internal SDK client was inspecting the response body, noticing the partial failure, and executing its own tight, un-jittered, un-metered retry loop before returning to our caller.
Our dashboards showed zero retries and perfect uptime, while the downstream database was quietly being hammered into the ground. That incident was a harsh reminder that HTTP-level metrics are pure theater if the retry loop lives a layer deeper in the response parser.
Your case is the nastier version of the one I described. A 200 with a logical error inside it means every layer above the parser is measuring the transport, not the work. So "green" is true at every layer while the actual operation keeps failing.
The part I'd underline: the retry loop was inside an SDK client. It never appeared in your own retry code and never reached your wrapper's counters, so no amount of dashboards would have caught it. Counting only helps once you know which layer owns the attempt, and the only layer that can see the semantics is the parser. Counting attempts there and treating anything deeper as a separate named budget is usually the cheapest fix.
A detector that wouldn't have needed you to know about the SDK at all: watch the ratio of downstream queries/statements to application requests. If that ratio moves and nobody shipped a new code path, something underneath is retrying. Green HTTP plus a rising query-per-request ratio is the fingerprint of a hidden retry loop.
And partial failure in the body is the case where "retry the whole call" is just wrong. If the parser can see which parts failed, the retry belongs to those parts, not to the operation. Otherwise a partial success spends its budget re-doing work it already committed, which is how you get duplicate writes out of a read path.
The version of this I keep hitting isn't the storm, it's what the retry hides afterwards.
A wrapper that succeeds on attempt 2 turns a 100% reproducible failure into an intermittent one, and intermittent failures never get fixed. The logged request is a 200. The user saw nothing. The dashboard is green. Meanwhile the real fault (a stale connection, one bad pool entry, a path that only breaks on a native Windows separator) stays in place, quietly costing one timeout per request until the day it stops being recoverable.
Two things I'd add to the budget:
The single-layer, jittered, budgeted retry is fine. The one that never tells anyone it fired is the one that hurt me.
This is gold, Nathan. That first paragraph cuts right to the heart of the problem: a retry that succeeds on attempt 2 turns a 100% reproducible bug into an intermittent ghost.
Your two additions nail the exact operational blind spots:
The quietest retries are always the most dangerous ones. Thanks for adding such practical operational depth to the discussion! π
Yeah. The orphaned state is the part that outlives the incident. A retry that lands after a timeout has already written has nothing left to fail on, so it leaves a row nobody is looking for while the queue reports success.
The cheap fix is the one robertpaschal put in the thread: make the idempotency key a unique constraint and let the losing insert fail loudly. Then a duplicate retry is a counted conflict, not a ghost you meet three weeks later during a reconciliation nobody scheduled. A counter tells you the duplicates happen; the constraint is the only thing that stops them existing.
If you ever catch one of these in your own service, tell me what it did. I'll dig into it, no charge, and you can tell me if my read is wrong.
That's the cleanest mental model for it: a counter measures the problem, but the database constraint is the only thing that actually prevents it.
Relying on application-level checks (
if (!exists) create()) under retry storms is always a race condition waiting to happen. Pushing the constraint down to the storage engine and letting duplicate retries fail loudly turns an invisible state-corruption bug into a predictable conflict you can handle cleanly.Appreciate the offer, Nathan!
The layered multiplication point is the one most retry audits miss: jitter fixes synchronization at one layer, but if the client, the gateway, and the downstream service each retry three times, a single user request still becomes twenty-seven attempts against a dependency that is already struggling. That is why the retry budget matters as much as the backoff curve β jitter spreads the load out, but only a budget caps its total size. Budgets also force the honest conversation about which requests are actually worth retrying, instead of letting every layer assert, selfishly and independently, that its request is the important one.
"Jitter spreads the load out, but only a budget caps its total size" is the perfect summary.
Jitter solves the phase synchronization problem, not the volume amplification problem. And your point about the "selfish layer" mindset is spot-on. Every tier assumes its request is mission-critical, completely blind to the fact that downstream capacity can't survive 3Β³ attempts. A centralized retry budget is the only mechanism that forces systemic discipline over local optimism.
The layered multiplication is the sneaky one. The SDK retries, the HTTP client retries, then the queue redelivers, and nobody owns the total. One rule fixes most of it: exactly one layer owns retries and everything else fails fast. Honestly, retry budgets with full jitter should be the default in every client library, not something you learn from an outage.
Exactly. If everyone in the call chain is responsible for retrying, nobody is responsible for the aggregate load.
"Exactly one layer owns retries, everything else fails fast" should be standard architectural doctrine. The naive
retries: 3default hardcoded into so many SDKs and HTTP clients without budgets or circuit breakers is easily one of the most widespread footguns in backend systems today. Thanks for reading! πThe prose says to respect Retry-After, but the safe wrapper still chooses its delay only from the local jitter window. With a 429 asking for a longer pause, that could send the next attempt before the server's requested wait has elapsed. I'd carry that wait into the scheduling decision and stop if it exceeds the caller's remaining deadline. How would you combine the server's minimum wait with jitter without collecting every throttled client at the same wake-up time?
Great catch on Retry-After. To respect the server's minimum pause without creating a synchronized wake-up wave, you can apply additive jitter on top of the requested wait: delay = retryAfter + random(0, jitterWindow), or scale it proportionally like retryAfter * (1 + random(0, 0.2)). That ensures no client retries prematurely while still spreading out the wake-up times.
And spot on about deadlines: if now + delay exceeds the caller's remaining budget, fail fast immediately and drop the work instead of waiting on a response nobody will ever read.
The 4^4 example is the clearest way I have seen to explain why retries at every layer are dangerous. Each layer looks sensible on its own, and together they turn one slow dependency into 256 requests.
Mudassir's case of an SDK default retry sitting under a gateway retry is the one that catches teams, because nobody chose it. It came with the library. On client systems we now list every retry in the request path during review, including library defaults, and decide which single layer owns retries. The others fail fast and pass the error up.
Nathan's point deserves more attention than it usually gets. A retry that quietly succeeds turns a reproducible failure into an intermittent one, so we log every retried call with the original error even when the final result is a success. That log is often the first sign a dependency is getting worse.
Do you prefer the retry budget at the gateway, as Ali suggested, or per client?
Definitely at the gateway (or service mesh layer) for anything crossing network boundaries or handling external ingress.
Per-client budgets fall apart the moment autoscaling kicks in. If an ongoing slowdown triggers an HPA scale-up from 10 to 100 pods, a 10% per-client budget suddenly multiplies the total retry volume by 10x against a dependency that is already failing to recover.
Enforcing the token bucket centrally at the gateway (or via Envoy cluster-level retry budgets) solves two major problems:
Also, listing every retry in the call path during PR review, especially library defaults - is fantastic practice. Making those implicit SDK retries explicit before merging prevents most of these storms before they ever reach production.
the multiplying retries per hop math is the part everyone nods at in review and then ships anyway. most of the multiplication we found in our own stack wasn't in app code, it was an sdk default retry sitting under a gateway retry that nobody remembered enabling.
what fixed it was one layer owning retries and everything below it getting max attempts of 1, plus a small token bucket so retries stop once failures cross a threshold.
do you set the budget per client instance or per downstream service? per instance felt too leaky for us once we autoscaled.
Per downstream service (ideally enforced at the gateway or service mesh layer) is definitely the way to go once autoscaling enters the picture.
As you experienced, per-instance budgets scale linearly with replica count. If an ongoing slowdown triggers an HPA scale-up from 5 to 50 pods, a 10% retry budget per instance suddenly floods the downstream dependency with 10x the absolute retry volume, right when it's already struggling to recover. Enforcing the token bucket centrally per target service (like Envoy cluster-level retry budgets) keeps total retry amplification capped regardless of how wide your app fleet scales.
And spot-on about SDK defaults. Hidden
maxRetries: 3configurations inside client libraries sitting beneath an API gateway or reverse proxy are probably the most common source of accidental retry multipliers in production today.That 256 requests from one click number stayed with me.
We keep the retry budget in the app code right now.
Reading this, the gateway feels like the better place for it.
One thing in the opening card: the two numbers count differently. 4 attempts per hop over 4 hops is 4^4 = 256, and 3 attempts per layer over 5 layers is 3^5 = 243, so the same chain lands in a different place depending on whether you count calls or services. The bound is attempts^(number of retrying callers), so the lever is how many layers retry, which is why your single-layer rule beats tuning the retry count.
On the budget class: +1 per request and a retry cost of 10 gives a long-run cap of 10%, but the bucket starts full at 100, so every fresh process can spend 10 retries at once before the cap bites. Restart 200 pods during an incident and that is 2,000 retries in the first seconds. Starting the bucket at 0 avoids it. Do you keep the budget per process or per client instance?
Great catches. You nailed the math nuance - the exponent (number of retrying layers) is always the dominant lever, which is exactly why stripping retries out of inner hops matters so much more than tuning the retry count itself.
You're also right about the cold-start burst with tokens = 100. During a rolling restart or autoscaling event mid-incident, pre-filled buckets across hundreds of pods can easily trigger a mini retry storm. Initializing at 0 (or at most 10 to allow just a single retry for cold-start network blips) is definitely safer.
Regarding scoping: I keep the budget per downstream dependency at the process level (a singleton per upstream service client). Scoping it globally per process causes a single failing dependency to starve retries for healthy ones, while scoping it per non-singleton client instance makes it too easy to accidentally instantiate multiple budgets and multiply the cap.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.