When a downstream database or microservice experiences a brief latency spike, your application's retry logic is supposed to mask the glitch. Instea...
For further actions, you may consider blocking this person and/or reporting abuse
Really good article πΈ
The point about retries multiplying across layers stood out to me. Itβs easy to think βmore retries = more resilience,β but under failure it can become the opposite very quickly.
The retry budget idea was especially interesting! π
Thanks! Thatβs exactly the intuitive trap - retries feel like free safety nets until a real incident turns them into a synchronized load amplifier. Glad the retry budget concept resonated with you!
Thanks John. And thanks for actually answering people in your own thread, that's rarer than it should be.
One thing I left out above that your post pushed me on: the counter is only half the fix, because the counter gets muted too. Somebody adds "succeeded on attempt >1" to a dashboard, it sits at 3%, nobody knows what 3% means, and six months later it reads 11% and still nobody knows. A number with no baseline is just decoration. It needs a memory: recorded at deploy time, compared to last week, and owned by whoever owns the dependency. Then "non-zero in steady state" is a claim you can actually argue with.
So, since you clearly collect these: what's the retry bug in your own stack that the counter stayed flat on? The one where the metric looked fine and the failure was real anyway. Those are the cases I'm after.
The "baseline decoration" problem is so true - if nobody owns the baseline, a metric drifting from 3% to 11% just becomes accepted background noise until a major outage hits.
As for the retry bug where the counter stayed completely flat: our HTTP-level retry wrapper was monitoring standard transport failures and 5xx/429 status codes. Everything looked 100% green and the retry counter was dead zero.
The catch? A downstream service was returning HTTP 200 OK with logical error payloads (a classic GraphQL/RPC pattern). An internal SDK client was inspecting the response body, noticing the partial failure, and executing its own tight, un-jittered, un-metered retry loop before returning to our caller.
Our dashboards showed zero retries and perfect uptime, while the downstream database was quietly being hammered into the ground. That incident was a harsh reminder that HTTP-level metrics are pure theater if the retry loop lives a layer deeper in the response parser.
Your case is the nastier version of the one I described. A 200 with a logical error inside it means every layer above the parser is measuring the transport, not the work. So "green" is true at every layer while the actual operation keeps failing.
The part I'd underline: the retry loop was inside an SDK client. It never appeared in your own retry code and never reached your wrapper's counters, so no amount of dashboards would have caught it. Counting only helps once you know which layer owns the attempt, and the only layer that can see the semantics is the parser. Counting attempts there and treating anything deeper as a separate named budget is usually the cheapest fix.
A detector that wouldn't have needed you to know about the SDK at all: watch the ratio of downstream queries/statements to application requests. If that ratio moves and nobody shipped a new code path, something underneath is retrying. Green HTTP plus a rising query-per-request ratio is the fingerprint of a hidden retry loop.
And partial failure in the body is the case where "retry the whole call" is just wrong. If the parser can see which parts failed, the retry belongs to those parts, not to the operation. Otherwise a partial success spends its budget re-doing work it already committed, which is how you get duplicate writes out of a read path.
The version of this I keep hitting isn't the storm, it's what the retry hides afterwards.
A wrapper that succeeds on attempt 2 turns a 100% reproducible failure into an intermittent one, and intermittent failures never get fixed. The logged request is a 200. The user saw nothing. The dashboard is green. Meanwhile the real fault (a stale connection, one bad pool entry, a path that only breaks on a native Windows separator) stays in place, quietly costing one timeout per request until the day it stops being recoverable.
Two things I'd add to the budget:
The single-layer, jittered, budgeted retry is fine. The one that never tells anyone it fired is the one that hurt me.
This is gold, Nathan. That first paragraph cuts right to the heart of the problem: a retry that succeeds on attempt 2 turns a 100% reproducible bug into an intermittent ghost.
Your two additions nail the exact operational blind spots:
The quietest retries are always the most dangerous ones. Thanks for adding such practical operational depth to the discussion! π
Yeah. The orphaned state is the part that outlives the incident. A retry that lands after a timeout has already written has nothing left to fail on, so it leaves a row nobody is looking for while the queue reports success.
The cheap fix is the one robertpaschal put in the thread: make the idempotency key a unique constraint and let the losing insert fail loudly. Then a duplicate retry is a counted conflict, not a ghost you meet three weeks later during a reconciliation nobody scheduled. A counter tells you the duplicates happen; the constraint is the only thing that stops them existing.
If you ever catch one of these in your own service, tell me what it did. I'll dig into it, no charge, and you can tell me if my read is wrong.
That's the cleanest mental model for it: a counter measures the problem, but the database constraint is the only thing that actually prevents it.
Relying on application-level checks (
if (!exists) create()) under retry storms is always a race condition waiting to happen. Pushing the constraint down to the storage engine and letting duplicate retries fail loudly turns an invisible state-corruption bug into a predictable conflict you can handle cleanly.Appreciate the offer, Nathan!
The layered multiplication point is the one most retry audits miss: jitter fixes synchronization at one layer, but if the client, the gateway, and the downstream service each retry three times, a single user request still becomes twenty-seven attempts against a dependency that is already struggling. That is why the retry budget matters as much as the backoff curve β jitter spreads the load out, but only a budget caps its total size. Budgets also force the honest conversation about which requests are actually worth retrying, instead of letting every layer assert, selfishly and independently, that its request is the important one.
"Jitter spreads the load out, but only a budget caps its total size" is the perfect summary.
Jitter solves the phase synchronization problem, not the volume amplification problem. And your point about the "selfish layer" mindset is spot-on. Every tier assumes its request is mission-critical, completely blind to the fact that downstream capacity can't survive 3Β³ attempts. A centralized retry budget is the only mechanism that forces systemic discipline over local optimism.
The layered multiplication is the sneaky one. The SDK retries, the HTTP client retries, then the queue redelivers, and nobody owns the total. One rule fixes most of it: exactly one layer owns retries and everything else fails fast. Honestly, retry budgets with full jitter should be the default in every client library, not something you learn from an outage.
Exactly. If everyone in the call chain is responsible for retrying, nobody is responsible for the aggregate load.
"Exactly one layer owns retries, everything else fails fast" should be standard architectural doctrine. The naive
retries: 3default hardcoded into so many SDKs and HTTP clients without budgets or circuit breakers is easily one of the most widespread footguns in backend systems today. Thanks for reading! πThe prose says to respect Retry-After, but the safe wrapper still chooses its delay only from the local jitter window. With a 429 asking for a longer pause, that could send the next attempt before the server's requested wait has elapsed. I'd carry that wait into the scheduling decision and stop if it exceeds the caller's remaining deadline. How would you combine the server's minimum wait with jitter without collecting every throttled client at the same wake-up time?
Great catch on Retry-After. To respect the server's minimum pause without creating a synchronized wake-up wave, you can apply additive jitter on top of the requested wait: delay = retryAfter + random(0, jitterWindow), or scale it proportionally like retryAfter * (1 + random(0, 0.2)). That ensures no client retries prematurely while still spreading out the wake-up times.
And spot on about deadlines: if now + delay exceeds the caller's remaining budget, fail fast immediately and drop the work instead of waiting on a response nobody will ever read.
The 4^4 example is the clearest way I have seen to explain why retries at every layer are dangerous. Each layer looks sensible on its own, and together they turn one slow dependency into 256 requests.
Mudassir's case of an SDK default retry sitting under a gateway retry is the one that catches teams, because nobody chose it. It came with the library. On client systems we now list every retry in the request path during review, including library defaults, and decide which single layer owns retries. The others fail fast and pass the error up.
Nathan's point deserves more attention than it usually gets. A retry that quietly succeeds turns a reproducible failure into an intermittent one, so we log every retried call with the original error even when the final result is a success. That log is often the first sign a dependency is getting worse.
Do you prefer the retry budget at the gateway, as Ali suggested, or per client?
Definitely at the gateway (or service mesh layer) for anything crossing network boundaries or handling external ingress.
Per-client budgets fall apart the moment autoscaling kicks in. If an ongoing slowdown triggers an HPA scale-up from 10 to 100 pods, a 10% per-client budget suddenly multiplies the total retry volume by 10x against a dependency that is already failing to recover.
Enforcing the token bucket centrally at the gateway (or via Envoy cluster-level retry budgets) solves two major problems:
Also, listing every retry in the call path during PR review, especially library defaults - is fantastic practice. Making those implicit SDK retries explicit before merging prevents most of these storms before they ever reach production.
the multiplying retries per hop math is the part everyone nods at in review and then ships anyway. most of the multiplication we found in our own stack wasn't in app code, it was an sdk default retry sitting under a gateway retry that nobody remembered enabling.
what fixed it was one layer owning retries and everything below it getting max attempts of 1, plus a small token bucket so retries stop once failures cross a threshold.
do you set the budget per client instance or per downstream service? per instance felt too leaky for us once we autoscaled.
Per downstream service (ideally enforced at the gateway or service mesh layer) is definitely the way to go once autoscaling enters the picture.
As you experienced, per-instance budgets scale linearly with replica count. If an ongoing slowdown triggers an HPA scale-up from 5 to 50 pods, a 10% retry budget per instance suddenly floods the downstream dependency with 10x the absolute retry volume, right when it's already struggling to recover. Enforcing the token bucket centrally per target service (like Envoy cluster-level retry budgets) keeps total retry amplification capped regardless of how wide your app fleet scales.
And spot-on about SDK defaults. Hidden
maxRetries: 3configurations inside client libraries sitting beneath an API gateway or reverse proxy are probably the most common source of accidental retry multipliers in production today.That 256 requests from one click number stayed with me.
We keep the retry budget in the app code right now.
Reading this, the gateway feels like the better place for it.
One thing in the opening card: the two numbers count differently. 4 attempts per hop over 4 hops is 4^4 = 256, and 3 attempts per layer over 5 layers is 3^5 = 243, so the same chain lands in a different place depending on whether you count calls or services. The bound is attempts^(number of retrying callers), so the lever is how many layers retry, which is why your single-layer rule beats tuning the retry count.
On the budget class: +1 per request and a retry cost of 10 gives a long-run cap of 10%, but the bucket starts full at 100, so every fresh process can spend 10 retries at once before the cap bites. Restart 200 pods during an incident and that is 2,000 retries in the first seconds. Starting the bucket at 0 avoids it. Do you keep the budget per process or per client instance?
Great catches. You nailed the math nuance - the exponent (number of retrying layers) is always the dominant lever, which is exactly why stripping retries out of inner hops matters so much more than tuning the retry count itself.
You're also right about the cold-start burst with tokens = 100. During a rolling restart or autoscaling event mid-incident, pre-filled buckets across hundreds of pods can easily trigger a mini retry storm. Initializing at 0 (or at most 10 to allow just a single retry for cold-start network blips) is definitely safer.
Regarding scoping: I keep the budget per downstream dependency at the process level (a singleton per upstream service client). Scoping it globally per process causes a single failing dependency to starve retries for healthy ones, while scoping it per non-singleton client instance makes it too easy to accidentally instantiate multiple budgets and multiply the cap.
The single-layer ownership rule is the one I would tattoo on every SDK: I once found three layers of retries stacked between a webhook handler, an ORM driver, and the gateway β each reasonable on its own, and together they turned one Postgres deadlock into a 60x replay of the same write. On your idempotency checklist item, the detail I would sharpen is "enforced atomically at the storage layer" β I learned the hard way that a check-then-insert for the key still races under parallel retries, so I put a unique constraint on the key column and let the insert itself be the arbiter. When your budget trips and you shed load, do you fail the retried request outright or return a degraded response β and which has bitten you less in practice?