Your Retries Are a Load Multiplier: How a 5-Second Blip Becomes a 40-Minute Outage

The problem

A downstream dependency gets slow for five seconds — a GC pause, a leader election, a brief network hiccup. Normally you'd never notice. But every client is configured to retry three times, so the moment things get slow, the struggling service starts receiving several times its normal traffic. It gets slower. More requests time out. More retries fire. Five seconds later the original trigger is gone, but the outage isn't — the system is now DDoSing itself.

That's a retry storm, and it's one of the most common ways a minor blip turns into a major incident. The nasty part: it's self-sustaining. It can outlive the thing that started it.

Why it happens

Two mechanisms, and most engineers only ever think about the first one.

1. Retries multiply — and they multiply per layer.

Everyone knows a retry adds load. What people miss is that retries compound across every layer of the stack. If your gateway retries 3×, and it calls a service that also retries 3×, and that service calls a data layer that retries 3× on its own, a single user request can become up to 3 × 3 × 3 = 27 attempts at the bottom.

attempts at the bottom layer = R ^ N   (R retries per layer, N layers)

3 layers × 3 attempts =  27×
4 layers × 3 attempts =  81×

The bottom of the stack is almost always your database — the least horizontally scalable layer — and it eats the amplified load. You provisioned for 1× and handed it 27×.

2. Past capacity, retries become a feedback loop.

Below capacity, a few retries are harmless. Above capacity, they're rocket fuel. Failures cause retries → retries raise offered load → higher load causes more failures. Goodput collapses toward zero while the system is doing maximum work. In a quick simulation of a service capped at 1,000 req/s: offered load of 1,200 req/s with naive retries pushes effective load past 50,000 req/s as the loop runs away — and useful throughput stays pinned at 1,000. This is congestion collapse (a "metastable failure"): the retry load is now self-sustaining, so the system stays down even after the original cause clears. You often have to shed load to break it.

What to do about it

Retry at exactly one layer. This is the biggest lever and the one most teams miss. Pick the layer with the most context — usually the edge or the direct caller — and make everything below it fail fast. Retrying at every layer is what manufactures the 27×.

Use backoff with jitter — the jitter is the whole point. Plain exponential backoff synchronizes the herd: everyone fails at t=0, everyone sleeps 1s, everyone retries at t=1s. You just rebuilt the spike one second later. Add full jitter:

# Full jitter (AWS): sleep a random amount in [0, cap]
import random
def backoff(attempt, base=0.1, cap=10.0):
    return random.uniform(0, min(cap, base * 2 ** attempt))

In a sim of 10,000 clients retrying at once: no jitter → all 10,000 land in the same 50 ms window; full jitter → the peak in any 50 ms window drops to ~580 (~5.8% of the herd). Same total retries, ~17× lower peak.

Cap with a retry budget, not just a per-call count. "3 retries per request" tells you nothing about system-wide load. A budget does: allow retries only up to, say, 10% of live request volume (token bucket); above that, fail fast. That bounds amplification at 1.1× no matter how bad the failure gets. Envoy and Finagle both ship this.

Add circuit breakers and propagate deadlines. When a dependency is clearly failing, stop calling it for a cooldown instead of hammering it. And pass a deadline down the call chain so a request that the client already abandoned doesn't get retried three more times underneath.

Only retry idempotent, retryable errors. Retrying a non-idempotent write, or a deterministic 400, just doubles the damage with zero chance of success.

Key takeaways

• Retries are a load multiplier. Across N layers each retrying R times, one request can become R^N attempts — and the bottom (your DB) pays.
• Past capacity, unbounded retries cause congestion collapse: goodput craters while work peaks, and it can outlive the original trigger.
• Retry at one layer, back off with jitter, cap with a retry budget (~10%), add circuit breakers + deadlines, and only retry idempotent failures.

If you want to go deeper, the AWS Architecture Blog's "Exponential Backoff And Jitter" and Marc Brooker's writing on metastable failures are the canonical references.