- Book: System Design Pocket Guide: Fundamentals
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
The database fails over. Thirty seconds later it is back, healthy, with plenty of capacity for normal traffic. Your dashboards stay red for the next hour anyway. Every node is at 100% CPU and almost nothing succeeds. Someone restarts the database, which changes nothing. Someone else finally turns off the retries in the API gateway, and the whole system recovers in about a minute.
That shape has a name. In 2021, Nathan Bronson and three co-authors called it a metastable failure in a HotOS '21 paper: a trigger pushes the system into a bad state, and that state persists even when the trigger is removed. A follow-up study at OSDI '22, Metastable Failures in the Wild, went through public incident reports and found that at least 4 of 15 major AWS outages in the last decade were metastable failures.
Retries are one of the most common things that keep the bad state going. The simulator below reproduces the effect with numbers you can check by hand, then adds the fix that production RPC stacks ship: a retry budget.
The multiplication nobody adds up
Each retry policy looks harmless in isolation. "Try three times" is a sensible default for a service calling a flaky dependency. The problem is that your stack has several services, and each one has its own sensible default.
The AWS Builders' Library article on timeouts and retries walks through a five-deep stack with three retries at each layer and puts it plainly: if each layer retries independently, load on the database goes up 243x, "making it unlikely to ever recover." With three layers and three tries each, the bottom of the stack sees up to 3 × 3 × 3 = 27 attempts for every user request.
Here is why 27 is the exact count when every call fails. Layer A calls B up to 3 times. Each of those B calls makes up to 3 calls to C. Each of those makes up to 3 calls to the database. When the database fails every call, the attempts count up like an odometer, and the database sees all 27 before the user gets an error.
Your normal traffic might be 70% of the database's capacity. Multiply it by 27 and you need 19 databases. The fault that started it is gone, and the retries are now the outage.
Build the simulator
The model is deliberately small. A database serves 100 requests per second, oldest first. Users send 70 new requests per second. A caller waits one second, then gives up and retries. A request the caller abandoned stays in the database's queue, and the database still does the work when it gets there, which is how real servers behave without deadline propagation. For 30 seconds, the database serves nothing.
Because a 3-deep stack of 3 tries collapses to a flat count of 27 attempts when every call fails, the simulator models the whole stack as one caller with max_attempts = 27. It runs on plain Python 3.8 or later, no dependencies:
# retry_storm.py: one 30-second blip, three retry policies
import math
from collections import defaultdict, deque
CAPACITY = 100 # requests the database serves per second
BASE_LOAD = 70 # new user requests per second
TIMEOUT = 1 # seconds a caller waits before giving up
BLIP = range(60, 90) # 30 seconds with zero capacity
DURATION = 3600 # one simulated hour
def simulate(max_attempts, budget=None):
queue = deque() # FIFO of [enqueued, attempt, n]
by_tick = defaultdict(list) # enqueue tick -> its batches
due = defaultdict(list) # tick -> [(attempt, n)] retries
queued = useful = served = 0
peak_load = peak_queue = 0
last_bad = None
def enqueue(now, attempt, n):
nonlocal queued
batch = [now, attempt, n]
queue.append(batch)
by_tick[now].append(batch)
queued += n
for now in range(DURATION):
# 1. new requests, then the retries the budget allows
enqueue(now, 1, BASE_LOAD)
sent = BASE_LOAD
room = math.inf
if budget is not None:
room = round(BASE_LOAD * budget)
for attempt, n in due.pop(now, []):
n = min(n, room)
room -= n
sent += n
if n:
enqueue(now, attempt, n)
peak_load = max(peak_load, sent)
# 2. the database works through the oldest requests
cap = 0 if now in BLIP else CAPACITY
while cap and queue:
batch = queue[0]
take = min(cap, batch[2])
if now - batch[0] <= TIMEOUT:
useful += take
served += take
batch[2] -= take
queued -= take
cap -= take
if batch[2] == 0:
queue.popleft()
# 3. callers still waiting past the timeout give up
for batch in by_tick.pop(now - TIMEOUT, []):
attempt, n = batch[1], batch[2]
if n and attempt < max_attempts:
due[now + 1].append((attempt + 1, n))
peak_queue = max(peak_queue, queued)
if now >= BLIP.start and queued > CAPACITY * TIMEOUT:
last_bad = now
if last_bad == DURATION - 1:
recovered = None
else:
recovered = last_bad + 1 - BLIP.stop
return (recovered, peak_load / BASE_LOAD, peak_queue,
1 - useful / served)
def report(label, max_attempts, budget=None):
rec, load, peak_q, waste = simulate(max_attempts, budget)
when = "never" if rec is None else f"after {rec}s"
print(f"{label:<16} recovered {when:<11} "
f"peak load {load:4.1f}x peak queue {peak_q:>9,} "
f"wasted {waste:4.0%}")
report("no retries", 1)
report("3 layers x 3", 3 ** 3)
report("same + budget", 3 ** 3, budget=0.1)
A few things to know before you run it. Time moves in one-second ticks, and requests travel in batches so an hour of millions of queued requests stays fast. "Recovered" means the queue has dropped to one timeout's worth of work (100 requests) and stays there, because past that point a new request gets served before its caller gives up. "Wasted" is the share of the database's work that went to requests whose caller had already walked away.
Predict the numbers, then run it
Work each run out on paper first. That is the point of a model this small: if the output disagrees with your arithmetic, one of you has a bug, and finding it teaches you more than the run does.
No retries. During the blip, 30 seconds × 70 requests = 2,100 requests pile up. Afterwards the database drains them at 100 − 70 = 30 per second. 2,000 requests at 30 per second is roughly 67 seconds, so expect recovery a little over a minute after the blip ends. Peak load: 1.0x.
Three layers, three tries. Every request queued during the blip times out, and so does every retry, because each one lands at the back of a growing queue. After 26 rounds of retries, every second carries 70 new requests plus 26 × 70 retries, which is 1,890 requests per second against a capacity of 100. The queue grows by about 1,790 every second. It never drains, so expect recovered never, a peak load of 27.0x and a wasted-work figure close to 100%: the database spends the hour busy, almost entirely on answers nobody is waiting for. This is the metastable state. The blip ended at second 90, capacity came back, and the system stayed down because of its own retries.
Same stack plus a 10% budget. Retries can never exceed 10% of new requests, so arrivals are capped at 77 per second. The backlog after the blip is about 2,300, draining at 100 − 77 = 23 per second. That is around 95 to 100 seconds. Peak load: 1.1x.
Run python3 retry_storm.py and compare. The budget run recovers a little slower than the no-retry run, because retries still eat some capacity. That is the cost of the budget, and it is small. The difference between a minute and a half and never is the whole argument.
What a retry budget is
A retry budget caps retries as a fraction of real traffic, instead of capping them per request. Per-request limits ("try three times") say nothing about how many requests are failing at once. A budget does, because it is shared by every request a client sends.
This is not a new idea. The production stacks you already use ship it:
- The Google SRE book's chapter on handling overload describes two limits: up to three attempts per request, and a per-client budget where a request is only retried while the client's retry ratio is below 10%. The chapter says the budget reduces worst-case growth from "somewhere just below 3X" to "just 1.1x in the general case."
-
gRPC's retry design (gRFC A6) has retry throttling as a token bucket. Every failed RPC removes one token, every success adds
tokenRatio(0.1 in the example config), and retries stop when the count drops to half ofmaxTokens. In a storm where everything fails, the bucket empties fast and retries switch off by themselves. - Finagle's RetryBudget by default lets about 20% of requests retry immediately, with a floor of 10 retries per second so quiet clients can still retry at all.
The gRPC and SRE versions also attack the layering problem from the other side. The SRE chapter says requests should only be retried at the layer immediately above the one rejecting them, and an overloaded backend can return an "overloaded; don't retry" error so the layers above stop multiplying. gRPC sends a grpc-previous-rpc-attempts header on retries, so a server can see that a request is already a retry.
One caveat about the simulator: it applies the budget to every extra attempt the database sees. A real stack only gets that if each layer runs its own budget (three layers at 10% gives at most 1.1³, about 1.33x) or if retries are tagged so lower layers can count them. Either works. A budget in one service and blind retries in the other two does not.
Experiments worth running
Change one constant at a time and predict the result first.
- Set
BASE_LOAD = 90. A 10% budget then puts 99 requests per second on a database that serves 100. One request per second of headroom against a backlog of about 2,950 is a drain of most of the simulated hour. Budgets need spare capacity to work. That is a capacity-planning problem. - Set
budget=0.2, Finagle's default ratio. Work out the drain rate before you run it. - Make the blip 5 seconds instead of 30. Predict whether the naive run still tips over. In this model, a trigger tips the system over once the backlog grows past one timeout's worth of work, however small the trigger looks.
- Add a queue cap in
enqueuethat rejects requests past, say, 500 queued. Rejections fail fast instead of wasting work, which changes the numbers. Then check whether retries alone can still keep the system down.
Backoff and jitter deserve a mention here too, because they are the first thing people reach for. Exponential backoff spreads the 27 attempts over a longer window, which lowers the per-second peak. It does not change the count: each failing request still sends 27 attempts to the bottom of the stack. The budget is what limits the total.
Add up the retry policies along your deepest call path and multiply them. If the product is bigger than the headroom your bottom dependency has, a short blip can turn into a long outage, and the place to fix it is the retry policy.
If this was useful
Retry budgets only work next to timeouts, rate limits and capacity headroom. System Design Pocket Guide: Fundamentals walks through those building blocks and the availability trade-offs behind them. If this simulator made you want to multiply out your own call paths, start there.
Top comments (1)
The assumption that keeps your naive run down for the whole hour is the one you name in passing: the database still serves requests whose caller has already left. I ran your script with one change in step 2, discarding any batch older than TIMEOUT instead of serving it. The 3 × 3 × 3 stack, with no budget, then recovers 36 seconds after the blip ends instead of never, and wasted work drops from about 99% to zero. Peak load still reaches 16x, so the budget is still doing a job the deadline check can't.
That makes it a cheap experiment to add next to the queue cap. A gRPC server already cancels a call once the client's deadline passes, and Java and Go propagate that deadline to outgoing calls by default, so the application only has to check for the cancellation before it starts expensive work.