Here's a pattern I've watched play out more times than I can count, and it always looks the same from the outside: a service has a small hiccup — one dependency gets slow, not even down, just slow — and instead of recovering in seconds, the whole system falls over completely and stays down long after the original hiccup passed. The postmortem finds no dramatic cause. No bad deploy, no data-center fire. Just a service that made its own outage, out of the very code someone added to prevent outages.
The culprit is almost always retries. Naive retries. The most well-intentioned code in the building.
The mechanism, because the mechanism is the whole point
Picture a service calling a dependency that briefly slows down. Requests start timing out. Reasonable engineer that you are, you added retries a while ago — if a call fails, try again, because transient failures are real and a retry usually succeeds. Good instinct, in isolation.
Now watch it at scale. The dependency slows. A thousand in-flight requests time out. Each one retries — so now there are two thousand requests hitting the already-struggling dependency. Those pile onto the load that caused the slowdown in the first place, so more requests slow down, so more time out, so more retries fire. You have built a feedback loop where failure generates traffic, and the traffic generates more failure. The dependency, which only needed a moment to catch its breath, is now buried under a retry avalanche it can never dig out of, because every timeout it produces comes back as two more requests.
This has a name: a retry storm, and its ugly cousin the thundering herd. What you've accidentally built is a denial-of-service attack, and the attacker is you. The dependency didn't fail because it was fragile. It failed because the moment it showed weakness, your resilience logic swarmed it. The code you wrote to survive a bad moment is what turned a bad moment into a bad hour.
The reason this is so dangerous is that it's invisible until load. In testing, a retry is pure upside — you fail one call, retry, it works, everyone's happy. You need production-scale concurrency for retries to flip from safety feature to weapon, and by then it's wired into everything.
Three changes, in order of importance
You don't remove retries. Retries are genuinely useful. You make them polite.
1. Exponential backoff with jitter. A naive retry fires again immediately, or on a fixed delay. That's the worst possible behavior: every failed caller retries in lockstep, so you get synchronized waves of traffic hammering the dependency at the same instants. Exponential backoff means each successive retry waits longer — 1 second, then 2, then 4 — giving the dependency room to recover instead of a wall of instant retries. And jitter — randomizing each delay — is the part people skip and shouldn't: without it, a thousand callers that failed at the same moment all back off for the same duration and then all retry at the same later moment, so you've just moved the stampede, not stopped it. Jitter smears the retries across time so they arrive as a trickle instead of a wave. Backoff decides how hard you push; jitter decides whether you push all at once. You need both.
2. A retry budget, or: give up. The hidden assumption in "retry on failure" is that failures are rare and independent. During an outage they're neither — everything is failing at once, and retrying is worse than useless because there's nothing healthy to retry into. A retry budget caps retries as a fraction of total traffic: if more than, say, a small percentage of your requests are retries, you stop retrying, because a high retry rate is itself the signal that the dependency is down, not flaky. The counterintuitive discipline is that the moment retries matter most emotionally is the moment they help least. When everything's on fire, the correct move is often to fail fast and shed load, not to try harder. Trying harder is what lit the fire.
3. A circuit breaker. This is the structural fix. A circuit breaker watches the failure rate to a dependency, and when it crosses a threshold, it trips — it stops sending requests entirely for a cooldown, failing them instantly at the caller instead. That sounds worse ("you're failing requests on purpose!") and is dramatically better, because it does the one thing a struggling dependency needs: it takes the load off. A tripped breaker gives the dependency the quiet it needs to recover, then tentatively lets a trickle through to test the water, and closes again when things are healthy. Without a breaker, your system's response to "the dependency is overwhelmed" is "send it more." With one, the response is "leave it alone until it's better." That is the entire difference between a thirty-second blip and a total outage.
The real lesson is about resilience itself
The thing I want you to take away is bigger than retries. It's that resilience mechanisms have failure modes, and theirs are worse than the failures they prevent, because they fire exactly when the system is already stressed. Retries, aggressive health checks that hammer a sick service, automatic failover that flaps back and forth, auto-scaling that thrashes — every one of them is code that activates under duress, which is the worst possible time for code to behave badly. A retry misbehaving on a calm Tuesday is nothing. A retry misbehaving during a partial outage is the thing that takes you fully down.
So when you add anything whose whole job is to handle failure, ask the second question, the one that's easy to skip because the first answer felt so responsible: what does this do when a thousand copies of it fire at the exact same moment? The naive version of every resilience feature has the same bug — it assumes it's acting alone, when the definition of an outage is that it's acting in a crowd. Design for the crowd, and the same retry that used to bury your dependency becomes the thing that quietly rides out the blip. Same instinct. Opposite outcome. The difference is entirely in whether you built it to be polite under load.
Top comments (0)