A downstream service we depend on had a minor hiccup — a few seconds of elevated errors, the kind of thing that should have been a non-event. Instead it turned into a forty-minute outage, and the thing that turned a hiccup into an outage was us. Our own retry logic took a service that was struggling and finished the job.
The mechanics are almost darkly funny once you see them. The downstream slowed down. Our calls started timing out. Every timeout triggered a retry. So the moment the service was weakest, we tripled our request volume against it. Every other client integrated the same naive way, so the whole fleet did this at once. The struggling service, already at its limit, got buried under a synchronized wall of retries and stopped responding entirely. It couldn't recover, because the instant it served one request, a thousand queued retries slammed back in. We had built a self-sustaining hammer and pointed it at the exact thing we needed to survive.
This is the retry storm, and I've since learned it's one of the most common ways distributed systems kill themselves. The instinct behind retries is sound — transient failures are real, retrying often works. But a naive retry is a loaded weapon, because it does the most damage precisely when the target can least take it.
We rebuilt our client behavior around a few hard rules. Exponential backoff, so each retry waits longer instead of hammering. Jitter, so a thousand clients don't retry in lockstep at the same millisecond — randomness is what breaks the synchronized wall. A hard cap on total retries, because past a point you're not being resilient, you're being cruel. And a circuit breaker: when a dependency is clearly down, stop calling it entirely for a while, fail fast, and give it room to breathe and recover.
The reframe that stuck with me: your retry policy is not a private decision about your own resilience. It's a load pattern you impose on someone else's system, and it fires hardest at their worst moment. Retry like you're trying to help the thing recover, not like you're entitled to a response.
Back off, add jitter, cap it, and break the circuit. Otherwise your resilience strategy is just a denial-of-service attack you scheduled against your own dependencies.
– Sergey Shinder
Top comments (0)