retries are one of the simplest ideas in distributed systems and one of the easiest to implement in a way that quietly makes an outage worse rather than better. the basic instinct, if a call fails, try again, is reasonable. the naive implementation of that instinct, retrying immediately and repeatedly without any further consideration, is a well-documented way to turn a struggling downstream service into a fully collapsed one, since retries add load precisely when the target system is least able to handle additional load.
exponential backoff exists specifically to avoid this failure mode
retrying immediately after a failure, and retrying again immediately after that failure, produces a tight loop of requests hitting a struggling service at the exact moment it needs relief rather than additional load. exponential backoff, increasing the wait time between successive retry attempts, typically doubling or otherwise scaling up each time, gives a struggling downstream system progressively more breathing room with each retry attempt rather than compounding the pressure on it.
the specific backoff parameters matter less than having genuine backoff at all, a fixed short delay between retries is meaningfully better than no delay, but exponential backoff specifically handles the case where a downstream outage lasts longer than a single short delay would accommodate, since the growing delay between attempts means retry traffic naturally tapers off the longer an outage persists, rather than maintaining constant pressure throughout.
jitter prevents synchronized retry storms across many clients
a subtle but important addition to backoff logic: if many clients experience the same downstream failure simultaneously, which is common since they're often failing due to the same root cause, and all of them retry using identical backoff timing, their retry attempts arrive synchronized in bursts rather than spread out, which can itself overwhelm a downstream system that's trying to recover, even though each individual client is technically following a reasonable backoff strategy.
adding randomized jitter to the backoff delay, so each client's actual retry timing varies slightly around the base backoff schedule rather than following it exactly, spreads retry traffic out over time rather than concentrating it into synchronized bursts, which meaningfully improves a downstream system's ability to actually recover during an outage affecting many clients simultaneously.
not every failure should be retried the same way, or at all
a critical distinction that's easy to overlook: not every failure represents the same kind of problem, and treating all failures with identical retry logic misses important nuance. a failure due to a transient network issue or a temporarily overloaded downstream service is often genuinely worth retrying, the same request might succeed on a second attempt once the transient condition passes. a failure due to invalid input, an authentication error, or a request that violates a business rule will fail identically on every retry attempt, since the underlying cause isn't transient at all, and retrying these failures wastes resources on both sides without any realistic chance of a different outcome.
designing retry logic that distinguishes between these categories, generally by checking the specific error type or status code returned, and only applying retry logic to genuinely transient-seeming failures while failing fast and clearly on non-transient ones, avoids both the wasted resource cost and the misleading delay of retrying a request that was never going to succeed regardless of how many attempts are made.
a maximum retry count and an overall timeout budget need to be explicit
retry logic without an explicit ceiling on total attempts or total elapsed time risks a request effectively hanging indefinitely from the caller's perspective, continuing to retry through an extended outage well past the point where the caller would have preferred to receive a clear failure and move on with alternative handling. setting an explicit maximum number of retry attempts, and ideally an overall timeout budget for the complete retry sequence, ensures that a caller receives a definitive failure response within a bounded, predictable time window rather than an indefinite, unbounded wait.
this ceiling should generally be informed by what the calling context can actually tolerate, a background job processing queue can often tolerate a longer overall retry budget than a synchronous request a user is actively waiting on, which means the appropriate retry ceiling isn't necessarily uniform across every call site in a system, but should be considered explicitly for each context based on its actual tolerance for delay.
circuit breakers complement retry logic rather than replacing it
retry logic operates at the level of a single request, deciding whether and how to retry that specific call. circuit breaker logic operates at a broader level, tracking the recent failure rate of an entire downstream dependency and, once that failure rate crosses a threshold, temporarily stopping new requests, including retries, from being sent to that dependency at all, giving it a window to recover without continued pressure from any caller, not just the caller currently in the middle of its own retry sequence.
these two mechanisms work together rather than substituting for each other, retry logic handles the case of an individual, likely transient failure, while circuit breaker logic handles the case of a genuinely sustained outage where continuing to send any traffic, retried or not, is actively counterproductive to the downstream system's recovery.
idempotency needs to be confirmed before retrying anything that changes state
retrying a read operation is inherently safe, since reading data twice produces no different effect than reading it once. retrying an operation that changes state, without confirming that operation is genuinely idempotent, risks the exact duplicate-effect problem that idempotency design specifically exists to prevent, a payment charged twice, a record created twice, because the original request actually succeeded server-side but the response was lost, making a retry indistinguishable from a fresh, unrelated attempt from the client's perspective.
retry logic for state-changing operations specifically needs to be paired with genuine idempotency guarantees, typically through an idempotency key included with the original request and honored consistently across retries of that same logical operation, rather than being treated as a purely transport-layer concern independent of what the underlying operation actually does.
the underlying discipline
a sane retry strategy requires treating retry logic as a deliberate design decision with several interacting parameters, backoff timing, jitter, failure classification, retry ceiling, circuit breaker coordination, and idempotency guarantees, rather than a single global setting applied uniformly and without much further thought across every call in a system. the systems that handle downstream failures gracefully aren't the ones that retry the most aggressively. they're the ones that retry deliberately, in a way that actually helps a struggling dependency recover rather than compounding the pressure on it further.
Top comments (0)