The task read simple: make the outbound call survive a temporary failure. The agent returned a retry wrapper with exponential backoff: wait 1s, 2s, 4s, cap at three attempts. The test fixture returned 500 on the first call and 200 on the third, the suite went green, and the change merged. Under real load the upstream started returning 429 Too Many Requests, and the retry made things worse instead of better.
Two assumptions the agent never checked: that every failed response should be retried the same way, and that a fixed backoff schedule is enough. Both held against a generic 500 fixture and broke against the response traffic actually produced.
A clean backoff becomes a lockstep parade
Without jitter, backoff is a fixed formula: same base, same multiplier, same cap. When many clients hit the same throttled endpoint at roughly the same time, they also sleep the same computed duration and fire the next attempt at nearly the same instant. The server throttles again, they wait the same next interval, and the cluster retries in lockstep. A quiet recovery window turns into repeated synchronized spikes.
| Backoff shape | What happens under shared load |
|---|---|
| Fixed interval, no jitter | All clients retry together; highest overlap |
| Exponential, deterministic | Still synchronized; overlap shrinks slowly, stays tight |
| Exponential plus jitter | Attempts spread across the window; smoother retry curve |
Jitter does not reduce how often you retry; it spreads when the retries land. That distinction matters when the bottleneck is concurrency rather than a single slow request.
429 is not the same failure class as 500
A 500 means the server failed while handling the request; retrying later may help. A 429 means the server already accepted the request but told you to slow down, usually with a Retry-After header stating the earliest acceptable wait. Treating both as generic transient errors and applying one self-chosen schedule ignores the message the server sent on purpose.
The agent's fixture modeled only the first shape: a server that errors and recovers. It never modeled a throttle that answers with a wait instruction, so honoring that header never entered the code.
The rules that actually hold in traffic
- Read
Retry-Afteron 429 and treat it as a minimum wait, not a suggestion. Your own backoff cap should not be shorter than the delay the server already published. - Add jitter around the interval so retries fan out instead of lining up. A random offset within the computed window is enough; you do not need a custom distribution.
- Split retryable classes before choosing the action. Connection failures and 503 may retry; a 401, a 403, a validation 400, or a permanent 404 should stop, never back off.
- Stop the fanout under sustained throttle. After N attempts on 429, back off further or surface overload rather than keep hammering a server that asked you to wait.
- Keep the idempotency question in view: backoff on a safe read is cheap; repeating a mutating call on throttle carries the same duplicate risk as any blind retry.
Before baking a retry policy into code, look at what the real throttled response returns instead of inventing it. Keeping the spec and the actual request and response surface local, as Powerduck does, makes the Retry-After field and the response shape something you inspect directly rather than a guess.
Write the fixture the agent skipped
A retry suite that only simulates a failing-then-succeeding server keeps endorsing the lockstep path. Add a case that returns 429 with a Retry-After value, assert the client waits at least that long, and add a small fanout check that two simulated clients do not land their retries on the same timestamp.
Resilience is not retrying every failure on a schedule you picked. It is respecting what the server told you, spreading your own retries, and stopping when the answer is permanent.
Where does your retry logic currently treat 429 like a generic error, and does any test ever check the Retry-After header?
Top comments (0)