We hit our own rate limit at 200 requests per second, and the retry logic turned it into 600. The retries did not cause the incident. They caused the incident to last four hours instead of twenty minutes.
Three codes, three meanings
Most retry helpers treat "retryable" as one bucket. It is three, and they want opposite things.
429 Too Many Requests. You are over your quota. The server is not overloaded; it is protecting itself from you specifically, and it usually tells you exactly how long to wait. Retrying immediately is a request to be told no again. The correct response is to honour Retry-After and reduce your own rate — a client-side token bucket that the 429 is the feedback signal for.
503 Service Unavailable. The server is under pressure from everyone. Retrying is correct, but only with backoff and jitter, and only a bounded number of times. Here the retry is a hope that the load shifts; a retry storm makes the load worse but the 503 is not addressed to you.
Timeouts and connection resets. The outcome is unknown. The request may have been processed. Retrying is required for the client's behaviour and dangerous for the system, which is why this is the only case that demands an idempotency key rather than a backoff policy.
If your retry wrapper cannot tell these apart, it is guessing. Ours looked like this:
for attempt in range(3):
resp = session.post(url, json=body)
if resp.status_code != 429:
return resp
sleep(2 ** attempt)
Two seconds after a 429 is two more requests inside the window that just closed. With three attempts and 200 concurrent workers, the number of requests in flight tripled.
What we changed
Retry on 429 became "do not retry, slow down". The caller got a token bucket sized to the documented limit, shared across the process, refilled per second. A 429 now drains the bucket and the request waits for a token instead of failing and retrying. Concurrency to the upstream dropped from burst-shaped to flat.
Retry-After is respected even when it is impolite. An upstream answering Retry-After: 30 is a thirty-second pause for that key, not a suggestion. The alternative is a ban.
Bounded queue with a deadline in front. The requests that used to retry now sit in a short queue with a hard deadline. When the queue is full, the caller gets a 429 from us immediately, which is honest and cheap. When the deadline passes, the work is dropped rather than executed late — a payment captured forty seconds after the user gave up is a worse outcome than a clean failure.
Retries got a cap per key, not just per request. Three attempts per request, but also a total budget per idempotency key. Without that, a client retrying a failing request adds its own attempts on top of ours, and the two multiplication tables compound.
The counter-intuitive part
The instinct under load is to try harder: more attempts, shorter sleeps, more parallelism. Every one of those increases pressure on the thing that is signalling distress. Backpressure is the system telling you its capacity, and the correct response to a capacity signal is to send less traffic, not the same traffic more persistently.
Two numbers worth knowing for any dependency:
- Your steady-state request rate, so a 429 is a comparison and not a surprise.
- The p99 latency of a successful request, which sets the minimum sane timeout. A 500ms timeout on a dependency with a 2s p99 turns latency into errors and errors into retries.
We ended with roughly the same throughput as before the incident, at a third of the peak request count. The retries were never doing useful work. They were just loud.
I write about production failures in Postgres, queues, and distributed systems.
Top comments (1)
This distinction between 429, 503 and timeouts is the part I would keep. One thing to watch is that a bucket shared only inside one process can still let each copy of the service use the full quota, so the total can grow with the number of replicas. Splitting the quota across copies, or using one shared limiter, keeps the client side limit honest when the service scales.