Production-Ready API Retry Logic: Backoff, Jitter, and Idempotency
Your payment API call failed at 2 a.m. Was it a network blip, a rate limit, or a dead server? The answer determines whether retrying saves the transaction—or double-charges a customer.
Retries are one of the most common resilience patterns in distributed systems—and one of the easiest to get wrong. A loop around an HTTP call may look defensive, but poorly designed retries can turn a 30-second outage into a 30-minute one by making thousands of clients hammer an already struggling server.
Done correctly, retries absorb transient failures so smoothly that users never notice them.
This guide covers:
- Which failures to retry
- Exponential backoff with full jitter
-
Retry-Afterheaders - Idempotency keys for
POST - Retry budgets and circuit breakers
- Testing retry behavior with Apidog mock servers
A retry pattern that has never been tested against a failing server is a guess, not a design.
Why naive retries make outages worse
Imagine a service handling 1,000 requests per second. It fails for five seconds, and every client retries immediately three times. Demand jumps from 1,000 requests per second to 4,000—directly at a server already under stress.
The server falls over completely, and every client retries again.
This feedback loop is called a retry storm. When clients retry in sync as the server recovers, the result is a thundering herd. Google’s SRE guidance on addressing cascading failures explains why retries without backoff amplify load when a system can least afford it.
Most retry storms come from two design flaws:
- No delay: Immediate retries multiply load during the worst possible window.
- Fixed delays: If every client waits exactly one second, they all retry in lockstep.
The solution is not to avoid retries entirely. Retry selectively, use growing randomized delays, and impose a hard limit on additional load.
Retry these failures, not those
Before implementing backoff, define a decision table. Retrying an invalid request wastes capacity and pollutes logs; retrying a transient fault is the purpose of the mechanism.
Retry these
| Signal | Meaning |
|---|---|
429 Too Many Requests |
You hit a rate limit. Back off and return more slowly. |
502 Bad Gateway |
An upstream hop returned an invalid response. Often transient. |
503 Service Unavailable |
The server is overloaded or restarting. |
504 Gateway Timeout |
An upstream dependency took too long. |
| Connection resets, DNS failures, socket timeouts | The request may never have reached the server. |
A handling idempotency.
Do not retry these automatically
| Signal | Meaning |
|---|---|
400 Bad Request |
The payload is malformed and will fail again. |
401 Unauthorized |
Credentials are wrong or expired. Refresh the token instead. |
403 Forbidden |
The caller lacks permission. |
422 Unprocessable Entity |
Validation failed. Fix the data, not the timing. |
The general rule:
- Retry when the failure concerns the server’s state or the network.
- Fail fast when the failure concerns the request itself.
A 429 is retryable, but repeated 429 responses also indicate that your overall request rate needs attention. Solve that with client-side throttling, caching, or another rate-limiting strategy, not an increasingly aggressive retry loop.
Exponential backoff and full jitter
Exponential backoff increases the delay after each failed attempt:
delay = base * 2^retry_count
With a base delay of 500 ms, the schedule is:
0.5s, 1s, 2s, 4s, 8s
Always add a cap:
delay = min(cap, base * 2^retry_count)
For example, a 30-second cap prevents delays from growing indefinitely.
Plain exponential backoff prevents immediate hammering, but it does not prevent synchronization. If 5,000 clients fail at the same time, they may all retry at 0.5 seconds, then 1 second, then 2 seconds.
That still creates traffic waves.
Full jitter
Full jitter randomizes the delay between zero and the exponential ceiling:
delay = random_between(0, min(cap, base * 2^retry_count))
The AWS analysis of exponential backoff and jitter found that backoff without jitter still produced clustered call spikes. Full jitter produced fewer total calls and near-shortest completion times.
Randomizing down to zero may feel less orderly than a fixed doubling schedule, but distributing clients across the entire retry window keeps load flatter. Full jitter is a simple, reliable default unless measurements show that another strategy is better.
Honor Retry-After
Backoff is your client estimating how long to wait. Sometimes the server provides a better answer.
The Retry-After header, commonly returned with 429 and 503 responses, contains either a number of seconds or an HTTP date:
HTTP/1.1 429 Too Many Requests
Retry-After: 12
When present, Retry-After should override your computed backoff. The server knows when its rate-limit window resets or maintenance ends; your exponential schedule does not.
Parse and respect the header, but still enforce:
- A maximum delay
- A maximum retry count
- A deadline for the overall operation
This prevents a malformed or hostile value such as Retry-After: 86400 from blocking a worker for an entire day. See the Retry-After header reference for parsing details.
Idempotency: the precondition for retrying POST
The 504 scenario creates a dangerous ambiguity. GET, PUT, and DELETE are generally idempotent: sending the same request more than once should leave the resource in the same state. POST is not inherently idempotent.
If POST /v1/payments times out after the server processes the payment, retrying can create a second charge.
The solution is an idempotency key: a unique, client-generated identifier—usually a UUID—sent as a request header for each logical operation. The server stores the key with the initial response and returns that stored response for duplicates. Stripe’s idempotent request implementation follows this model.
Two rules are essential:
- One operation, one key. Every retry of one payment must reuse the same key. A new user action gets a new key.
- Generate the key before the first request. Do not create it inside the retry loop.
If the API does not support idempotency keys, avoid automatically retrying non-idempotent writes. Surface the failure and let a human or reconciliation process decide what happened.
For more implementation guidance, see idempotency keys.
Retry budgets and circuit breakers
Backoff controls when retries happen. It does not control how many retries occur.
During a prolonged outage, even well-jittered clients can create substantial retry traffic. Layered retries make this worse: if an API gateway retries three times and the service client also retries three times, one user action can produce up to nine requests.
Retry budgets
Instead of allowing “three retries per request,” define a global budget such as:
Retries may add no more than 10% extra traffic over a sliding window.
When the budget is exhausted, return failures immediately. This bounds retry amplification regardless of how many requests are failing. Linkerd and Envoy both support retry-budget configurations.
Circuit breakers
A circuit breaker tracks failures for each downstream dependency. When the failure rate exceeds a threshold, it opens and fails calls immediately without sending network requests.
After a cooldown, the breaker permits a small number of probe requests. If the dependency has recovered, the breaker closes.
Backoff slows a stampede; a circuit breaker stops it. Production systems typically use both.
A production-ready Python example
This example combines:
- Retryable-status filtering
- Full jitter
-
Retry-Aftersupport - An idempotency key
- A maximum retry count
import random
import time
import uuid
import requests
RETRYABLE = {429, 502, 503, 504}
BASE = 0.5 # seconds
CAP = 30.0 # ceiling on any single delay
MAX_RETRIES = 5
def create_payment(payload):
idempotency_key = str(uuid.uuid4()) # one key per logical payment
headers = {"Idempotency-Key": idempotency_key}
for retry_count in range(MAX_RETRIES + 1):
try:
resp = requests.post(
"https://api.acmepay.com/v1/payments",
json=payload, headers=headers, timeout=10,
)
if resp.status_code < 400:
return resp.json()
if resp.status_code not in RETRYABLE:
resp.raise_for_status() # 400/401/403/422: fail fast
retry_after = resp.headers.get("Retry-After")
except (requests.ConnectionError, requests.Timeout):
retry_after = None # network fault: fall through to backoff
if retry_count == MAX_RETRIES:
raise RuntimeError("payment failed after all retries")
if retry_after and retry_after.isdigit():
delay = min(CAP, float(retry_after))
else:
delay = random.uniform(0, min(CAP, BASE * 2 ** retry_count))
time.sleep(delay)
Important details:
- The idempotency key is generated once, outside the loop.
-
Retry-Aftertakes precedence over calculated backoff. - The delay still respects the cap.
- Non-retryable HTTP statuses fail immediately.
- Network failures fall through to jittered backoff.
On JavaScript projects, axios-retry provides the same structure through retryCondition and retryDelay hooks. The decision table remains the same.
Test retry behavior before production does it for you
Many systems ship retry logic whose failure branch has never executed. The happy path was tested; the 503 path runs for the first time during a real outage.
You can test it with two Apidog features.
1. Simulate failures with mock servers
Use Apidog’s smart mock server to define an endpoint such as /v1/payments and control its responses:
- Return
503for the first two calls and200on the third. - Return
429withRetry-After: 5. - Add a 15-second delay to trigger a client timeout.
Point your client at the mock URL and observe the retry behavior without creating a production incident.
2. Assert behavior with test scenarios
Apidog test scenarios can chain requests and validate responses and timing. Build a scenario that:
- Calls the flaky mock endpoint.
- Verifies that the request eventually succeeds.
- Checks that elapsed time falls within the expected backoff envelope.
- Confirms that exactly one resource was created.
- Proves that the idempotency key deduplicated retries.
Run the scenario in CI so retry logic is exercised on every commit instead of during every outage.
That is the difference between “we added retries” and “we verified that our client survives a rate-limited, partially unavailable dependency.” You can download Apidog and run a failing mock server against your client in about ten minutes.
FAQ
Should I retry a 429?
Yes. It is the status where the server most often tells you how long to wait. Read Retry-After and wait at least that long. If the header is missing, use exponential backoff with jitter.
Repeated 429 responses should also trigger client-side throttling, caching, or other changes to reduce request volume.
What is full jitter?
Full jitter chooses each delay uniformly at random between zero and the exponential ceiling:
random(0, min(cap, base * 2^n))
It prevents synchronized retry waves. In AWS simulations, it outperformed plain backoff and equal jitter in both total calls and completion time, which is why it is a common default in AWS SDKs.
Is it safe to retry POST requests?
Only when the operation is idempotent in practice. For POST, that usually means supplying an idempotency key that the server uses to deduplicate requests.
Without one, retrying after a timeout can duplicate a payment, order, or record because the server may have processed the request before the client received the response. AI agent error recovery patterns use the same principles: keyed writes, capped retries, and circuit breakers.
How many times should I retry?
Three to five attempts handle most transient faults. Beyond that, success rates typically flatten while latency and load continue to increase.
Pair the per-request retry cap with a global retry budget—for example, allowing retries to add no more than 10% extra traffic. If a dependency remains unavailable after the final attempt, use circuit-breaker behavior instead of retrying indefinitely.
Top comments (0)