Production failures: the things that page you · 3. Retry storms

Production failures: the things that page you · Chapter 3 of 24 · Reliability · new chapter every Friday morning

By the end of this chapter: See how a well-intentioned client takes down a recovering service, and bound it.

The problem

You are an on‑call engineer for a payments platform that processes about 150 k requests per minute. A downstream fraud‑check service (v2.3.1) suffered a transient network partition at 02:17 UTC. The service returned HTTP 502 for a few seconds, then recovered.

Your service has a generic retry wrapper that blindly retries any non‑2xx response up to three times with no delay. As the partition cleared, the retry wrapper started hammering the now‑healthy fraud service with a burst of duplicated requests. Within a minute the downstream service’s CPU spiked to 100 % and its latency grew from 30 ms to several seconds, causing your own service to time‑out and generate a cascade of alerts. The pager went off again at 02:20 UTC, and you lost roughly 2 % of daily transaction volume (≈ 3 M USD) while you chased the symptom.

You need to stop the retry‑induced overload, restore normal traffic, and put safeguards in place so a future transient failure does not cause a “retry storm”.

Before you start

• Python 3.11.4
• Flask 2.2.5
• requests 2.31.0
• redis‑py 5.0.1 and a local Redis 7.0.11 instance (Docker image redis:7.0-alpine)
• The repository git clone https://github.com/example/payments-service.git (branch retry‑storm‑demo)
• Verify the demo runs:

cd payments-service
docker compose up -d redis
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
flask --app app run --port 5000 &
curl -s http://localhost:5000/health
# Expected output: {"status":"ok"}

If you see {"status":"ok"} the environment is ready.

Why the naive retry wrapper fails

1. What does “retry” really mean in a distributed system?

A retry is a new request, not a continuation of the original. If the upstream service is already recovering, each retry adds load. Without back‑off, retries fire back‑to‑back, turning a brief glitch into a sustained load spike.

2. How can duplicated requests break business logic?

Your payment service is idempotent only at the database level, not at the fraud‑check API. The downstream service records each request as a separate check, inflating its internal queue and causing CPU exhaustion.

3. Which safety nets are missing?

• Exponential back‑off – spreads retries over time.
• Jitter – prevents many clients from synchronising their back‑off intervals.
• Circuit breaker – stops sending traffic to a failing downstream service.
• Rate limiting – caps the number of retries per second per client.

Adding exponential back‑off with jitter

1. How to implement it safely?

Wrap the requests call in a helper that respects Retry-After headers, applies exponential back‑off, and adds full jitter. Use the tenacity library (v8.2.3) which is battle‑tested.

# retry_helper.py
import random
import requests
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type

class RetryableError(requests.exceptions.RequestException):
    pass

def is_retryable(response: requests.Response) -> bool:
    return response.status_code >= 500 or response.status_code == 429

@retry(
    reraise=True,
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(initial=0.2, max=2.0),
    retry=retry_if_exception_type(RetryableError)
)
def post_with_retry(url: str, json: dict, timeout: float = 1.0) -> requests.Response:
    try:
        resp = requests.post(url, json=json, timeout=timeout)
    except requests.exceptions.RequestException as exc:
        raise RetryableError from exc

    if is_retryable(resp):
        raise RetryableError(f"Retryable status {resp.status_code}")

    return resp

Why this works: wait_exponential_jitter starts at 200 ms and doubles each attempt, capping at 2 s, with a random jitter up to the current wait. This spreads retries and prevents lock‑step storms.

Confirm: Run a unit test that forces a 502 response and prints the elapsed time between attempts.

python - <<'PY'
import time, requests
from retry_helper import post_with_retry, RetryableError

def mock_server():
    from http.server import BaseHTTPRequestHandler, HTTPServer
    class H(BaseHTTPRequestHandler):
        def do_POST(self):
            self.send_response(502)
            self.end_headers()
    srv = HTTPServer(('localhost', 8001), H)
    import threading; threading.Thread(target=srv.serve_forever, daemon=True).start()
    return srv

srv = mock_server()
start = time.time()
try:
    post_with_retry('http://localhost:8001/fraud', json={})
except RetryableError:
    pass
print(f"Total time: {time.time() - start:.2f}s")
PY
# Expected output: Total time: ~2.3s (200ms + ~400ms + ~800ms jitter)

2. How to store retry state for observability?

Log each attempt with a correlation ID and the back‑off delay. Use the standard logging module; ensure logs are shipped to your central system.

import logging, uuid

logger = logging.getLogger('payments')
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def post_with_retry(...):
    corr_id = uuid.uuid4()
    @retry(...)
    def _inner():
        logger.info("Attempt %s for %s", corr_id, url)
        ...
    return _inner()

Check the logs after a simulated failure; you should see three lines with increasing delays.

Introducing a circuit breaker

1. When should the breaker open?

If more than 5 % of the last 100 requests to the fraud service fail with 5xx or timeout, open the circuit for 30 seconds.

2. Implementation with pybreaker (v1.2.0)

# circuit.py
import pybreaker
import redis

r = redis.Redis(host='localhost', port=6379, db=0)

breaker = pybreaker.CircuitBreaker(
    fail_max=5,
    reset_timeout=30,
    state_storage=pybreaker.CircuitRedisStorage(r, 'fraud_cb')
)

def post_with_circuit(url: str, json: dict):
    @breaker
    def _call():
        return post_with_retry(url, json)
    return _call()

Why: CircuitBreaker tracks failures in Redis, so all instances of the service share the same state. When the breaker is open, _call raises CircuitBreakerError immediately, preventing any traffic to the downstream service.

Confirm: Simulate 6 consecutive 502 responses and watch the breaker open.

python - <<'PY'
from circuit import post_with_circuit, breaker
from retry_helper import RetryableError
import requests

# Force the breaker to see failures
for i in range(6):
    try:
        post_with_circuit('http://localhost:8001/fraud', {})
    except Exception as e:
        print(f"Attempt {i+1}: {type(e).__name__}")
print("Breaker state:", breaker.current_state)
PY
# Expected output:
# Attempt 1: RetryableError
# ...
# Attempt 6: CircuitBreakerError
# Breaker state: open

Rate‑limiting retries per client

1. Why limit retries?

Even with back‑off, a sudden surge of clients can still exceed downstream capacity. Rate limiting caps the total retry traffic.

2. Token‑bucket implementation using Redis

# limiter.py
import time, redis

r = redis.Redis(host='localhost', port=6379, db=0)
RATE = 100   # tokens per second
CAPACITY = 500

def allow_retry(client_id: str) -> bool:
    key = f"retry:bucket:{client_id}"
    now = int(time.time() * 1000)
    pipe = r.pipeline()
    pipe.hgetall(key)
    pipe.pexpire(key, 1000)  # keep key alive for 1 s
    data = pipe.execute()[0]

    tokens = int(data.get(b'tokens', CAPACITY))
    last = int(data.get(b'timestamp', now))

    # Refill tokens
    elapsed = now - last
    refill = (elapsed * RATE) // 1000
    tokens = min(CAPACITY, tokens + refill)
    if tokens < 1:
        return False

    # Consume a token
    tokens -= 1
    pipe.hmset(key, {'tokens': tokens, 'timestamp': now})
    pipe.execute()
    return True

Integrate into the request flow:

# app.py (excerpt)
from limiter import allow_retry
from flask import request, abort

@app.route('/pay', methods=['POST'])
def pay():
    client_id = request.headers.get('X-Client-ID')
    if not allow_retry(client_id):
        abort(429, description="Too many retries")
    # proceed to call fraud check via post_with_circuit(...)

Confirm: Run a script that fires 200 requests from the same client within one second; you should see 100 succeed and 100 receive 429.

python - <<'PY'
import requests, threading

def fire():
    r = requests.post('http://localhost:5000/pay', json={'amount':10}, headers={'X-Client-ID':'test'})
    print(r.status_code)

threads = [threading.Thread(target=fire) for _ in range(200)]
[t.start() for t in threads]
[t.join() for t in threads]
PY
# Expected: roughly half 200, half 429 responses

Build it

1. Add the retry helper
   
   Create retry_helper.py with the code from “Adding exponential back‑off with jitter”.
   
   Why: Centralises retry logic and guarantees back‑off.
   
   Confirm: Run the unit test shown earlier; total time should be ~2.3 s.

2. Integrate the circuit breaker
   
   Create circuit.py and replace direct requests.post calls in app.py with post_with_circuit.
   
   Why: Prevents traffic to a failing downstream service.
   
   Confirm: Simulate six 502 responses; the seventh call should raise CircuitBreakerError and return a 503 from your service.

3. Deploy Redis and the limiter
   
   Add the limiter.py file, update the /pay endpoint to call allow_retry. Ensure Docker Compose includes Redis (already running).
   
   Why: Caps total retry traffic per client, protecting downstream capacity.
   
   Confirm: Run the “200 requests” script; you should observe a mix of 200 and 429 status codes.

4. Add logging
   
   In retry_helper.py, ensure the logger is configured as shown. Deploy the updated code and verify logs contain correlation IDs and back‑off delays.
   
   Why: Observability for future incidents.
   
   Confirm: Trigger a retry storm (e.g., by temporarily blocking the fraud service) and grep logs for a single correlation ID; you should see three attempts spaced by increasing delays.

5. Smoke test the whole stack
   

   curl -X POST http://localhost:5000/pay -H "X-Client-ID: demo" -d '{"amount":5}'
   # Expect 200 on success, 429 if rate limit exceeded, 503 if circuit open.

Why: Guarantees the end‑to‑end path works.

Confirm: All responses match expectations under normal and failure conditions.

After these steps the service will no longer amplify a downstream glitch into a retry storm, and you will have metrics to detect the first signs of overload.

When this breaks

Symptom	Cause	Fix
RetryableError: Max retries exceeded after only 1 s	wait_exponential_jitter max value too low (default 1 s). The back‑off never exceeds the downstream timeout, causing rapid fire.	Raise max=5.0 in the wait_exponential_jitter decorator. Verify with the unit test that total time grows to ≈ 4 s.
All requests return 429 “Too many retries” even when the downstream is healthy	allow_retry bucket key TTL too short; tokens are not refilled because the key expires before the next request.	Increase pexpire to 2000 ms or store a permanent key and rely on timestamp diff for refill. Test by sending a burst, then a single request after 1 s – it should succeed.
Circuit breaker never opens despite repeated 5xx	fail_max set to 5 but pybreaker counts only exceptions, not HTTP status codes. The wrapper raises RetryableError, but the breaker decorates the outer function, swallowing the exception.	Wrap the inner request (post_with_retry) with the breaker, not the outer post_with_circuit. Re‑run the failure simulation; breaker should now open after 5 failures.
Redis connection errors (ConnectionError: Error 111 connecting to localhost:6379)	Redis container not running or bound to a different host.	Run docker compose up -d redis and ensure redis-cli ping returns PONG. Re‑start the service after confirming connectivity.
Log lines missing correlation IDs	Logger configured after import of retry_helper, causing the module‑level logger to use the default configuration.	Initialise logging before importing retry_helper (e.g., in app.py at the top). Verify by checking a fresh log entry.

What it costs

• Latency – Adding exponential back‑off and circuit‑breaker latency adds up to ~2 s per request in the worst case. For latency‑sensitive flows you may need a separate fast‑path that bypasses retries (e.g., for low‑value transactions).
• Complexity – Introducing Redis for rate limiting and circuit‑breaker state adds an external dependency; you must monitor Redis health and handle its failure gracefully (fallback to local in‑memory bucket).
• Operational overhead – You now have three new metrics to track: retry attempt latency, circuit‑breaker state, and rate‑limit rejections. Alerting on these adds to the on‑call load.
• Cost – Running a small Redis instance costs a few cents per hour on most cloud providers; the added CPU for the back‑off logic is negligible compared to the saved downstream capacity.

In environments where downstream services are always highly available, you might accept the risk of occasional spikes and skip the rate limiter, but for any public‑facing API the cost of a retry storm usually outweighs the added latency.

In the interview

Interviewer: “We have a microservice that calls an external credit‑score API. When the API briefly returns 502, we see a surge of retries that brings the whole system down. How would you design a defence?”

Weak answer: “I’d just add a time.sleep(1) before each retry.”

Why it’s weak: It shows no understanding of exponential back‑off, jitter, circuit‑breaker semantics, or how to bound retries across many instances.

Strong answer: “First, I’d replace the naïve retry loop with an exponential back‑off with full jitter (e.g., start at 200 ms, double each attempt, cap at 2 s). I’d also add a circuit breaker that opens after 5 % of the last 100 calls fail, staying open for 30 s, storing its state in a shared Redis so all instances act consistently. Finally, I’d rate‑limit retries per client using a token bucket in Redis to ensure we never exceed the downstream’s capacity. This combination limits the retry storm, gives the downstream time to recover, and provides observability via logs and metrics.”

Senior vs junior:

Junior should be able to explain back‑off and why jitter matters, and maybe sketch a simple circuit‑breaker.

Senior must discuss distributed state (Redis), the trade‑off of added latency, how to monitor the breaker, and how to handle Redis failure (fallback to local bucket).

Director/Engineering‑manager will probe the business impact: “What does the 2‑second worst‑case latency cost us? How do we measure the ROI of adding Redis?” and expect you to talk about cost‑benefit and risk mitigation.

Follow‑up that separates experience: “Can you walk me through a post‑mortem you wrote after a retry storm, and what metric you added to detect it earlier?” An experienced engineer will reference a specific alert (e.g., “retry‑rate > 500 rps”) and a concrete change (adding a Prometheus alert on circuit‑breaker open events).

Your tasks

1. Reproduce a retry storm
   
   Disable the fraud‑check endpoint (stop the mock server), trigger 10 k payment requests, and verify that the service logs show rapid retries and that the downstream CPU spikes (use top on the mock server).
   
   Done when you can see the 502 responses and the rapid log lines.

2. Apply exponential back‑off
   
   Implement retry_helper.py as shown, run the same load test, and confirm the downstream CPU never exceeds 30 %.
   
   Done when the logs show increasing delays and the mock server stays idle.

3. Introduce a circuit breaker
   
   Add circuit.py, configure fail_max=5, reset_timeout=30. Simulate 6 consecutive 502s and verify the seventh request returns a 503 from your service.
   
   Done when the breaker opens and you see CircuitBreakerError in logs.

4. Rate‑limit per client
   
   Deploy Redis, add limiter.py, and send 200 requests from a single client in one second. Verify you receive roughly 100 200 responses and 100 429 responses.
   
   Done when the distribution matches the token bucket limits.

5. Break and diagnose
   
   Change the Redis host to an invalid address, restart the service, and observe the failure mode. Use the “When this breaks” table to identify the symptom, cause, and fix.
   
   Done when you can recover the service by correcting the Redis connection and confirm the logs now show successful rate‑limit checks.

---

---

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.

Production failures: the things that page you

Chapter 3 of 24. New chapter every Friday morning.
Next: Connection pool exhaustion.

· The full syllabus and every chapter so far
· Subscribers also get the condensed notes for this chapter, the running
recap of everything the series has covered, and the extended guidance:
subscribe

---

Written by Amit Chakraborty — founding engineer and senior architect: React Native, AI and RAG systems, production architecture. Portfolio · LinkedIn · GitHub.

Need this built, reviewed or taught to your team? Get in touch or email [e-mail]. Available for senior and founding engineering roles, consulting and training, remote worldwide.

Comments

[Dhruv Malaviya]

"A retry is a new request, not a continuation" should head every retry wrapper ever written. Most treat it as a continuation and inherit none of the original request's budget.

Two additions. Backoff without jitter is half a fix , clients that all fail at 02:17 and all back off exponentially retry in lockstep at 02:18, 02:20, 02:24. The spike just gets scheduled.

And the control most teams are missing is a retry budget rather than a retry policy. A token bucket per caller bounds aggregate retry load regardless of how many independent clients decide to retry. Per-request backoff limits each caller; only a shared budget limits the total.

Worth noting the circuit-breaker interaction too , a breaker that opens mid-storm can block the recovery traffic that would have cleared it.