Postmortem: A Retry Without a Deadline Took Down Staging

You get paged at 02:14 because staging stopped answering health checks, and the graph looks like a cliff. Postgres connections sit at the pool ceiling, and every worker logs the same timeout you thought you had already handled. The deploy from yesterday still looks green in CI, which is the part that makes your stomach drop. You did not ship a new feature; you shipped a small retry an assistant drafted while you reviewed something else.

This write-up is a reconstructed incident you can replay on a laptop, not a claim about one company's private outage. Treat the timestamps as a teaching tape, then run the commands against a copy of the logs you actually own. The useful part is the durable fix: a retry helper with a deadline, jitter, and a test that fails when those guards disappear. If you only remember one thing, remember that a polite retry is still a load generator once many workers share one database.

What you thought you merged

The change looked like hygiene. A flaky upstream had been throwing ReadTimeout, and the suggested patch wrapped the call in exponential backoff that "would settle down on its own." You skimmed the function, saw the familiar 1-2-4 pattern, and let CI do the talking. Nothing in the diff touched schema, routes, or pool size, which is why the review felt cheap.

Here is the reconstructed helper, labeled as an example you should not copy into a shared environment. It compiles, it passes a single-process unit test, and it is still how you knock staging over.

# Example only — reconstructed bad retry. Do not point this at shared staging.
import time
import random

def fetch_with_retry(call, attempts=8):
    delay = 0.25
    last_exc = None
    for _ in range(attempts):
        try:
            return call()
        except TimeoutError as exc:
            last_exc = exc
            time.sleep(delay)
            delay *= 2  # no jitter, no budget, no unique cap
    raise last_exc

A single worker looks patient. Twenty workers that miss the same 400ms blip look like a metronome. Each sleep lands on the same beat, and the database meets a chorus of reconnects instead of a drizzle. That is the difference between a retry and a stampede, and it does not show up in a test that never runs concurrently.

Timeline you can reconstruct from logs

Start with the health-check gap rather than the first exception, because the first exception is usually a symptom. You want the minute when the pool stopped returning connections, then walk backward through retries, deploys, and the upstream blip. Keep the commands boring so another engineer can replay them without your laptop lore.

# Reconstruct the cliff. Adjust paths to logs you are allowed to read.
awk '$0 ~ /02:1[0-9]/' /var/log/app/staging.log | grep -E 'timeout|retry|pool|health'

# Count retry lines per second around the page.
awk '$0 ~ /retrying after/ {print $1,$2}' /var/log/app/staging.log \
  | awk -F: '{print $1":"$2":"int($3)}' \
  | uniq -c | sort -n

On the reconstructed tape, 02:08 shows a brief upstream timeout and a normal recovery. At 02:09 the new helper starts doubling sleep across every web worker, and the retry lines per second climb in lockstep instead of spreading out. By 02:12 the pooler reports remaining: 0, and health checks fail because workers are blocked on connect(), not on application code. The deploy itself is innocent in git blame; the behavior change is the missing deadline.

You will be tempted to restart the pooler and call the night finished. That clears the symptom the way opening a window clears smoke, and it teaches you nothing about the stove. Leave one worker process up long enough to capture a stack, then snapshot the retry counters before you bounce anything.

# One stack from a stuck worker beats a dozen opinions in Slack.
ps aux | grep gunicorn
# Example: replace PID after you identify a blocked worker.
sudo py-spy dump --pid 18422

If the stacks sit in time.sleep or in psycopg connect, you are not looking at an application deadlock. You are looking at coordinated delay, which is a social problem among processes that all trust the same clock.

Contributing factors, not a villain

The assistant did not invent a new class of outage. Coordinated retry is an old failure mode, and it shows up whenever backoff lacks jitter and a wall-clock budget. What changed in 2026 is how cheap it feels to merge that class of code while you are also reviewing a real feature. The patch sounded like engineering because it used the word retry, and the test suite sounded like proof because it never shared a pool with twenty friends.

Three conditions had to line up. The upstream blip was real and short, so the first timeout was honest. The helper had no deadline= and no jitter, so recovery became synchronization. Staging used a small connection ceiling that production would have padded, so the stampede became visible here first. None of those conditions is exotic, which is why the incident is worth writing down.

Think of it as a marching band that all turn the same corner because the drummer only knows powers of two. One drummer is percussion. Twenty drummer clones are a parade, and the parade does not care that each clone intended to be polite.

Reproduce it somewhere that cannot hurt anyone

Do not replay a stampede against the staging database you just recovered. You need a throwaway process, a tiny pool, and a fake upstream that times out on cue. The artifact below is a local reproduction you can run in one terminal; it is deliberately small so the failure is obvious in under a minute.

# replay_stampede.py — local reproduction, not a load test against real staging.
import concurrent.futures
import time
from threading import Barrier

POOL = 4  # intentionally tiny, like a cramped staging cap
in_use = 0
lock_err = []

def flaky_call():
    time.sleep(0.05)
    raise TimeoutError("upstream blip")

def worker(barrier):
    global in_use
    barrier.wait()
    delay = 0.05
    for _ in range(6):
        if in_use >= POOL:
            lock_err.append("pool exhausted")
            return
        in_use += 1
        try:
            flaky_call()
        except TimeoutError:
            time.sleep(delay)  # synchronized on purpose
            delay *= 2
        finally:
            in_use -= 1

if __name__ == "__main__":
    barrier = Barrier(12)
    with concurrent.futures.ThreadPoolExecutor(max_workers=12) as pool:
        list(pool.map(lambda _: worker(barrier), range(12)))
    print("exhausted_events", len(lock_err))
    raise SystemExit(0 if lock_err else 1)

Run it with python3 replay_stampede.py and you should see exhausted_events well above zero. That exit is the incident in miniature: more workers than slots, the same sleep schedule, no jitter to desynchronize them. If the script ever prints zero, your reproduction drifted, and you should not trust the later fix.

When your laptop is a poor stand-in for the real process layout, move the same script onto an isolated box you do not share with other teams. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option as a scratch environment for drafting the failing replay and running it away from shared staging, without treating that option as a production control plane or as a promise about capacity.

A free scratch box is only doing its job if a mistake there cannot page anyone else. Copy logs in, run the replay, and throw the machine away when the test is green. That is the whole relationship between the incident and the tool.

Durable fix: budget, jitter, and a test that refuses the old helper

The repair is not "retry less forever." You still want a second chance after a 400ms blip. You want a wall-clock budget, a spread in the sleep, and a hard cap that does not care how confident the caller feels. The helper below is still an example, but it encodes the three guards the incident actually needed.

# Example fix — deadline plus jitter. Keep tests that freeze time nearby.
import random
import time

def fetch_with_budget(call, budget_s=2.0, cap_s=0.4):
    start = time.monotonic()
    delay = 0.05
    last_exc = None
    while time.monotonic() - start < budget_s:
        try:
            return call()
        except TimeoutError as exc:
            last_exc = exc
            remaining = budget_s - (time.monotonic() - start)
            if remaining <= 0:
                break
            sleep_for = min(cap_s, delay) * (0.5 + random.random())
            time.sleep(min(sleep_for, remaining))
            delay = min(cap_s, delay * 2)
    raise TimeoutError("budget exceeded") from last_exc

Jitter is the unglamorous part, and it is the part reviews skip because it looks like noise. The noise is the point: you are breaking the marching band so the database hears a crowd instead of a drumline. The budget is the adult in the room, because an infinite series of polite sleeps is still an infinite hold on a pool slot.

Now pin the behavior with a test that would have failed the original helper. Freeze randomness only where you assert spread, and keep a concurrent case that asserts the pool never goes negative or stuck. If you only test a single call, you are back where the incident started.

# test_retry_budget.py — fail the merge if the deadline disappears.
import time
from retry_budget import fetch_with_budget

def test_budget_stops_within_bound():
    calls = {"n": 0}

    def always_timeout():
        calls["n"] += 1
        raise TimeoutError("still down")

    start = time.monotonic()
    try:
        fetch_with_budget(always_timeout, budget_s=0.3, cap_s=0.1)
        raised = False
    except TimeoutError:
        raised = True
    elapsed = time.monotonic() - start
    assert raised
    assert elapsed < 0.6  # wall clock, not attempt count
    assert calls["n"] >= 1

Run python3 -m pytest test_retry_budget.py -q after you place the helper in retry_budget.py. The test is not clever, and that is why it survives the next assistant session. If someone deletes budget_s to "simplify," the elapsed assertion should fail on a cold CPU without needing staging as a witness.

After you merge the helper, add one operational breadcrumb: a metric for retry_budget_exhausted_total next to upstream_timeout_total. Pages should fire on pool waiters, not on the first timeout, or you will train yourself to ignore the only signal that mattered at 02:12.

Limitations, and who should not use this replay

This approach is for application-level retry storms you can copy into a throwaway process. It is the wrong tool when the incident involves unknown PII in the logs, a lock inside a managed database you cannot clone, or a failure that only appears across availability zones. A twelve-thread script will not teach you about disk saturation, DNS, or a vendor outage, and pretending otherwise wastes the next night.

Skip the free scratch-server path if your company forbids sending incident logs to any third-party host, even a temporary one. Skip the assistant-drafted retry entirely if you cannot explain the sleep math out loud in one sentence. And skip jittered backoff as a cure-all when the correct fix is to stop calling the upstream from the hot path, because no budget will save a synchronous call that should have been a queue.

You should also refuse this workflow as a substitute for capacity planning. A tiny staging pool made the bug visible; a huge production pool would have hidden it until a real blip arrived with more workers. The test above guards the helper, not your instance size, and those are different jobs.

If you replay the stampede at all, keep the target local or clearly disposable, keep the disclosure in the postmortem, and keep the conclusion boring: retries need a deadline the way loans need a due date. The rest is commentary.

When you want a quiet box for that replay instead of borrowing shared staging, MonkeyCode's free server option is one place to run the script above and then delete the machine.