Anatomy of a Retry Storm: A Postmortem That Changes Code

At 02:14 the queue depth graph went vertical, and by 02:40 the invoice graph had followed it. Nothing in the application logs said "outage" — every worker was healthy, every health check green, and every one of them was happily retrying the same four hundred failed calls. The thing that actually broke was not the model provider and not the network; it was the fact that three independent retry layers had been composed by people who each assumed the other two did not exist. What follows is the postmortem format that turns that night into a code change instead of a paragraph everyone nods at.

A recap tells the story of the night, and a postmortem tells you which design decision made the night possible. If you cannot point at a diff that would have prevented the incident, you have written a recap with timestamps. The template I keep has four sections: a timeline in absolute UTC with the trigger separated from the detection, a list of contributing factors ranked by how cheaply they could have been removed, the quantitative amplification of the failure, and a durable fix with an owner and a test that fails today.

Here is the timeline shape, filled with the composite walkthrough used throughout this article. Treat it as a format to copy, not as a claim about any specific company's outage.

Time (UTC)	Event	Signal
01:58	Upstream provider begins returning 429 on one model	error rate 0.4% → 3%
02:06	Job-level retry re-enqueues failed jobs	queue depth flat, attempts +1
02:14	SDK-level retry multiplies in-flight calls	in-flight ×3
02:22	Client-level retry adds a third multiply	in-flight ×9
02:31	Circuit never opens; budget alarm fires	spend 11× baseline
02:40	Manual stop of the worker pool	queue frozen
03:15	Drain with idempotency keys, no duplicate writes	backlog cleared

The arithmetic is the part people skip, so put it in the document as a number. One logical call passed through three retry layers at three attempts each, which is twenty-seven HTTP requests per unit of work; four hundred backlogged jobs became roughly eleven thousand requests in under twenty minutes. None of those retries was wrong in isolation, and that is precisely why the factor ranking matters more than the blame ranking. The cheapest removal was a single retry layer, and the second cheapest was a budget ceiling that refuses to spend rather than a dashboard that reports spending.

The durable fix is small enough to live in one file, and it needs to be boring because you will read it at 2am. The guard below does three things: it charges the budget pessimistically before the call instead of after, it opens a circuit after consecutive failures, and it derives an idempotency key from the payload so a retried job cannot write twice. The example is illustrative and has not been run in your environment, so dry-run it against a fake transport before wiring it to anything that costs money.

# guard.py — one retry layer, a hard budget ceiling, and an idempotency key
import hashlib, random, time
from dataclasses import dataclass

class BudgetExceeded(Exception): ...
class CircuitOpen(Exception): ...

@dataclass
class CallGuard:
    cap_usd: float
    fail_threshold: int = 5
    cooldown_s: float = 120.0
    spent_usd: float = 0.0
    failures: int = 0
    opened_at: float | None = None

    def check_circuit(self) -> None:
        if self.opened_at is None:
            return
        elapsed = time.monotonic() - self.opened_at
        if elapsed < self.cooldown_s:
            raise CircuitOpen(f"open, retry in {self.cooldown_s - elapsed:.0f}s")
        self.opened_at, self.failures = None, 0

    def charge(self, estimated_usd: float) -> None:
        projected = self.spent_usd + estimated_usd
        if projected > self.cap_usd:
            raise BudgetExceeded(f"cap {self.cap_usd} vs projected {projected:.4f}")
        self.spent_usd = projected

    def record(self, ok: bool) -> None:
        self.failures = 0 if ok else self.failures + 1
        if self.failures >= self.fail_threshold and self.opened_at is None:
            self.opened_at = time.monotonic()

def idem_key(job_id: str, model: str, prompt: str) -> str:
    digest = hashlib.sha256(f"{model}\x00{prompt}".encode()).hexdigest()[:16]
    return f"{job_id}:{digest}"

def backoff(attempt: int, base: float = 0.5, cap: float = 20.0) -> float:
    return min(cap, base * 2 ** attempt) * random.uniform(0.5, 1.5)

The call site becomes the only place that retries, which is the actual durable fix. Pre-charging is deliberately pessimistic: you charge the estimated cost, and on success you refund the difference once the provider reports usage. That inversion matters because a post-hoc meter can always be outrun by a storm, while a pre-charge stops the storm at the ceiling. If your ceiling is hit, the job goes to a dead-letter queue with its idempotency key intact, and replay is safe.

def run_job(guard: CallGuard, job_id: str, key: str, fn, attempts: int = 3):
    for attempt in range(attempts):
        guard.check_circuit()
        guard.charge(estimated_usd=0.002)
        try:
            result = fn(idempotency_key=key)
            guard.record(True)
            return result
        except CircuitOpen:
            raise
        except Exception:
            guard.record(False)
            if attempt == attempts - 1:
                raise
            time.sleep(backoff(attempt))

Now the uncomfortable part of the postmortem: not every call in that queue deserved a paid frontier model. Roughly two thirds of the backlog was classification and extraction work — routing tags, pulling fields out of a log line — and that class of work is exactly where free model access changes the economics of an incident. This is where MonkeyCode entered the fix for this workflow rather than the diagnosis: its free model tier handles the low-stakes bulk path, and the free server option hosts the drain worker that replays the dead-letter queue even when the paid pool is frozen. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator advertises a free token allowance alongside the free server, and you should confirm the current number on the site, because allowances on any free tier move.

The decision table is what keeps the fallback honest, because a free tier used indiscriminately is just a second way to fail. Classify by blast radius and by whether a wrong answer is recoverable, not by how impressive the prompt looks.

Workload	Wrong answer costs	Route	Why
Log line tagging	A re-run	Free tier	Cheap to redo, high volume
Field extraction for search	A bad filter	Free tier, then verify	Deterministic schema check catches most misses
Customer-facing summary	Trust	Paid, guarded	Not recoverable after the user reads it
Anything with PII	Compliance	Neither without review	Data handling is your call, not the tier's
Billing-adjacent decisions	Money	Do not automate	Needs a human in the loop

Before you trust any of this, write three tests that fail against the current code, because a postmortem without a failing test is a wish. The first is a kill-switch test: force five consecutive transport errors and assert that the sixth call raises CircuitOpen without touching the network. The second is a duplicate-delivery test: deliver the same job twice with the same idempotency key and assert exactly one write downstream. The third is a ceiling test: set cap_usd to a value the first call can afford and the second cannot, then assert the second raises BudgetExceeded before the transport is invoked. Run all three in CI, and the incident stops being a story you retell and becomes a property the build enforces.

Limitations belong in the document too, or the next reader will over-trust the fallback. A free tier is a shared resource with its own rate limits and its own quality ceiling, so latency guarantees and hard SLAs do not belong there, and neither does anything a regulator would ask you to explain. If your workload is customer-facing, latency-sensitive, or bound by a data-processing agreement, the free path is the wrong tool and the guarded paid path plus a queue is the right one. You should also assume the allowance changes and design the guard so that losing the free path degrades throughput rather than correctness.

The last section of the template is one sentence, and it is the only part that decides whether the document worked. Write down the single diff that would have prevented the incident, ship it with a test, and delete the rest of the prose until only that remains. If you want a place to run the drain worker while you write that diff, the free server option is a reasonable sandbox for exactly this kind of rehearsal.