The Retry Storm That Delayed a Report by Six Hours

You open the dashboard at 08:10 and the nightly summary column is empty, yet every job in the run history is green. Nothing paged, nothing failed, and forty thousand rows sit there with "summary": "" in them. You already know the shape of the answer before you finish scrolling: the pipeline did not crash, it gave up one item at a time and then reported success.

The timeline below is a reconstructed worked example rather than one company's log, so treat the clock as a template to fill in with your own timestamps. The code is the part that reproduces, and the ranking of contributing factors is the part worth arguing about with your team. If your nightlies call a metered model API, the shape will feel familiar.

The timeline, rebuilt from logs

Time	What the system did	What you could see
01:00	Batch starts, 32 concurrent workers, one shared credential	Job status: running
01:12	First 429s, retried inside the worker	Nothing, because retries are not events
01:40	Retries outnumber successes, queue depth climbs	Queue length is not on any dashboard
02:05	Worker hits its exception branch, writes an empty summary, marks the item ok	Item status: ok
03:30	The pool is fully occupied by sleeps, each item waits minutes	Still "running"
06:15	Batch completes, the report builds from empty rows	Job status: success
08:10	A human opens the dashboard	The only real detector

A retry is invisible by design, which is why the first twelve minutes of this incident look identical to a healthy run in every log you keep. By the time the 429s are frequent enough to matter, the pool is already spending its capacity waiting instead of working. The metric that would have caught it at 01:40 is retries per successful item, and almost nobody exports that number.

Contributing factors, ranked by how much they cost

The shared credential comes first, because the nightly batch and the interactive preview were using the same API key. That meant the 01:00 burst consumed the headroom humans needed when they arrived at nine, and the two workloads had no owner in common to notice the collision. Rate limits are a shared budget, and a key that fans out to three callers will always be spent by whichever caller is loudest at night.

The retry policy comes second. Exponential backoff without jitter turned 32 workers into a metronome, so they retried at the same instants and the provider saw synchronized waves instead of a smooth trickle. You do not have a retry storm because your retries are too aggressive; you have one because they are too polite and too synchronized at the same time.

The catch-all comes third, and it is the most expensive line in most pipelines: except Exception: return {"summary": ""}. An empty string passes schema validation, renders fine in a table, and never trips an alert, so the failure quietly became data. Every pipeline that reports success while holding a placeholder is one night away from this exact morning.

Finally, there was no dead-letter path. Re-running the batch meant redoing all forty thousand items at full cost, which is why the honest fix on the second day was a full backfill rather than a targeted replay of the eleven percent that actually failed.

Fix one: reserve the budget before you spend it

The guard below fails closed at a soft cap that keeps roughly fifteen percent of the day's budget for humans and retries. It records spend before the call instead of after, because a request that times out at sixty seconds has already cost you tokens. Set DAILY_TOKENS from your plan's current published terms and never from a number you remember.

# budget_guard.py — stop the batch before it can consume the day's headroom
import json, os, pathlib, time

BUDGET_FILE = pathlib.Path(os.environ.get("BUDGET_FILE", "./quota_budget.json"))
DAILY_TOKENS = int(os.environ["DAILY_TOKENS"])      # copy from the plan's current terms
RESERVE_RATIO = float(os.environ.get("RESERVE_RATIO", "0.15"))

def reserve(estimated_tokens: int) -> None:
    today = time.strftime("%Y-%m-%d")
    state = json.loads(BUDGET_FILE.read_text()) if BUDGET_FILE.exists() else {"day": today, "spent": 0}
    if state["day"] != today:
        state = {"day": today, "spent": 0}
    soft_cap = DAILY_TOKENS * (1 - RESERVE_RATIO)
    if state["spent"] + estimated_tokens > soft_cap:
        raise SystemExit(f"budget stop: {state['spent']} + {estimated_tokens} > soft cap {soft_cap:.0f}")
    state["spent"] += estimated_tokens
    BUDGET_FILE.write_text(json.dumps(state))

A file is enough for a single host, and it is the wrong tool the moment you have two workers on two machines, so move the counter into whatever you already run for coordination. The point is not the storage, it is the ordering: reserve, then call, then reconcile. Once the guard raises, your batch stops at a predictable boundary instead of discovering the boundary at 03:30 with half a report left.

Fix two: bounded retries with jitter, and a loud failure

The wrapper below makes three deliberate decisions. It retries only the status codes that mean "try again later", it caps attempts so a persistent failure cannot loop, and it raises rather than returning something shaped like success. The caller marks the item failed and pushes the original payload into a dead-letter table for replay.

import asyncio, os, random, httpx

RETRYABLE = {429, 500, 502, 503, 504}

async def call_model(client: httpx.AsyncClient, payload: dict, *, attempts: int = 4) -> dict:
    for attempt in range(1, attempts + 1):
        r = await client.post(f"{os.environ['MODEL_BASE_URL']}/chat/completions",
                              json=payload, timeout=60)
        if r.status_code not in RETRYABLE:
            r.raise_for_status()
            return r.json()
        if attempt == attempts:
            raise RuntimeError(f"retry budget exhausted after {attempt} attempts, last={r.status_code}")
        ceiling = min(30.0, 1.5 * 2 ** (attempt - 1))
        await asyncio.sleep(random.uniform(0, ceiling))   # full jitter

Full jitter spreads the herd across the whole window instead of stacking every worker on the same recovery instant. Combined with the budget guard, an overloaded night now ends with a partial report, a dead-letter count above zero, and an alert, which is a far better outcome than forty thousand confident empty strings. The empty string is the disease; the retry storm is only the fever.

Fix three: replay the night in ninety seconds

You cannot test this fix against a live provider at three in the morning, so capture a night as fixtures and replay it whenever you change the policy. Dump the raw inputs and the recorded responses once, keep them in the repository, and run the pipeline in dry-run mode against them in CI.

# capture last night as fixtures, then replay it as often as you like
mkdir -p fixtures/nightly replay
python -m pipeline.dump --date "$(date -d yesterday +%F)" --out fixtures/nightly
for f in fixtures/nightly/*.json; do
  python -m pipeline.enrich --input "$f" --out "replay/$(basename "$f")" --dry-run
done

Write three assertions and keep them boring. No completed item may carry an empty or placeholder summary, retries per item may not exceed the configured cap, and estimated token spend must stay under the soft cap for the fixture set. Add a fourth that injects a 429 on every third call, and you have a regression test for the exact night you just spent eight hours reconstructing.

Where a free tier keeps the guard cheap

The classification pass in front of the expensive model does not need to be clever, and the sentinel that checks spend does not need to be fast. That is the part of this workflow where a free tier genuinely participates rather than decorating the conclusion: MonkeyCode offers free model access and a free server option, which is enough capacity to run the cheap tier and to host the sentinel plus a small dead-letter inspector. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Be honest about the shape of that dependency. Free-tier quotas and server specifications change, so read the current terms and set DAILY_TOKENS from them rather than from a number in an old post, and do not route anything through free capacity if it must finish before a fixed morning deadline. If the sentinel itself lives on free capacity, give it an external heartbeat from a different system, because a guard that can be interrupted needs something outside itself to notice. Treat the free tier as the place where the cheap, idempotent, replayable work runs, and keep the irreversible step behind your paid commitment.

Who should not adopt this

If your batch runs on a single worker and a human reads every row before it ships, the timeline above cannot happen to you and the budget guard is ceremony. If you have contractual uptime or data residency obligations attached to this pipeline, do not put any part of it on free capacity, including the monitoring. And if you cannot write down an estimate of tokens per item, fix that measurement first, because every number in this design depends on it.

The part that outlives the incident

The durable fix was not the jitter, although the jitter helped. It was deciding that rate limits are a shared budget with a named owner, that an empty summary is a failure rather than a value, and that a retry which nobody counts is indistinguishable from a request that never happened. Start with the reserve() guard and the fixture replay, because those two together turn next week's outage into a red test instead of a Tuesday morning surprise.