Postmortem: the agent that spent $200 retrying a 400

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

An agent I was running burned roughly $200 overnight retrying an HTTP 400 — a bad request, the one status code that means "sending this again will fail in exactly the same way." Nothing crashed. No component was individually buggy. Every layer did what it was told: it saw an error and retried, politely, with backoff. The bill was the sum of a dozen reasonable local decisions with no one bounding the total.

This is the incident teardown. The root cause is one idea — retryability is a property of the specific error, not a default you apply to all of them — and the reason it got expensive is a second one: nested retry caps multiply, and a per-step cap bounds a step, never a run and never a fleet. The retry-budgets post worked out the cost of retrying transient failures in detail, modeling how recovery architecture controls the multiplier. This is what happens when the failure isn't transient and you retry it anyway — when the architecture assumes retrying is always safe.

The incident

The agent was an overnight batch job: pull a queue of records, and for each one call an internal enrichment API (a tool) that adds a few fields. It had run clean for weeks. That evening the enrichment service shipped a schema change — one field went from optional to required-and-typed. Records that omitted it now got a 400 Bad Request instead of a 200.

About 40% of the queue was missing that field. So 40% of the calls started returning a 400 that would never stop returning a 400, because the request itself was now malformed. The agent's response to an error was to retry it.

The timeline, reconstructed from logs:

• ~01:00 — the poisoned records start hitting the tool. Each one begins retrying.
• 01:00–07:00 — twelve workers grind in parallel, each one stuck re-attempting doomed calls, each attempt separated by a polite exponential backoff so the storm is slow rather than instant. Nobody's awake. No alert fires, because the only cost alarm was a daily aggregate evaluated at 9am.
• ~08:50 — I check the provider dashboard for an unrelated reason and see the day's spend line already vertical.
• ~09:00 — kill the job. Final damage: $198 and change, and the queue wasn't even finished. Title rounds up.

Zero records were enriched by any of that spend. It was pure waste, and it was waste the system was designed to produce given a permanent error.

Root cause: a 400 is not a 500

Here is the retry wrapper, lightly anonymized. Read it and the bug is right there in the except.

import time

def call_tool_buggy(client, payload, retries=5, backoff=1.0):
    for attempt in range(retries):
        try:
            r = client.post("/v1/enrich", json=payload)
            r.raise_for_status()
            return r.json()
        except Exception as e:          # <-- every error is treated as retryable
            if attempt == retries - 1:
                raise
            time.sleep(backoff * 2 ** attempt)   # backoff makes the storm last hours

raise_for_status() turns any 4xx or 5xx into an exception, and except Exception swallows all of them identically. But a 500, a timeout, and a 400 are three different claims about the world:

• A 500 or a timeout says the server failed to handle a valid request. Transient. Retrying is correct — the next attempt might land on a healthy replica.
• A 429 says slow down. Retryable, but only if you honor Retry-After.
• *A 400 (or 404, 422) says the request is wrong. Deterministic. The server evaluated your input and rejected it. Sending byte-identical input again is defined to produce the identical rejection. Retrying it isn't optimistic; it's a guarantee of wasted spend.

The default was backwards. The wrapper treated retry as the rule and success as the exception, when for client errors the opposite holds. The fix is to make the retryable set an allowlist and everything else terminal:

import time, httpx

RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}

class TerminalToolError(Exception):
    """The request is wrong; retrying it will fail identically. Do not retry."""

def call_tool(client, payload, retries=5, backoff=1.0):
    for attempt in range(retries):
        try:
            r = client.post("/v1/enrich", json=payload)
            r.raise_for_status()
            return r.json()
        except httpx.HTTPStatusError as e:
            status = e.response.status_code
            if status not in RETRYABLE_STATUS:
                raise TerminalToolError(f"{status}: {e.response.text[:200]}") from e
            if attempt == retries - 1:
                raise
            wait = _retry_after(e.response) or backoff * 2 ** attempt
            time.sleep(wait)
        except (httpx.TimeoutException, httpx.TransportError):
            if attempt == retries - 1:
                raise
            time.sleep(backoff * 2 ** attempt)   # transport failures *are* transient

Default-terminal, not default-retry. A 400 now fails on the first attempt instead of the fifth. That one change would have cut the tool-layer waste by 5×. It was not, by itself, enough — because the tool layer wasn't the only thing retrying. This is a concrete example of the distributed retry patterns problem: nested caps multiply when there's no upper bound on the total.

---

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article