Postmortem: When Agent Retries Hit the Paid Endpoint

You notice the bill before you notice the bug, which is rarely how you wanted the morning to start. The overnight agent job was supposed to classify a handful of failing tests, rewrite two fixtures, and then stop. Instead you open the provider dashboard at 09:14 and find a retry storm that treated every timeout as another paid call. The original task is still red, and the invoice is the only artifact that looks complete.

This write-up is not about a model that cannot write code, but about an agent that never learned which calls cost money. You should treat the timeline below as a reconstructed incident you can replay, not as a claim about one vendor outage. The durable fix is a fail-closed gate, a free unpaid route, and an audit log that still makes sense after the next retry storm.

The night the loop stopped being cheap

At 01:07 the scheduler starts agent-classify against a staging git worktree, with MODEL_ENDPOINT inherited from a laptop profile that was never meant for unattended jobs. At 01:09 the first tool call times out because the test runner is cold, and the framework retries the entire reasoning step instead of the tool. By 01:11 you already have four billed completions for a job that has not yet read a single assertion message, which is the moment the incident actually begins.

At 01:18 the agent decides the fixtures are probably flaky and asks the same paid model to rewrite them from memory. It does not re-read the failure output, because the retry wrapper cached an empty stdout blob and then treated that blob as evidence. At 01:31 a second worker, spawned by a health check that only wanted a heartbeat, joins the same queue and photocopies the loop. You notice none of this until 09:14, when finance Slack lights up and the staging tests are still failing on the same off-by-one.

If that sequence feels familiar, it is because agent runtimes optimize for keep going and cloud SDKs optimize for call the default client. Neither component wants to be the adult in the room when a timeout turns into a second invoice line. You have to put a gate between them, and that gate has to survive retries, forked workers, and copied environment files that look harmless at 01:07.

Contributing factors, without the mythology

The process inherited a production API key because .env was sourced by the same direnv hook you use for manual debugging on a Tuesday afternoon. The agent did not choose the paid model in any meaningful sense; it constructed the official client helper, and the helper did what helpers do. That is less a villain origin story than a missing seatbelt, and you should write the fix as if the next intern will copy the same hook.

Retry amplification did the rest of the damage, the way a photocopier with the lid open keeps printing the same expensive sheet. A single flaky tool call is annoying, but a completion that embeds the tool call inside one billed prompt turns every timeout into a full-price mulligan. Staging, local experiments, and production also looked like the same bearer token on the wire, so the provider logs had no room numbers when you later tried to write a deny rule.

None of those factors require a smarter model, and chasing a frontier upgrade would have hidden the leak for another week. They require a cheaper sandbox for noisy work, a loud failure when that work tries to escape, and an identity on every request that still exists after the worker exits. The rest of this postmortem is that sandbox, written so you can fail the next incident in a test file instead of a finance channel.

The durable fix is a budget gate, not a pep talk

You stop letting the agent construct the real client, which is the only change that still works when a retry library gets creative. You give it a tiny proxy that reads an allowlist, stamps every request with a purpose, and refuses paid routes unless that purpose is explicit and human-approved. Exploratory retries, fixture drafts, and think about the traceback steps go to a free model endpoint, while only a labeled job may touch the paid key.

The example below is a worked gate you can read in one sitting, not a library to paste into production without tests. It fails closed when the paid flag is missing, and it writes an audit line even when the unpaid route is the one that answered. Keep the prompt bodies out of the log unless you already have a retention story, because tomorrow’s helpful patch will try to store them.

# budget_gate.py — example gate, not a production SDK
from __future__ import annotations

import json
import os
import time
import urllib.request
from dataclasses import dataclass

PAID_PURPOSES = frozenset({"release-summary", "customer-facing-draft"})

@dataclass(frozen=True)
class Route:
    purpose: str
    paid: bool
    endpoint: str

class BudgetDenied(RuntimeError):
    pass

def resolve_route(purpose: str) -> Route:
    purpose = purpose.strip()
    free_endpoint = os.environ.get(
        "FREE_MODEL_ENDPOINT",
        "http://127.0.0.1:8080/v1/chat/completions",
    )
    paid_endpoint = os.environ.get("PAID_MODEL_ENDPOINT", "")

    if purpose in PAID_PURPOSES:
        if os.environ.get("ALLOW_PAID_MODELS") != "1":
            raise BudgetDenied(
                f"purpose {purpose!r} needs ALLOW_PAID_MODELS=1"
            )
        if not paid_endpoint:
            raise BudgetDenied("paid endpoint is not configured")
        return Route(purpose=purpose, paid=True, endpoint=paid_endpoint)

    return Route(purpose=purpose, paid=False, endpoint=free_endpoint)

def complete(prompt: str, purpose: str) -> str:
    route = resolve_route(purpose)
    payload = {
        "model": os.environ.get("MODEL_NAME", "local"),
        "messages": [{"role": "user", "content": prompt}],
        "metadata": {"purpose": route.purpose, "paid": route.paid},
    }
    req = urllib.request.Request(
        route.endpoint,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    started = time.time()
    with urllib.request.urlopen(req, timeout=30) as resp:
        body = json.loads(resp.read().decode("utf-8"))
    _audit(route, started, prompt)
    return body["choices"][0]["message"]["content"]

def _audit(route: Route, started: float, prompt: str) -> None:
    line = {
        "ts": time.time(),
        "elapsed_ms": int((time.time() - started) * 1000),
        "purpose": route.purpose,
        "paid": route.paid,
        "endpoint": route.endpoint,
        "prompt_chars": len(prompt),
    }
    path = os.environ.get("AGENT_AUDIT_LOG", "agent-audit.jsonl")
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(line) + "\n")

The interesting part is not the HTTP client, which you can replace with whatever SDK your agent already drags in. The interesting part is that complete("rewrite the fixture", purpose="retry") can never see the paid key, even if a retry library calls it fifty times before breakfast. You force paid work to name itself, and you keep that name out of the default retry path so a health-check worker cannot photocopy a bill.

A small test makes the rule durable instead of tribal knowledge that lives in one person’s head. If the test ever needs a live network, the gate has already failed, because money protection that requires Wi-Fi is just another implicit default. Run it in CI on a box that has never seen the paid key, then run it again on the worker image that starts at 01:07.

# test_budget_gate.py — unexecuted until you run pytest locally
import budget_gate
import pytest

def test_retry_purpose_never_resolves_paid(monkeypatch):
    monkeypatch.setenv("FREE_MODEL_ENDPOINT", "http://127.0.0.1:9/free")
    monkeypatch.delenv("ALLOW_PAID_MODELS", raising=False)
    route = budget_gate.resolve_route("retry")
    assert route.paid is False
    assert "127.0.0.1" in route.endpoint

def test_paid_purpose_fails_closed_without_flag(monkeypatch):
    monkeypatch.delenv("ALLOW_PAID_MODELS", raising=False)
    with pytest.raises(budget_gate.BudgetDenied):
        budget_gate.resolve_route("release-summary")

export FREE_MODEL_ENDPOINT="http://127.0.0.1:8080/v1/chat/completions"
unset ALLOW_PAID_MODELS
unset PAID_MODEL_ENDPOINT
python -m pytest test_budget_gate.py -q

Where the unpaid route should live

You still need somewhere for the unpaid route to land when the laptop is asleep and the agent is not. A local process is fine for one developer chasing one traceback, but a shared worker needs a server that can disappear without taking the paid account with it. That is the crash-test dummy for prompts: you would not debug airbags by driving the company car into a wall, and you should not debug retries by driving them into a billed API.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only because it offers free model access and a free server option you can point FREE_MODEL_ENDPOINT at while the gate is doing its job. This article does not attach model names, quotas, hardware sketches, or duration promises, because those change and a postmortem should not depend on a brochure.

The workflow stays small on purpose. You stand up the free server, you point the unpaid route at it, you keep ALLOW_PAID_MODELS unset in the agent unit, and you export the paid key only in a separate human-triggered job. If the free path is slower or clumsier than the paid one, that is acceptable during incident rehearsal, because the sandbox is there to be loud and cheap while you teach the agent to stop assuming.

Reading the next incident from the log, not the invoice

After the gate is in place, the audit file becomes the timeline you wished you had at 09:14. You can reconstruct whether a 01:11 spike was a free retry or a paid leak without opening a finance tool, which is the whole point of stamping paid on every line. A one-liner is enough to separate the two during a review, and a non-zero number during a staging loop is a regression rather than a mystery.

python - <<'PY'
import json
paid = 0
with open("agent-audit.jsonl", encoding="utf-8") as handle:
    for raw in handle:
        row = json.loads(raw)
        paid += int(bool(row.get("paid")))
print(f"{paid} paid calls in audit log")
PY

If that count is not zero while purpose=retry is on the queue, you do not need a longer narrative in the incident doc. You need to find which wrapper constructed a client behind the gate, then add a test that names that wrapper. The contributing factors from this night collapse into a single check: did the retry purpose ever resolve a paid route, and if the tests above are green, the answer stays no.

Who should not copy this, and what it will not fix

This approach will frustrate you if every step of the agent truly needs the same high-capability model, because the gate will keep shoving noisy work toward the free endpoint on purpose. It is also the wrong design if you do not control the runtime, for example a hosted agent that injects its own client and ignores process environment. In that case the durable fix lives in the vendor allowlist, not in a Python file you do not even get to import.

Do not use a shared free server for prompts that contain secrets, customer data, or production source you would not paste into a scratch org. A free sandbox is still a sandbox, and metadata logs that store prompt lengths can grow into prompt bodies if a well-meaning teammate improves _audit at the worst possible time. Keep customer traffic on the paid, controlled path, and keep the dummy endpoint disposable enough that wiping it is a boring decision.

The reconstructed incident also assumes you can change how the agent constructs its client, which is not true of every framework. If the SDK is hard-coded inside a closed binary, wrapping urllib will not save you, and you will need an HTTP proxy at the network edge with the same fail-closed rule. That is a different change-management conversation, and you should not pretend a twelve-line gate replaced a platform control you do not have.

What this postmortem should change is the default you carry into the next agent job. You should assume retries will happen, that environment files will leak into workers, and that a just this once paid call will be copied into cron by someone who is trying to be helpful. The gate is how you make those assumptions cheap, and the unpaid model route is how you keep the agent useful while the gate is doing the unglamorous work.

If you need a throwaway box for that unpaid half, pointing FREE_MODEL_ENDPOINT at MonkeyCode’s free server option and rerunning the tests above is a reasonable rehearsal.