Postmortem: The Agent Loop That Kept Retrying

It is 09:14 on a Tuesday, and your deploy agent has just written the same billing row for the third time. Nothing crashed, no page fired, and the test suite is green, because the agent did exactly what you asked it to do. The damage is invisible until someone reconciles invoices two weeks later.

What follows is a composite reconstruction rather than a named company's incident report. The timestamps and payloads are synthesized, but the failure shape is one worth rehearsing before it finds you. Treat it as a lab reproducer you can rebuild in an afternoon.

Timeline

At 09:02 the pipeline hands a failing integration test to an agent with one instruction: make this pass, and use the tools available. The agent reads the test, concludes that a missing subscription record is the cause, and calls a create endpoint through its tool layer. The endpoint answers with a 429 because a shared rate limiter is already saturated by the morning batch.

At 09:04 the agent retries, because retrying is what the prompt tells it to do. The retry succeeds at 09:05, and the agent moves on, satisfied that its reasoning was sound. At 09:07 the test still fails for an unrelated reason, so the agent calls the create endpoint again with the same arguments, and this time it gets a 201 from a service that has never heard of idempotency keys.

By 09:11 the loop has run nineteen times and the transcript has grown from 4k to 61k tokens, because every retry appends the previous retry to the context. By 09:14 three duplicate rows exist, the run is still going, and the only guardrail in place is a one-hour wall-clock timeout that nobody expected to matter. The bill for that hour arrives later, and it costs more than the feature being fixed.

Contributing factors

The first factor is that the retry policy lived in the prompt instead of in code. Prose instructions are advisory, they drift as context grows, and they cannot be unit tested when you change them. A model that has been told to be persistent will be persistent about the wrong things.

The second factor is the missing idempotency key on a write that was never safe to repeat. The tool layer passed the model's arguments through untouched, so a retried call became a new record rather than a no-op. Retries are inevitable; duplicate writes are optional.

The third factor is the absent token budget for a single run. You had a timeout, which caps time, but time and money diverge badly once each iteration resends the whole transcript. Nineteen iterations of a growing context cost far more than nineteen flat calls would.

The fourth factor is the observability gap that let all of this stay quiet. Tool results were logged as human-readable strings rather than structured status codes, so no alert could distinguish a 429 from a 201 at a glance. You cannot write a rate-of-duplicate-writes alert against prose.

The durable fix

The fix is not a better prompt, because prompts are the layer that failed. Move retries down into a wrapper you own, give every side-effecting call a deterministic idempotency key, and give every run a hard token ceiling that raises instead of warns. The reference implementation below is written for a test harness with a fake transport, and it is the smallest version of the pattern that still fails loudly.

import hashlib, json

class BudgetExceeded(RuntimeError):
    pass


def idem_key(run_id: str, step: str, args: dict) -> str:
    payload = json.dumps(args, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(payload.encode()).hexdigest()[:24]
    return f"{run_id}:{step}:{digest}"


class BudgetGovernor:
    RETRYABLE = {429, 503}

    def __init__(self, max_tokens: int, max_attempts: int = 4):
        self.max_tokens = max_tokens
        self.max_attempts = max_attempts
        self.spent = 0

    def charge(self, prompt_tokens: int, completion_tokens: int) -> None:
        self.spent += prompt_tokens + completion_tokens
        if self.spent > self.max_tokens:
            raise BudgetExceeded(f"spent={self.spent} cap={self.max_tokens}")

    def call(self, transport, *, run_id: str, step: str, args: dict):
        key = idem_key(run_id, step, args)
        attempt = 0
        while True:
            attempt += 1
            resp = transport.send(args, idempotency_key=key)
            if resp.status in self.RETRYABLE and attempt < self.max_attempts:
                continue
            if resp.status in self.RETRYABLE:
                raise RuntimeError(f"retries exhausted for {key}")
            return resp

The key detail is that the wrapper decides when to retry and the model never does. Because the key is derived from the run, the step, and the canonical arguments, a repeated call with identical arguments becomes a single logical write that the downstream service can deduplicate. If your downstream service ignores the header, the wrapper still gives you the key to reconcile against later.

A test plan you can actually run

Write the test before the incident, with a fake transport that records every call, and a governor whose cap you set absurdly low on purpose. Assert three things: that a 429 followed by a 201 produces exactly one recorded write in the dedupe store, that exceeding the cap raises instead of degrading quietly, and that a transcript large enough to blow the cap never reaches the model at all.

def test_budget_stops_before_the_second_call():
    gov = BudgetGovernor(max_tokens=10_000)
    gov.charge(6_000, 500)
    try:
        gov.charge(6_000, 500)
        assert False, "governor should have raised"
    except BudgetExceeded:
        pass

Run this suite on every change to your tool layer, because tool schemas drift silently and a renamed argument changes the idempotency key. If you want to replay long production traces while you tune the cap, this is the step where a cheap staging environment earns its place.

Where to run the replay harness

Situation	Sensible environment	Reasoning
One-off reproduction of a retry storm	Local container with a fake transport	Fastest loop, no network variance
Replaying thousands of log lines through a summarizer	Free server tier, with free model access for triage	Keeps noisy replay off paid quota and off your production limits
Sustained load testing at production rates	Paid VM with reserved capacity	Free tiers carry fair-use limits and no throughput promise

For the middle row, MonkeyCode's free model access and free server option are a reasonable place to park a replay harness that summarizes traces and classifies tool status codes before a human reads them. The operator describes the current free allowance as up to 10M tokens plus a free server, so check the live console yourself before you design around that number, because free-tier terms move. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The triage step is a genuinely good fit for cheap or free models, since classifying a status code and pulling the failing step out of a transcript is a narrow task with an easy correctness check. Keep the actual fix, the patch that removes the duplicate write, on whatever model your team already trusts for code review.

Limitations and who should skip this

Idempotency keys only help when the downstream service honors them or gives you a reconciliation endpoint, and sends are the classic counterexample, because a duplicate email cannot be unsent. If your writes are inherently non-idempotent, you need a dedupe store keyed on the idempotency value instead of a header convention.

Streaming responses can also report token usage late or in fragments, so your governor should treat missing usage as an error rather than as zero. The ceiling here is per run, not per tenant, which means a hundred cheap runs can still add up to an expensive morning. And free-tier availability is not a durability guarantee, so never put anything load-bearing on it.

Skip the governor entirely if you run a handful of agent jobs per day and every write already sits behind a transactional check. The overhead is real, and the pattern only pays for itself once retries are frequent enough that a single bad afternoon can outspend a quarter of your inference budget.

If you have already felt that afternoon, the fastest next step is to write the failing test with the fake transport first. Park the replay harness on a free server so the reproducer never touches paid quota, and let the fix land against a budget that raises instead of a prompt that pleads.