Postmortem: The Retry Storm That Burned the Token Budget

Every green test can hide a burning budget. This postmortem covers one week. An AI patch-review pipeline spent 4.2x its expected token allowance. The root cause was not a weak model. It was silent retries and growing context. The durable fix was cost observability, not cheaper prompts.

Background

The team ran an AI review gate for AI-generated patches. The pipeline had two stages. A planner summarized each diff into a plan. A reviewer used that plan to emit findings.

Both stages called a hosted LLM API. Quality checks passed. Human reviewers approved the output. Nobody watched token spend.

Why the eval stayed green

The quality eval compared outputs on a golden set. Outputs stayed correct. Retries and extra context did not change semantics. Cost never appeared in the eval matrix. Green measured quality only, not economics. The pipeline needed a second matrix.

Timeline

Monday, 09:12 — A teammate raised the retry policy from 2 to 5 attempts. The upstream provider had been flaky.

Monday, 11:40 — The planner prompt gained a new section. It said: "Include the last three review decisions." Input context doubled.

Tuesday, 16:05 — The monthly budget alert fired. Finance asked for a breakdown.

Wednesday, 10:00 — The cost dashboard showed the 4.2x spike. No deploy had changed prompts that week. Nobody noticed until now.

Wednesday, 14:30 — The team traced the spike to two changes. The retry policy amplified every timeout. The new prompt section doubled every planner call.

Thursday, 11:00 — The durable fix shipped. It had three parts: a token ledger, run-level budgets, and routing rules.

Contributing factors

1. Retry storms. One timeout triggered five attempts. Each attempt re-sent the full context. Timeouts became five times more expensive.
2. Context bloat. The new section doubled input tokens on every planner call. Review decisions were verbose.
3. No cost assertion in CI. Tests asserted quality. They never asserted token usage.
4. Wrong eval unit. Green meant "correct output." It never meant "efficient run."

Root cause

Cost had no feedback loop into engineering. Behavior changed, spend changed, and only finance noticed. The pipeline treated tokens as free. An alert proved otherwise.

The artifact: token ledger

Every run needs a ledger entry. Every run needs a budget. This script wraps any OpenAI-compatible endpoint:

#!/usr/bin/env python3
"""Token ledger for LLM runs. Fails the run when the budget is exceeded."""
import argparse
import csv
import json
import os
import sys
import time
import urllib.request
from datetime import datetime, timezone

LEDGER = os.environ.get("LLM_LEDGER", "ledger.csv")
BUDGET = int(os.environ.get("RUN_BUDGET_TOKENS", "40000"))
MAX_RETRIES = int(os.environ.get("MAX_RETRIES", "3"))


def call_llm(messages):
    body = json.dumps({"model": os.environ["LLM_MODEL"], "messages": messages}).encode()
    req = urllib.request.Request(
        os.environ["LLM_ENDPOINT"],
        data=body,
        headers={
            "Authorization": f"Bearer {os.environ['LLM_API_KEY']}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)


def run_with_retries(messages):
    last_error = None
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            return call_llm(messages), attempt
        except Exception as exc:  # noqa: BLE001
            last_error = exc
            time.sleep(2**attempt)
    raise last_error


def append_ledger(run_id, stage, tokens, retries):
    row = {
        "ts": datetime.now(timezone.utc).isoformat(),
        "run_id": run_id,
        "stage": stage,
        "tokens": tokens,
        "retries": retries,
    }
    fresh = not os.path.exists(LEDGER)
    with open(LEDGER, "a", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=list(row))
        if fresh:
            writer.writeheader()
        writer.writerow(row)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--run-id", required=True)
    parser.add_argument("--stage", required=True)
    parser.add_argument("--input", type=argparse.FileType("r"), default=sys.stdin)
    args = parser.parse_args()

    messages = json.load(args.input)
    response, attempts = run_with_retries(messages)
    usage = response.get("usage", {})
    total = int(usage.get("total_tokens", 0))
    append_ledger(args.run_id, args.stage, total, attempts - 1)

    if total > BUDGET:
        sys.exit(f"budget exceeded: {total} > {BUDGET} tokens")
    print(json.dumps(response))


if __name__ == "__main__":
    main()

Usage:

export LLM_ENDPOINT=https://api.example.com/v1/chat/completions
export LLM_MODEL=review-model
export LLM_API_KEY=$KEY
cat messages.json | python3 token_ledger.py --run-id pr-147 --stage planner

The script records tokens and retries per run. A ledger row looks like this:

2026-09-01T09:12:33+00:00,pr-147,planner,21403,4

The exit code breaks CI. A spike becomes a failed build. A failed build becomes a conversation. A conversation beats a surprise bill.

Where each run goes

The team then split routing by purpose. New prompts and experimental tool calls ran first against free model access in MonkeyCode, an open-source AI coding assistant. Its free plan included a 10M token allowance and a free server option for sandboxed runs at the time of writing. Production patch reviews stayed on the paid endpoint. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Run type	Route
First 50 runs of a new prompt	Free model access
Retry and timeout experiments	Free server
Scheduled production reviews	Paid endpoint
Compliance-sensitive code	Reviewed infrastructure only

The routing table is the durable policy. Free access is an evaluation sandbox, not a production line.

Durable fix

1. Ledger every run. Fail the build when a run exceeds its budget.
2. Budget retries. Keep MAX_RETRIES=2 with exponential backoff. A retry is a cost decision, not a hope.
3. Sandbox new prompts. Promote a prompt only after cost and quality metrics stay stable.
4. Alert at 80%. Monthly allowance alerts fire before finance does.
5. Re-benchmark on the target endpoint. Free-tier latency and rate limits differ from paid endpoints. Measurements do not transfer.

Limitations

• The 10M token allowance and free server are availability claims from the project at the time of writing. Terms can change.
• Free endpoints may have lower rate limits. Do not assume production throughput.
• The ledger only works if the provider returns usage in the response. Some providers omit it.
• Retry budgets trade availability for cost. A hard limit can turn a timeout into a failed run.

Who should not use this

Teams with strict data residency rules should not route customer code to a free shared server without a data-processing review. Teams that need guaranteed latency should keep experiments on the paid tier. The ledger is observability, not a legal review.

Verdict

The incident was boring and expensive. Two small changes, zero deploys, 4.2x spend. The fix was not a cheaper model. It was a ledger, a budget, and a sandbox. The fastest way to test this workflow is to point the ledger at a free endpoint and let the first week of runs decide.