Retries Are How Cheap Jobs Get Expensive

Retries are how cheap jobs get expensive. A failed generation does not refund the tokens you already burned, and it does not restore your place in line. It opens a second job that looks identical in a per-request dashboard and twice as costly in any honest ledger.

You already know the first attempt has a price. The trap is treating the second attempt as a mulligan. In shipping, a returned parcel is not free to send again. You pay postage, you pay handling, and you wait behind everyone who shipped once. LLM work behaves the same way once you stop staring at status codes and start staring at attempts.

Per-request charts hide that. One green check after four failures still plots as a single success. The queue saw four arrivals. The tokenizer saw four growing prompts. Your bill saw four completions, including the ones you threw away. If you only alert on error rate, a retry storm looks like resilience. It is spend with a friendly name.

Think of an agent loop as a multiplier, not a safety net. The first call ships a system prompt, a file excerpt, and a tool schema. The second call ships all of that plus the failed tool trace. The third call ships the argument about the second call. Context does not reset when hope resets. Each retry is a fatter invoice re-entering the same contested lane.

That is why free capacity is often the wrong bet after the first miss, even when the work is not on a deadline. Free lanes are shared. Shared lanes punish re-entry. You do not merely wait. You wait, then you pay to reconstruct state, then you wait again. The cheap path becomes the long path, and the long path becomes the expensive path because tokens and queue time compound together.

Price the next attempt before you launch it. Not the model. The attempt. Give every job an attempt budget with three hard numbers you can defend: a maximum retry count, a cumulative token ceiling, and a re-queue time ceiling. When any ceiling trips, stop or promote. Do not ask the model if it feels lucky.

The script below is a worked example, not a production benchmark and not a claim about any vendor's quota. Run it locally. Feed it your own traces. Treat the numbers as a method, not a result.

#!/usr/bin/env python3
"""Attempt ledger for LLM jobs. Example only; plug in your tracer."""
from __future__ import annotations

from dataclasses import dataclass, field
from typing import Literal, Optional

Route = Literal["free", "paid"]

@dataclass
class Attempt:
    job_id: str
    n: int
    tokens_in: int
    tokens_out: int
    queue_ms: int
    ok: bool
    reason: str

@dataclass
class Budget:
    max_attempts: int = 3
    max_tokens: int = 24_000
    max_queue_ms: int = 20_000

@dataclass
class Ledger:
    job_id: str
    route: Route
    budget: Budget
    attempts: list[Attempt] = field(default_factory=list)

    def tokens(self) -> int:
        return sum(a.tokens_in + a.tokens_out for a in self.attempts)

    def queue_ms(self) -> int:
        return sum(a.queue_ms for a in self.attempts)

    def next_allowed(self) -> tuple[bool, str]:
        if len(self.attempts) >= self.budget.max_attempts:
            return False, "attempt_cap"
        if self.tokens() >= self.budget.max_tokens:
            return False, "token_cap"
        if self.queue_ms() >= self.budget.max_queue_ms:
            return False, "queue_cap"
        return True, "ok"

    def record(self, tokens_in: int, tokens_out: int, queue_ms: int, ok: bool, reason: str) -> Attempt:
        attempt = Attempt(
            job_id=self.job_id,
            n=len(self.attempts) + 1,
            tokens_in=tokens_in,
            tokens_out=tokens_out,
            queue_ms=queue_ms,
            ok=ok,
            reason=reason,
        )
        self.attempts.append(attempt)
        return attempt

    def promote_or_stop(self) -> str:
        allowed, why = self.next_allowed()
        if allowed:
            return f"retry_on_{self.route}"
        if self.route == "free" and why in {"queue_cap", "attempt_cap"}:
            return "promote_to_paid_once"
        return f"stop:{why}"

def simulate_retry_storm(job_id: str) -> None:
    # Synthetic trace so you can see the multiplier. Replace with real spans.
    growing_prompt = 1800
    ledger = Ledger(job_id=job_id, route="free", budget=Budget())
    for i in range(6):
        decision = ledger.promote_or_stop()
        print(f"pre-attempt decision={decision} tokens={ledger.tokens()} queue_ms={ledger.queue_ms()}")
        if not decision.startswith("retry"):
            print("circuit open")
            return
        queue = 1500 * (i + 1)  # re-entry gets slower on a busy free lane
        out = 400
        ok = False
        ledger.record(growing_prompt, out, queue, ok, reason="tool_schema_mismatch")
        growing_prompt += 350  # failed tool trace sticks to the next prompt
    print("unbounded loop would have continued")

if __name__ == "__main__":
    simulate_retry_storm("codegen-414")

Run it as a dry rehearsal before you wire a tracer.

python3 attempt_ledger.py

You should see the circuit open on the attempt cap or the queue cap, not on the sixth hopeful call. That is the point. The job does not get to negotiate after it has already multiplied. If your real traces never trip the cap, your budget is theater. Tighten it until a noisy tool failure stops itself.

Wire the same object to whatever you already log. You do not need a new platform. You need three fields on every span: attempt_n, tokens_in+tokens_out, and queue_ms. If a vendor hides queue time, record scheduled_at and first_token_at yourself. Subtract. Attach both timestamps to the job id, not to the HTTP request id. Request ids change on retry. Job ids must not.

A small assertion keeps the ledger honest in CI. The test is a contract on accounting, not on model quality.

import unittest
from attempt_ledger import Budget, Ledger

class LedgerContract(unittest.TestCase):
    def test_retry_is_a_new_invoice(self):
        led = Ledger("j1", "free", Budget(max_attempts=2, max_tokens=10_000, max_queue_ms=9_000))
        led.record(2000, 500, 4000, False, "parse_error")
        led.record(2350, 500, 4000, False, "parse_error")
        self.assertEqual(led.tokens(), 5350)
        self.assertEqual(led.queue_ms(), 8000)
        self.assertEqual(led.promote_or_stop(), "stop:attempt_cap")

    def test_free_lane_promotes_on_queue_cap(self):
        led = Ledger("j2", "free", Budget(max_attempts=5, max_tokens=50_000, max_queue_ms=3000))
        led.record(800, 200, 3500, False, "timeout")
        self.assertEqual(led.promote_or_stop(), "promote_to_paid_once")

if __name__ == "__main__":
    unittest.main()

python3 -m unittest attempt_ledger_test.py -v

Notice what the contract refuses to do. It does not ask whether the second answer would have been better. It does not average tokens across successes. It does not bless a free lane because the sticker price is zero. Zero is the price of admission, not the price of re-entry. Re-entry is still work.

Use a routing rule you can explain to someone who does not write agents. First attempt may sit on a free lane if the job is batch, interruptible, and has a tiny prompt. After one failure, look at why. Schema errors and missing files will not heal because you waited. Retrying those on the same lane is how you buy a longer queue for the same bug. Promote once, or stop and fix the tool. Model flakes and truncated outputs can retry, but only inside the budget, and only if the prompt did not grow more than a ratio you set. A doubling transcript is not a retry. It is a different job wearing the same ticket.

That last distinction matters more than model choice. Teams argue about which checkpoint is smarter while the loop quietly restates three files of dead context. You can watch it with a one-liner on a JSONL trace if you already emit token counts.

python3 - <<'PY'
import json, sys
from collections import defaultdict
jobs = defaultdict(list)
for line in sys.stdin:
    row = json.loads(line)
    jobs[row["job_id"]].append(row["tokens_in"] + row["tokens_out"])
for job, series in jobs.items():
    if len(series) < 2:
        continue
    print(job, "attempts", len(series), "first", series[0], "last", series[-1], "sum", sum(series))
PY

Pipe real logs, not hopes. If last is much larger than first, your retry is eating its own history. Cap context on retry the way you cap attempts. Summarize the failure in one paragraph. Do not reship the whole argument. The analogy is packing: you do not tape the damaged box inside a bigger box and call that a fix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a scratch place to run the ledger against a live coding loop, MonkeyCode's free model access and free server option are enough to instrument attempts without standing up your own box. Treat both as capacity you still have to budget. They do not cancel a retry storm. They only make the storm cheaper to observe.

Limitations sit in the open. This method does not estimate quality. A stopped job can be the right cost decision and the wrong product decision. Token counters differ by vendor and by whether they bill cached prefixes; your ledger is only as true as the meter you trust. Queue time on a laptop is not queue time on a shared free lane, so do not publish the sample numbers above as evidence. They exist to show multiplication, not to rank providers. Promotion to a paid lane can thrash if you promote on the first timeout and the paid lane is also busy. Promote once. Then stop.

You should not use this approach if you cannot attach a stable job id across retries. Without that, you will keep paying for the same failure under new request ids and congratulate yourself on throughput. You should not use it for interactive chat where a human is the circuit breaker. You should not use it as an excuse to skip tests. A parse error retried six times is still a parse error. The budget is there to keep the mistake small, not to launder it into a success metric.

Cheap jobs stay cheap when the second attempt has a price tag before it starts. Everything else is a story you tell after the invoice arrives.