Don't Put a Retry Loop on Free Capacity

The first failed call is not your cost problem. The retry is. Free capacity looks like a gift until a tool error, a timeout, or a flaky compile turns one prompt into a chain, and the chain writes a bill you never put in the ticket.

You already treat tokens as a line item. Treat retries as a multiplier on that line item. Every extra attempt resends the conversation, waits again, and can fail for a different reason. That is not a second chance. That is a second invoice stacked on the first.

A coding agent is not a function call. It is a loop with memory. You ask it to patch a test. It calls the compiler. The compiler returns a wall of errors. The model reads the wall, calls the compiler again, then again. You see “it is still working.” The meter sees three full transcripts and two queue delays. If that loop sits on free, interruptible capacity, you are betting a committed job on a spot lane.

The analogy is a taxi that charges for the ride and for every U-turn. The destination did not change. The path did. You still pay for the extra miles.

Finish is a boolean. Cost is not. For any assistant job that can retry, you need three numbers on every attempt: tokens in, tokens out, and seconds spent waiting. Then you need a fourth number after the job ends: the retry multiplier. That is total tokens divided by the tokens of the successful path only. If the successful path is 4,000 tokens and the job burned 18,000, the multiplier is 4.5. You did not buy one answer. You bought four and a half.

Queue time belongs in the same ledger, but not as a separate sermon about waiting. Here the wait is fuel for the next retry. Free lanes stall. A stall is not a refund. While you wait, the ticket is still open, the human is still context-switching, and a later retry may need a warmer, larger prompt because the first error aged out of working memory. The longer the gap, the more the model re-reads, restates, and re-sends.

Retries also poison the transcript. A failed tool result stays in context unless you strip it. The next call is larger. The call after that is larger still. You are not looping on a constant prompt. You are looping on a snowball. That snowball is why “one more try” on a gift lane is an ops decision, not a mood.

Do not guess the multiplier. Log it. The script below is a method, not a production study. It does not call a vendor and it does not claim a fleet result. It records attempts you already made, or rows you simulate, and it prints the multiplier plus a verdict. Wire it to your runner. If you cannot fill the fields, you are not ready to put the job on a shared free lane.

#!/usr/bin/env python3
"""Retry ledger: measure the multiplier, not the last status.

This is an accounting method. Plug real attempt rows from your
agent runner. Do not treat the sample rows as a benchmark.
"""
from __future__ import annotations

import json
from dataclasses import dataclass, asdict
from typing import List, Optional


@dataclass
class Attempt:
    job_id: str
    attempt: int
    tokens_in: int
    tokens_out: int
    queue_s: float
    outcome: str  # ok | tool_error | timeout | cancelled
    keep_error_in_context: bool


def tokens(a: Attempt) -> int:
    return a.tokens_in + a.tokens_out


def multiplier(attempts: List[Attempt]) -> Optional[float]:
    success = [a for a in attempts if a.outcome == "ok"]
    if not success:
        total = sum(tokens(a) for a in attempts)
        return None if total == 0 else float("inf")
    useful = tokens(success[-1])  # last ok path, not the debris before it
    total = sum(tokens(a) for a in attempts)
    return total / useful if useful else float("inf")


def queue_tax(attempts: List[Attempt]) -> float:
    return sum(a.queue_s for a in attempts)


def context_creep(attempts: List[Attempt]) -> int:
    if not attempts:
        return 0
    return tokens(attempts[-1]) - tokens(attempts[0])


def verdict(mult: Optional[float], queued: float, creep: int) -> str:
    if mult is None:
        return "no_success_stop_the_loop"
    if mult == float("inf"):
        return "no_useful_tokens_do_not_retry_on_free"
    if mult >= 3.0 or queued >= 30.0 or creep >= 2000:
        return "free_capacity_is_the_wrong_bet"
    if mult >= 1.8:
        return "cap_retries_and_strip_errors"
    return "free_lane_ok_for_rehearsal_only"


def report(attempts: List[Attempt]) -> dict:
    mult = multiplier(attempts)
    queued = queue_tax(attempts)
    creep = context_creep(attempts)
    return {
        "job_id": attempts[0].job_id if attempts else None,
        "attempts": len(attempts),
        "total_tokens": sum(tokens(a) for a in attempts),
        "retry_multiplier": mult,
        "queue_seconds": queued,
        "context_creep_tokens": creep,
        "verdict": verdict(mult, queued, creep),
        "rows": [asdict(a) for a in attempts],
    }


# Example rows. Replace with your runner's log. Not a claim about any model.
SAMPLE = [
    Attempt("job-77", 1, 1800, 420, 4.0, "tool_error", True),
    Attempt("job-77", 2, 2400, 510, 11.0, "timeout", True),
    Attempt("job-77", 3, 3100, 680, 9.0, "ok", True),
]


if __name__ == "__main__":
    print(json.dumps(report(SAMPLE), indent=2))

Run it as a command, not as a vibe check.

python3 retry_ledger.py
python3 retry_ledger.py | jq '{multiplier: .retry_multiplier, verdict, queue: .queue_seconds, creep: .context_creep_tokens}'

On the sample you should see a multiplier above 2, a queue tax around 24 seconds, and a verdict that free capacity is the wrong bet. Those numbers are a shape. Swap them for your log. If your runner cannot emit tokens_in, tokens_out, and queue_s per attempt, stop. You cannot manage a meter you do not read.

Walk the sample like an incident, not like a demo. Attempt 1 failed with a tool error and still kept the error in context. Attempt 2 therefore started heavier, then timed out after a longer wait. Attempt 3 “succeeded” on a still larger prompt. The last status is green. The ledger is not. That is the whole lesson. Status is what you tell standup. The multiplier is what you owed the meter.

If your agent writes JSONL, fold the same functions over the file. Do not wait for a monthly invoice to discover the loop.

python3 - <<'PY'
import json, retry_ledger as rl
rows = []
with open("attempts.jsonl") as f:
    for line in f:
        d = json.loads(line)
        rows.append(rl.Attempt(**d))
print(json.dumps(rl.report(rows), indent=2))
PY

Before attempt n+1, print the running multiplier. If it crosses 3, stop. Do not “just try once more” on a gift lane. That extra try is how a rehearsal becomes unbudgeted production spend, even when the sticker price is zero, because zero is not the only cost. Human time, lock duration, and a polluted transcript all move.

A tiny guard in the runner looks like this. Keep it boring. Boring is what cost control is supposed to feel like.

MAX_MULT = 3.0
MAX_ATTEMPTS = 2

def should_retry(attempts):
    if len(attempts) >= MAX_ATTEMPTS:
        return False
    m = multiplier(attempts)
    if m is None or m >= MAX_MULT:
        return False
    last = attempts[-1]
    if last.outcome in {"timeout", "cancelled"}:
        return False  # timeouts retry poorly on contended free lanes
    if last.keep_error_in_context and last.tokens_in > 2500:
        return False  # snowball already started
    return last.outcome == "tool_error"

Notice the timeout rule. A tool error sometimes pays for a retry if you strip the huge stderr and keep a one-line cause. A timeout on free capacity usually means the lane is busy or preempted. Retrying a timeout on the same lane is how you donate more queue time to a job that already told you it cannot finish.

Strip before you loop. That is the cheapest retry you will ever buy. Keep the command, the exit code, and one diagnostic line. Drop the rest of the compiler novel. If you cannot strip, you should not retry, because the next prompt is a worse version of the last one. You would not paste three failed CI logs into a new ticket and call that “context.” Do not let the agent do it either.

def compact_tool_error(stderr: str, exit_code: int, cmd: str) -> str:
    lines = [ln.strip() for ln in stderr.splitlines() if ln.strip()]
    last = lines[-1] if lines else "(no stderr)"
    return f"{cmd} exited {exit_code}: {last[:240]}"

Use free capacity when the job is a rehearsal: a prompt you are still shaping, a script you are still wiring, a one-off question you can drop. Kill it when the multiplier climbs. Do not promote that lane to “the way we ship the patch.”

Refuse the free lane when the work has a human waiting on it. Refuse it when the loop can mutate a repo or an environment. Refuse it when the prompt is already large, when the last failure was a timeout or a cancel, or when you cannot log tokens per attempt. In those cases the cheap lane is the expensive decision. The retry multiplier turns a gift into a stall, and the stall still burns context.

Think of free capacity as a bus that stops when it is full. Committed work is a seat you bought. You do not put a deploy-blocking test fix on the bus and then act surprised when it gets bounced and you buy three more tickets trying to arrive on time. Spot is for experiments you can abandon. A retry loop is a promise that you will not abandon. Those two facts do not share a lane.

If you need a box to practice the ledger itself, MonkeyCode’s free model access and free server option are enough to run the accounting script and a short, capped rehearsal. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That is the right size of bet: learn the multiplier on a scratch box. It is the wrong size of bet for an unattended retry loop against a live branch.

This ledger does not know a vendor’s billing granularity. Some meters round. Some count cached prefixes differently. Some charge for tool payloads you never see in the chat UI. If you cannot map a row in the log to a row on the invoice, treat the multiplier as a lower bound. The sample rows are not a benchmark. Do not cite 4.5x, 24 seconds, or a threshold of 3.0 as measured results from a fleet. Set the cuts from your own traces. A research spike and a CI fixer are not the same job.

The verdict function is conservative on purpose. It will tell you to leave free capacity earlier than a hopeful operator would. That is the point. Hope is not a cost control. A green last attempt can still be a bad buy if the path to green was three transcripts and a queue.

Who should not use this approach: anyone who cannot instrument attempts; anyone running unattended agents against production credentials; anyone who needs a hard SLA and thinks a free lane will provide one; anyone shopping for a model beauty contest. This is ops arithmetic. It will not pick a model for you. It will tell you when to stop looping.

If the job matters, cap the retries, strip the errors, and put the loop on capacity you actually reserved. Free is for learning the shape of the bill. It is not for surviving the loop.