Retries Are Rent

Your retry loop does not bill you in tokens. It bills you in seconds you cannot buy back, and free capacity is the one lane where the second invoice is the only one that arrives. That is the whole argument. Everything below is a way to put a number on it before you hand a week of batch work to a queue you do not control.

Tokens are the visible line on the receipt. Wall clock is the line the vendor never prints. When you move a workload onto a free lane, the first line shrinks and the second one grows, and most teams never notice the trade because only one of the two numbers shows up in a dashboard.

The ledger you are not keeping

Every agent framework I read now ships a retry policy, and almost none of them log what a retry costs in time. So write the ledger yourself. The harness below is a proposal, not something I ran against a hosted endpoint, so treat the output as a placeholder until you point it at your own client.

# retry_ledger.py — proposal; run it against your own client before trusting it
from dataclasses import dataclass, field
from time import monotonic

RETRYABLE = {"rate_limited", "timeout", "upstream_5xx"}

@dataclass
class Attempt:
    queue_wait_s: float
    work_s: float
    tokens_in: int
    tokens_out: int
    outcome: str

@dataclass
class Task:
    name: str
    attempts: list = field(default_factory=list)

    @property
    def succeeded(self):
        return bool(self.attempts) and self.attempts[-1].outcome == "ok"

    @property
    def wall_s(self):
        return sum(a.queue_wait_s + a.work_s for a in self.attempts)

    @property
    def queue_share(self):
        return sum(a.queue_wait_s for a in self.attempts) / max(self.wall_s, 1e-9)

    @property
    def tokens(self):
        return sum(a.tokens_in + a.tokens_out for a in self.attempts)


def run_task(name, call, max_attempts=4, backoff=1.5):
    task = Task(name)
    delay = 1.0
    for _ in range(max_attempts):
        t0 = monotonic()
        result = call(name)          # your client returns outcome + timing + usage
        t1 = monotonic()
        task.attempts.append(Attempt(
            queue_wait_s=getattr(result, "queue_wait_s", 0.0),
            work_s=t1 - t0 - getattr(result, "queue_wait_s", 0.0),
            tokens_in=getattr(result, "tokens_in", 0),
            tokens_out=getattr(result, "tokens_out", 0),
            outcome=result.outcome,  # "ok" | "rate_limited" | "timeout" | "malformed" | "bad_input"
        ))
        if result.outcome == "ok":
            return task
        if result.outcome not in RETRYABLE:
            return task              # do not buy rent on a bug you can fix
        delay *= backoff
    return task

Two lines in that dataclass matter more than the rest. queue_share tells you how much of your latency was someone else's scheduling, and tokens counts every attempt, not just the successful one. Log both and the free-lane decision stops being a vibe.

Failure classes decide who pays

Retrying a bad_input error is the most expensive habit in this whole space. You pay queue wait, you pay tokens, and the answer cannot change. Treat malformed the same way until you have a repair step that actually alters the request.

Retries are rent because the landlord is the queue, not your code. A three-attempt success at 12 seconds of queue wait each costs you 36 seconds of clock whether or not a single token was billed. On an interactive path that is a product failure. On an overnight batch it is a rounding error.

Where the free lane actually wins

Free capacity is a scheduling choice, not a discount. It wins when your deadline is loose, your tasks are idempotent, and your failures are retryable. It loses when a human is watching the spinner, or when retry amplification pushes you past a wall-clock deadline that has money attached.

Workload shape	Free lane	Committed lane
Interactive edit, p99 under 2s	no	yes
Overnight batch, morning deadline	yes, if queue_share stays under ~50%	yes if it does not
Fan-out where 30%+ attempts are rate_limited	yes, with a jittered cap	yes
Anything with bad_input in the loop	fix the input first	fix the input first
Long cross-repo refactor you must restart often	risky: restarts multiply queue wait	yes

That table is a starting point, not a benchmark. Fill in your own columns from the ledger after a few hundred tasks.

Where MonkeyCode fits in the ledger

The project currently advertises free model access and a free server option, which makes it a reasonable candidate for the slow lane in the harness above: batch jobs where you care about the total and not the tail. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not going to quote a quota number here. Allowances, model availability, and server terms are set by the project and they move; read the current terms instead of a blog post, including this one. What matters for the ledger is only that you can point call() at it, capture queue wait if the client exposes timing, and count attempt-level tokens.

One natural way to start: take a batch you already run, cap max_attempts at two, run it twice — once on the lane you pay for, once on the free lane — and compare tokens / successes and wall_s / successes side by side. Two numbers, one afternoon.

Test plan you can run this week

Write a stub call() that returns rate_limited 30% of the time and ok otherwise, with a fixed 8-second simulated queue wait. Run 200 tasks and assert the ledger reports an amplification near 1.43 and a queue_share above 0.6. Then flip the stub to 2 seconds of queue wait and rerun. The amplification barely moves; the wall clock collapses.

That experiment is the point. Free capacity changes the constant, not the shape. If your retry temperature is wrong, a faster queue just gets you to the wrong answer sooner.

Limitations and who should skip this

Do not use the free lane for anything with a human on the other end, for regulated data whose residency rules you have not confirmed, or for workloads you must restart often. Do not use it when the deadline penalty exceeds the token savings by an order of magnitude — usually the case for paid client work.

And do not use any of this if you cannot state your per-success budget. A ledger without a target is a diary. Pick the number first, then let the queue wait tell you which lane you can afford.