Budget the Retry Path, Not the Happy Path

Free capacity lies to you the moment you budget only the successful call. An agent that retries, backs off, and re-enters a queue is not getting a mulligan. It is buying a second invoice and a second ticket at the deli counter, and the first sandwich still never showed up.

You already know the happy-path math. One prompt, some completion tokens, a latency you can live with. That spreadsheet is comforting. It is also incomplete. The path that actually burns your afternoon is the one where the model times out, the tool call 500s, or the free endpoint parks you behind a crowd you cannot see.

A retry is capacity. Treat it like capacity or it will treat your calendar like a suggestion.

The bill you did not itemize

Picture a checkout line that resets you to the back every time the cashier drops a bag. You did not steal extra groceries. You still spent extra time. If the store also charges per scan attempt, you spent extra money too. LLM endpoints behave like that store when retries are unbounded and queues are shared.

Tokens are only one meter. Wall-clock wait is another. Occupancy in someone else's queue is a third. You can ignore two of those meters on a weekend toy. You cannot ignore them on a job that has a merge window, a customer demo, or a batch that must finish before morning traffic.

Failed calls are not free even when the sticker price is zero. They consume rate-limit budget, connection slots, log volume, and attention. If your agent loops, each loop multiplies the miss. That is the retry tax: expected tokens and expected minutes conditional on the failure path, not the brochure path.

A small estimator you can actually run

Do not argue about this in Slack. Price it. The script below is a worked example, not a production benchmark and not a claim about any vendor's hardware. Plug in numbers you measured. If you have not measured them, the output is a hypothesis, not a budget.

#!/usr/bin/env python3
"""retry_tax.py — expected tokens and wait under a retry policy.

Label: unexecuted template. Replace LATENCY_S, TOKENS, and P_FAIL
with values from your own traces before you trust the printout.
"""
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class CallShape:
    tokens_per_attempt: float
    latency_s: float
    p_fail: float  # independent per attempt; refine if your failures cluster


@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int
    backoff_s: float  # constant delay between attempts; swap for exp if needed
    queue_wait_s: float  # extra wait before each attempt reaches a worker


def expected_cost(shape: CallShape, policy: RetryPolicy) -> dict[str, float]:
    if not 0.0 <= shape.p_fail <= 1.0:
        raise ValueError("p_fail must be in [0, 1]")
    if policy.max_attempts < 1:
        raise ValueError("max_attempts must be >= 1")

    p_ok = 1.0 - shape.p_fail
    tokens = 0.0
    seconds = 0.0
    p_still_going = 1.0
    p_success = 0.0

    for attempt in range(policy.max_attempts):
        tokens += p_still_going * shape.tokens_per_attempt
        seconds += p_still_going * (policy.queue_wait_s + shape.latency_s)
        if attempt < policy.max_attempts - 1:
            seconds += p_still_going * shape.p_fail * policy.backoff_s
        p_success += p_still_going * p_ok
        p_still_going *= shape.p_fail

    return {
        "expected_tokens": tokens,
        "expected_seconds": seconds,
        "p_eventual_success": p_success,
        "p_hard_fail": p_still_going,
        "attempts_budgeted": float(policy.max_attempts),
    }


def hourly_drag(expected_seconds: float, loaded_hourly_rate: float) -> float:
    """Turn wait into money using YOUR loaded rate, not a vendor quote."""
    return (expected_seconds / 3600.0) * loaded_hourly_rate


if __name__ == "__main__":
    shape = CallShape(tokens_per_attempt=2_400, latency_s=8.0, p_fail=0.22)
    generous = RetryPolicy(max_attempts=4, backoff_s=3.0, queue_wait_s=12.0)
    tight = RetryPolicy(max_attempts=2, backoff_s=1.0, queue_wait_s=12.0)

    g = expected_cost(shape, generous)
    t = expected_cost(shape, tight)
    print("generous", g, "time_cost", hourly_drag(g["expected_seconds"], 90))
    print("tight   ", t, "time_cost", hourly_drag(t["expected_seconds"], 90))

Run it once with the placeholders, then run it again with traces. The first run teaches the shape of the function. The second run is the only one that should influence a buy-versus-wait decision.

python3 retry_tax.py
# generous {'expected_tokens': 5491.2, ...}
# tight    {'expected_tokens': 4272.0, ...}

Those printed tokens are not a promise. They are what a 22% independent failure rate does to a 2,400-token attempt if you allow four bites at the apple. Change p_fail to what your logs say. If your failures cluster during a regional blip, independence is optimistic and the tax is worse.

Measure the meters you actually have

You need three numbers that do not come from a landing page: tokens per attempt, including the failed ones; time from enqueue to response, including queue wait; and the fraction of attempts that do not produce a usable result. Everything else is decoration.

A probe that records those meters can be boring. Boring is the point. Point it at a scratch endpoint, not at the path that pages you.

#!/usr/bin/env python3
"""probe_attempt.py — one instrumented call. Template, not a load test."""
import json, os, time, urllib.request

URL = os.environ["LLM_URL"]  # your endpoint; do not hardcode secrets
BODY = json.dumps({
    "messages": [{"role": "user", "content": "Reply with the single word: pong"}],
    "max_tokens": 8,
}).encode()

req = urllib.request.Request(
    URL,
    data=BODY,
    headers={"Content-Type": "application/json"},
    method="POST",
)
started = time.monotonic()
try:
    with urllib.request.urlopen(req, timeout=30) as resp:
        raw = resp.read()
        status = resp.status
except Exception as exc:
    elapsed = time.monotonic() - started
    print(json.dumps({"ok": False, "seconds": elapsed, "error": type(exc).__name__}))
    raise SystemExit(1)

elapsed = time.monotonic() - started
payload = json.loads(raw.decode() or "{}")
usage = payload.get("usage") or {}
print(json.dumps({
    "ok": 200 <= status < 300,
    "seconds": elapsed,
    "prompt_tokens": usage.get("prompt_tokens"),
    "completion_tokens": usage.get("completion_tokens"),
}))

Wrap that probe in a loop of twenty calls at a human pace. Twenty is enough to see whether p_fail is a rounding error or a lifestyle. It is not enough to certify a platform. If you need statistical comfort, you need a longer window and a change-control note, not a hotter take.

When free capacity is the wrong bet

Free model access is a rehearsal room. A free server is a place to learn whether your retry policy is a plan or a superstition. Neither one is a high-availability contract. If the job cannot slip, spare capacity in a shared queue is a hope, not a control.

You should refuse the free path when any of these are true. The merge window is shorter than your expected wait under retries. A failed attempt still leaves a side effect you cannot undo, like a half-applied ticket comment or a duplicate charge intent. Your agent will retry on timeouts without idempotency keys. You cannot observe tokens on the failure path. Someone else's backlog can jump in front of you and you have no lever except waiting.

That last one is the queueing punchline. Throughput you do not control is not spare capacity. It is leftover capacity, and leftover capacity disappears the moment other people have the same idea.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option you can use as that rehearsal room: run the estimator, run the probe, and see whether your retry tax is a footnote or the whole plot. Keep the disclosure in mind. A scratch lane is for measurement. It is not a reason to skip a kill switch.

If you want a single operational rule, use this: no retry policy ships without a capped attempt count, a backoff that cannot collapse into a stampede, and a hard stop that does not ask the model for permission. The estimator tells you what that cap costs. The probe tells you whether the cap is live. The calendar tells you whether free is even in the conversation.

Limitations, and who should walk away

The math assumes per-attempt independence. Real outages correlate. Correlated failure makes expected tokens look polite while wall-clock wait goes vertical, so treat the script as a lower bound when the incident channel is already noisy.

It also prices one call shape. Agents that fan out tool calls, then retry the fan-out, do not have one tax. They have a tree. If you do not flatten that tree in logs, you will under-count by a factor you will only notice after the invoice, or after the standup.

Do not use this approach as a substitute for a paid, SLA-backed endpoint when the work is customer-facing, regulated, or timed to a launch. Do not use it to justify deleting observability because the tokens were free. Do not use a four-attempt policy as a personality trait. And do not load-test a shared free pool until you have confirmed that hammering it is allowed. Courtesy is part of cost discipline.

You should also walk away if you cannot name the loaded hourly rate of the human waiting on the queue. Token stickers without time are how teams convince themselves that a two-hour retry spiral was efficient.

Budget the retry path first. The happy path will still be there in the morning. The queue will not hold your place.