The Second Call Is a Different Job

Free capacity is priced on the first successful call. That is the wrong unit. A retry is not a continuation of the same job. It is a new ticket at the back of the same line, and the line does not remember that you already waited.

If you budget tokens for one prompt, you are budgeting a fairy tale. Real agent traffic fails on a tool schema, on truncated JSON, on a gateway that shrugged. Each failure puts you back in line. The second call has a different cost shape than the first, even when the prompt text looks identical.

Think of a bakery number dispenser. You took ticket 47. The display jumped to 61 while your request 500'd. Your retry is ticket 82. You did not keep your place. You bought a new place. Free lanes feel like an open counter until you notice the dispenser still clicks.

This is a cost-ops note, not a model-quality note. You can like the output and still lose the afternoon. Wall-clock and token-clock are two meters. Most pipelines read only one.

The happy path is a cost bug

You write a client with max_retries=3 and exponential backoff. That looks responsible. It is also a multiplier that never entered the spreadsheet. Backoff is delay you chose. Queue wait is delay the lane chose. They stack. They do not cancel.

An agent that "just retries" is an unpriced loop. One bad tool call can re-enter the lane four times, each time with a fatter context because you stuffed the error into the next prompt. Tokens grow. Wait grows. Your free call is now four queued jobs and a quieter bill of time.

You do not need a hard customer SLA for this to be the wrong bet. A deadline makes the miss loud. Retry tax makes it polite. You ship later, with a log full of 429s, and still tell yourself the tokens were free.

Agents assume the next step will work. Cost models assume the first call will work. Same habit. Different blast radius.

A local accounting artifact

Do not argue about vendors yet. Measure the retry path on a box you control. The script below is a method, not a benchmark. It does not prove anyone's speed. It records attempts, wait, backoff, and a crude retry multiplier so you can see when spare capacity is the wrong shape for the job.

Treat it as a probe. Point it at an endpoint you own or a sandbox you are allowed to hit. Do not point it at a production quota you cannot afford to burn.

#!/usr/bin/env python3
"""retry_ticket.py — account for retries as new jobs, not continuations.

Labeled measurement method. Not a vendor benchmark.
Run it only against a URL you are allowed to probe.
"""
from __future__ import annotations

import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass

@dataclass
class Attempt:
    n: int
    status: str
    wait_s: float
    backoff_s: float
    body_tokens_est: int
    error: str = ""

@dataclass
class TicketReport:
    attempts: int
    ok: bool
    total_wait_s: float
    total_backoff_s: float
    retry_multiplier: float
    est_tokens: int
    wall_s: float
    verdict: str

RETRYABLE = {408, 409, 429, 500, 502, 503, 504}

def est_tokens(text: str) -> int:
    # Rough and local. Do not treat this as a billing API.
    return max(1, len(text) // 4)

def post_once(url: str, payload: dict, timeout: float) -> tuple[int, str]:
    data = json.dumps(payload).encode("utf-8")
    req = urllib.request.Request(
        url,
        data=data,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = resp.read().decode("utf-8", errors="replace")
            return resp.status, body
    except urllib.error.HTTPError as e:
        body = e.read().decode("utf-8", errors="replace")
        return e.code, body
    except Exception as e:
        return 0, str(e)

def run_ticket(
    url: str,
    prompt: str,
    max_retries: int = 3,
    timeout: float = 30.0,
    base_backoff: float = 0.5,
) -> TicketReport:
    attempts: list[Attempt] = []
    t0 = time.monotonic()
    payload = {"prompt": prompt, "stream": False}
    ok = False

    for n in range(max_retries + 1):
        t_wait0 = time.monotonic()
        status, body = post_once(url, payload, timeout)
        wait_s = time.monotonic() - t_wait0
        token_est = est_tokens(prompt) + est_tokens(body)
        retryable = status in RETRYABLE or status == 0
        attempts.append(
            Attempt(
                n=n + 1,
                status=str(status),
                wait_s=round(wait_s, 3),
                backoff_s=0.0,
                body_tokens_est=token_est,
                error="" if status and status < 400 else body[:200],
            )
        )
        if not retryable and 200 <= status < 300:
            ok = True
            break
        if n == max_retries:
            break
        backoff = base_backoff * (2 ** n)
        attempts[-1].backoff_s = backoff
        time.sleep(backoff)
        # New ticket effect: the next job often carries the bruise.
        payload["prompt"] = prompt + f"\n# prior_status={status}"

    wall = time.monotonic() - t0
    total_wait = sum(a.wait_s for a in attempts)
    total_backoff = sum(a.backoff_s for a in attempts)
    est = sum(a.body_tokens_est for a in attempts)
    multiplier = len(attempts) * (1.0 + total_wait / max(wall, 0.001))

    if not ok:
        verdict = "no_success"
    elif len(attempts) == 1 and total_wait < 2.0:
        verdict = "first_ticket_fine"
    elif len(attempts) >= 3 or total_wait > 8.0:
        verdict = "retry_tax_high"
    else:
        verdict = "sample_again"

    report = TicketReport(
        attempts=len(attempts),
        ok=ok,
        total_wait_s=round(total_wait, 3),
        total_backoff_s=round(total_backoff, 3),
        retry_multiplier=round(multiplier, 3),
        est_tokens=est,
        wall_s=round(wall, 3),
        verdict=verdict,
    )
    print(json.dumps({"report": asdict(report), "attempts": [asdict(a) for a in attempts]}, indent=2))
    return report

if __name__ == "__main__":
    endpoint = os.environ.get("PROBE_URL", "http://127.0.0.1:8080/v1/complete")
    text = os.environ.get("PROBE_PROMPT", "Return a one-line JSON object with key ok.")
    run_ticket(endpoint, text)

Keep the first run on localhost. A noisy loop should not surprise a shared quota.

chmod +x retry_ticket.py
PROBE_URL="http://127.0.0.1:8080/v1/complete" python3 retry_ticket.py

If you do not have a local stub, stand one up that fails twice, then succeeds. You want ticket 2 and ticket 3 on screen. A green first call teaches you nothing about tax.

# flaky_stub.py — labeled example, not a production gateway
from http.server import BaseHTTPRequestHandler, HTTPServer

HITS = {"n": 0}

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        HITS["n"] += 1
        length = int(self.headers.get("Content-Length", "0"))
        _ = self.rfile.read(length)
        if HITS["n"] < 3:
            self.send_response(429)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(b'{"error":"busy"}')
            return
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"ok":true}')

    def log_message(self, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8080), H).serve_forever()

Start the stub in one terminal. Run the probe in another. You should see three attempts, two backoffs, and a verdict that is not first_ticket_fine. That is the shape of a polite failure. The first prompt was cheap. The job was not.

How to read the report

attempts is the number of tickets you bought. total_wait_s is time the lane spent not answering. total_backoff_s is time you volunteered. retry_multiplier is a combined signal, not a scientific constant. verdict is a suggestion you can override, not a policy engine.

If first_ticket_fine never shows up on a ten-run sample, stop calling the lane spare capacity. Call it a lottery window. Lotteries are fine for sketches. They are a bad backbone for an agent that retries on every malformed tool call.

You can hang the same check on CI. Fail the job when the last N probes land on retry_tax_high. That is cheaper than meeting the tax after an agent loop has already packed errors back into context.

Read wait and backoff as separate columns. If backoff dominates, you wrote the delay. If wait dominates, the lane wrote it. Mixing them into one "the model was slow" story is how teams keep the wrong client defaults.

Where a free lane still helps

You still need somewhere to learn the shape of the tax without pointing a paid key at a flaky stub. Measurement wants a lane you can fail in. That is different from a lane you can ship on.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is a reasonable sandbox for running retry_ticket.py against live responses without parking the experiment on a production key. It is not reserved capacity, and it does not change the accounting. If the probe says retry_tax_high, do not argue with the brand. Change the job: fewer retries, smaller context, or a paid lane with a known queue.

Use the sandbox to learn the ticket count. Then take the verdict back to whatever you actually ship.

When spare capacity is the wrong bet

Spare capacity is the wrong bet when your client hides retries, when each retry grows the prompt, when queue wait is not a first-class metric, and when success on attempt one is the only scenario in the design doc. It is also the wrong bet when a human is blocked on the result, even if nobody wrote an SLA. Blocked humans are a clock. You are already paying them.

Spare capacity can still be the right bet for offline eval, for prompt drafts, for a throwaway probe. The difference is intent. If a retry would make you start over in the same line, you are not using spare capacity. You are using a crowded counter and calling the wait a discount.

A backup generator is not cheaper electricity. It is insurance with a fuel cost. A free inference lane is the same kind of object. Insurance that you retry through is just a slow outage with extra tokens.

Interactive work and batch work do not share a ticket price. A nightly eval can absorb three 429s and a long wait. A developer sitting on a tool call cannot. If your client uses one retry policy for both, the policy is lying to one of them.

Limitations

This method estimates tokens by character count. Real tokenizers disagree. The multiplier is a heuristic, not a price. The stub is deterministic so you can learn the workflow. Live lanes are not. Availability of any free lane can change without notice. Do not publish one run as a vendor comparison. Do not aim this at endpoints you do not own or lack permission to test.

The script also treats HTTP status as truth. Some gateways return 200 with an error object in the body. Parse that if it is your world. A forty-line probe does not replace tracing. It only stops you from budgeting a single prompt as if retries were free sequels.

Thresholds in run_ticket are starting knobs, not physics. Two seconds and eight seconds will be wrong for some jobs. Change them after you watch your own wait column. Do not copy the constants into a slide as if they were measured on your traffic.

Who should not use this

Do not use this approach if you are on a hard customer SLA and you do not already have a paid fallback. Do not use it as load generation against a shared class cluster. Do not use it if your "retry" is actually a human edit in the loop; that is a different ticket with a different cost. Do not use it to keep max_retries high because the tokens still look free. That is how quiet tax becomes a culture.

If you cannot log attempt number, wait, and prompt size together, fix observability first. A calculator without those three fields will flatter you.

Teams that already buy reserved throughput do not need this to choose a vendor. They may still want the probe as a regression check when someone "temporarily" points a client at a spare lane.

Put the second call in the client

Price the second call in code, not in a slide. Cap retries at one for interactive work. Cap prompt growth: do not append the full error blob; append a short code. Keep a paid path next to the spare path, and pick the path before the first ticket, not after the third.

You can add a boring guard.

def allow_spare(report: TicketReport, interactive: bool) -> bool:
    if interactive and report.attempts > 1:
        return False
    if report.verdict == "retry_tax_high":
        return False
    return report.ok

Boring is the point. Cost ops is not a vibe. It is a gate that fires before the agent gets another ticket.

If you want one extra signal, log est_tokens per attempt, not per session. Session totals hide the bruise. Attempt totals show whether retry number two is a longer document pretending to be the same job.

The core conclusion does not move. The first prompt is not the job. The job is every ticket you buy until something answers. Count those tickets. Then decide whether free is still the honest word.