A retry costs the whole context, not the failed call

#aiagents #ai #llm #productivity

A retry in an agent loop is not a second try at a request. It is a second try at the conversation that produced the request, and you're billed for all of it.

Nobody prices that. Every cost dashboard I've seen counts calls, and calls are the wrong unit the moment a turn can fail.

The arithmetic

A plain HTTP client retries by re-sending one request. You pay for that request again, and it's usually the cheapest thing on the bill.

An agent turn does not work that way. To retry, you re-send the whole conversation: the system prompt, the tool results, everything the model has already been shown. And that conversation has grown since the attempt you're repeating, because the attempt itself added to it.

Here's the whole mechanism, with no abstraction layer in the way:

messages = build_turn(task)          # 1x context
for attempt in range(3):
    try:
        return call_model(messages)  # you pay for len(messages)
    except TransientError:
        messages.append(error_about_that_call)
        messages.append("try again") # context is now bigger

Two lines of bookkeeping between attempts, and the next call is priced on a longer prompt. Three attempts at 1x, 2x and 4x is not a retry policy. It is 7 units of spend for work that would have cost 1.

Why "did it succeed" hides this

The usual scoreboard has one column, and it is the wrong one:

model	success rate	cost per call	cost per correct answer
A	0.60	1.0	1.67
B	0.95	1.6	1.68
C	0.99	2.4	2.42

Model A looks best and is the worst deal in the table. On a per-call basis B is more expensive; on the only basis anyone pays for, A and B are level, and C is half again as expensive as either.

That table is worked arithmetic, not a measurement of real models. The shape is the point, and it holds for any model whose failures cluster early in a long turn.

The metric I want is cost per correct answer, because it's the only one that can't be improved by failing less loudly. And retries make it worse in a way that success rate hides: a model that fails and recovers is paying for the failure, and the recovery is counted as a success.

The retry also throws away its own evidence

The first attempt returning a 503 is a real observation about that endpoint. It's the most informative thing in the run.

The retry succeeds, the receipt goes out clean, and now the only surviving record says everything worked. You've paid twice and kept the worse of the two facts.

This is why I have started logging the retry path rather than only the outcome:

receipt.attempts.append({
    "n": attempt,
    "error": repr(e),
    "context_tokens": len(messages),   # the number you were actually billed for
    "outcome": "recovered",
})

Two fields. context_tokens is the one that tells you whether the retry was cheap, and almost nothing logs it.

What I'm not claiming

I argued this in a comment thread under someone else's benchmark and he told me it was a different experiment. He is right, and the scoping matters: his benchmark isolates the retry decision, for API requests, which is a question my argument does not touch.

What survives the concession is only the pricing. Retry-plus-context is a different question from retry-safety, and it is the one your bill answers.

I haven't measured this on my own runs, and I want to be plain about that. The charts and tables above are worked examples with the assumption written down, not telemetry. The reason I believe the shape is that it follows from what the API charges for, not from how any particular agent behaves.

The one thing worth adding

If you take one line from this, print len(messages) next to your retry count. It costs nothing and it's the difference between "retries are expensive" and knowing by how much.

Where this came from, and the argument it came out of, is in the comments on To Retry or Not to Retry? That Is the Question and What one agent run actually costs.

DEV Community