A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the model client and never put it in the budget.
That retry path only fires when the model is slow or down — i.e. at peak load. And every retry replays the entire context. So the one part of the system nobody accounts for is the part that costs the most, exactly when you can afford it least.
The number that surprises people
Take a support agent for an ASEAN e-commerce shop. A hard ticket pulls a 50k-token context (order history, policy docs, prior turns) and emits ~1k tokens.
- One attempt ≈ 51k tokens.
- With
retries: 3and the task still failing, you pay for 4 attempts: 51k × 4 ≈ 204k tokens for a task you budgeted at 51k.
That's a 4× tax — and it isn't spread evenly. The tasks that fail are the expensive ones: long context, heavy tool chains, messy inputs. Retries concentrate exactly where cost is highest.
retry_policy:
naive:
attempts: 3
on_failure: "replay full context to same model"
hidden_cost_multiplier: 4 # 1 attempt + 3 full-context replays
shows_up_in_dashboard: false
cost_aware:
attempts: 3
on_failure: "route to next equivalent model"
carries_context: true
hidden_cost_multiplier: 1 # context travels, never replayed
shows_up_in_dashboard: true
Why the bill stays invisible
Most cost dashboards report average cost per successful request. A task that fails three times and finally succeeds doesn't appear as "4× cost" — it appears as one success with a latency blip. The retry is a hidden fourth axis of spend:
- Latency triples → someone notices.
- Cost triples → almost nobody measures it, because it's buried in "failed then retried" traffic.
So teams tune the happy path and ignore the one code path that runs under stress and multiplies the bill.
The fix is routing, not retrying
You don't make retries cheaper by retrying harder. You stop replaying context: when a model is unavailable, advance the task to the next equivalent model that can take the same handoff context. The context travels with the task; you don't re-pay for it as a "retry."
With 25+ models behind one OpenAI-compatible endpoint, "equivalent" stops being theoretical — there's a tier you can actually route to without rewriting prompts or adapters. For teams under data-residency rules (PDPA-aligned, SG-hosted), this also matters: re-running context on a different provider just to "retry" raises compliance questions, not just cost.
flowchart LR
subgraph Naive[Naive retry: context replayed]
N1[Task · 50k ctx] --> N2[Attempt 1: 50k in]
N2 -->|fail| N3[Replay 50k in]
N3 -->|fail| N4[Replay 50k in]
N4 -->|fail| N5[Replay 50k in]
N1 -.paid 4x.-.-> N5
end
subgraph Smart[Cost-aware: context travels]
S1[Task · 50k ctx] --> S2[Model A call]
S2 -->|unavailable| S3[Model B · same ctx]
S3 -->|unavailable| S4[Model C · same ctx]
S1 -.ctx paid once.-.-> S4
enddef step(task, model_registry):
# cost-aware: context is paid ONCE, then travels with the task
for model in model_registry.ordered_for(task.capability):
try:
return model.call(task.handoff_context) # no replay
except Unavailable:
continue # next equivalent, same context
raise NoCapableModel
# for contrast — the naive version re-sends the FULL context every attempt
def naive_retry(task, model, attempts=3):
for _ in range(attempts):
try:
return model.call(task.full_context) # full context RE-SENT each time
except (Timeout, Unavailable):
continue
raise RetryExhausted
The router doesn't retry the model. It advances the task to the next model that can take its context. Same uptime, a fraction of the bill.
The question I'll leave you with
Before you ship retries: 3: for the tasks that fail, how many tokens does the retry actually replay — and is that number in any dashboard you look at?
No tool to sell, no link to click. Just this: most LLM systems that survive a model outage are just retry loops with better logging — and the loop is the most expensive thing they do.
Top comments (8)
The 4× tax number is the one I'll quote. One addition from the ops side: the retry path also needs idempotency on tool-call results — replaying the full context against the same model isn't just expensive, it can double-apply a tool call if the first attempt actually succeeded server-side and only the response was lost. So it's the router you describe plus an idempotency check; either one alone still leaves a nasty failure mode.
Great addition — the silent-success case is exactly the one people miss when they treat "retry" as just re-running the loop. If the tool call already executed server-side but only the response was dropped, a naive context replay re-derives the same decision and fires it again. Double-apply.
One refinement I'd push: idempotency keys only help when the tool can take one. The cleaner split is to treat the tool layer as its own retry boundary, separate from the LLM context replay. You don't actually need to replay the whole context to retry a failed tool call — re-issue the call with the same idempotency key, and the context-replay loop (the expensive 4× part) only triggers when the model's decision failed, not the tool transport. Two different retry loops, two different cost profiles.
Which raises the messy case: tools with no native idempotency (a bare POST that won't accept a key). Do you wrap them behind an idempotency proxy, or just accept the double-apply risk on the low-stakes ones and only guard the destructive ones?
"Two different retry loops, two different cost profiles" is the cleanest way to frame this, and it's the part most agent frameworks get wrong.
They treat "tool call failed" and "model call failed" as the same thing, so they replay the whole context for both. But a tool call doesn't need the model's context — it just needs the same idempotency key.
Curious how you handle the "tool call succeeded but the result was lost" case. If the model never sees the result, it might decide to call the tool again with different arguments. Does the retry loop re-inject the original result, or does it let the model re-decide?
This matches what I see running LLM traffic behind an API gateway. Retries replay the full context and the bill lands on the shared budget, not the caller. Making the task travel with its context instead of replaying it is the right call.
Tokenade is my project, and a retry is not a cheap second try: it replays the whole context, so each attempt pays the full prefix again. Cost per successful task shows that, where cost per call hides it.
The dashboard point is the one that generalizes furthest: any metric averaged per successful request will hide the retry tax, because the failures get folded into the cost of the eventual success. Retries also concentrate twice over — they fire at peak load, and they fire on the longest, messiest inputs — so the unbudgeted path is expensive in exactly the conditions where the system can least afford it. Budgeting retries as their own line item, capped by cost rather than by attempt count, is the version of this most teams could adopt without changing their architecture at all.
The prefix cache tax is where this bites hardest. On long contexts, the first attempt hits warm KV cache and costs fractional input rates. The retry that fires during an outage or failover lands on a cold replica or a fallback model without cached state, paying full prompt ingestion prices four times in a row. A failure doesn't just multiply tokens; it drops you back to cold-start unit economics on the largest inputs in the pipeline.
"You don't make retries cheaper by retrying harder" is the core insight here, and it's the opposite of what most teams do.
The default response to "the model is down" is "retry with the same model, hope it comes back". But if the model is down, retrying it doesn't help — routing to a different model does.
Curious how you handle the "no equivalent model available" case. If all the models in the registry are down, does the task fail gracefully, or does it queue and wait?