A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the m...
For further actions, you may consider blocking this person and/or reporting abuse
The 4× tax number is the one I'll quote. One addition from the ops side: the retry path also needs idempotency on tool-call results — replaying the full context against the same model isn't just expensive, it can double-apply a tool call if the first attempt actually succeeded server-side and only the response was lost. So it's the router you describe plus an idempotency check; either one alone still leaves a nasty failure mode.
Great addition — the silent-success case is exactly the one people miss when they treat "retry" as just re-running the loop. If the tool call already executed server-side but only the response was dropped, a naive context replay re-derives the same decision and fires it again. Double-apply.
One refinement I'd push: idempotency keys only help when the tool can take one. The cleaner split is to treat the tool layer as its own retry boundary, separate from the LLM context replay. You don't actually need to replay the whole context to retry a failed tool call — re-issue the call with the same idempotency key, and the context-replay loop (the expensive 4× part) only triggers when the model's decision failed, not the tool transport. Two different retry loops, two different cost profiles.
Which raises the messy case: tools with no native idempotency (a bare POST that won't accept a key). Do you wrap them behind an idempotency proxy, or just accept the double-apply risk on the low-stakes ones and only guard the destructive ones?
"Two different retry loops, two different cost profiles" is the cleanest way to frame this, and it's the part most agent frameworks get wrong.
They treat "tool call failed" and "model call failed" as the same thing, so they replay the whole context for both. But a tool call doesn't need the model's context — it just needs the same idempotency key.
Curious how you handle the "tool call succeeded but the result was lost" case. If the model never sees the result, it might decide to call the tool again with different arguments. Does the retry loop re-inject the original result, or does it let the model re-decide?
Good question — and it's the exact failure mode that breaks naive retries.
The fix is ownership: the retry controller, not the model, owns the tool result. On a silent-success (timeout, but the call executed server-side), you don't let the model re-decide. You key the tool call by signature — tool name + args hash — and cache the result; on retry you short-circuit and hand the model the cached result as if it came back clean. The model never gets a second pass at picking different arguments, because it's not in the loop for transport failures.
The trap is when retries live inside the model's own loop — then you can't stop the re-decision. So it's less "retry the tool" and more "the orchestrator replays the cached outcome." That only holds if the tool is idempotent-confirmed; otherwise you're replaying a side effect.
Which exposes the messier case: partial success. If the tool wrote something but you're unsure how much — a batch insert that committed half — the cached-result shortcut is wrong. You need either true idempotency or a fresh failure. How do you signal "ambiguous success" back to the orchestrator without forcing a full re-run?
"Ambiguous success" is the exact state that most retry systems don't have a word for.
You're right that the cached-result shortcut is wrong here, because replaying a partial write is worse than not replaying at all. The only honest answer is to treat "ambiguous success" as a first-class state, not as a failure or a success.
The orchestrator needs a way to say: "I don't know if this fully committed. Here's what I do know: the call was sent, the connection dropped, the tool reports partial completion." The model can then decide whether to retry, compensate, or escalate — but it decides with the actual state, not with a lie.
Treating "ambiguous success" as a first-class state is the part most retry code skips — and you're right that the cached-result shortcut is actively dangerous here, because a partial write replayed is worse than no replay. The line I'd draw: the retry controller owns the tool result, so "ambiguous success" is exactly the one state where it must refuse to auto-retry and instead surface the observed facts (sent / dropped / partial) the way you describe. That keeps the model deciding with real state instead of a guessed one.
The open question for me is where "ambiguous success" lives in the state machine — its own explicit node, or folded into a terminal "needs-human / escalate" bucket? The former gives finer recovery (compensate vs retry), the latter ships simpler. Which way did you land?
Routing-the-task-not-the-model is the right instinct, and the token math is brutal once you see it. One thing I'd put before the retry-budget discussion: idempotency.
A timeout doesn't tell you the call failed — it tells you that you didn't hear back. If the side effect already ran (the payment went, the email sent, the row was written), "carry context forward and advance to the next model" can apply it twice. The cheapest retry policy in the world still double-charges someone if the thing it retried had a side effect.
So I'd gate it: reads retry freely; writes retry only behind an idempotency key, or re-grade the outcome (did the action actually happen?) before re-running. Otherwise the retry tax you measure in tokens comes back later as a reconciliation bill — same lesson as your dashboard point: the real cost is on the path you're not looking at.
Glad this one traveled over from your local-LLM piece — and yes, idempotency first is the right ordering. The retry-budget discussion is moot if a retry can double-apply a side effect; you've capped the cost of the call but doubled the damage. The split I landed on: the retry controller owns the tool result keyed by signature, so a silent-success (heard nothing, but it ran) returns the cached outcome instead of re-firing. That's what makes a cost cap safe to actually enforce. You put it better than I did: a timeout is a communication failure, not an operation failure. How are you surfacing that distinction to the caller — same error type, or a distinct "unknown" status?
The observation that dashboards show cost per successful request is the uncomfortable one. Failed attempts disappear into the average, so the feature looks cheap right up until the invoice arrives.
SkepAgent's point deserves as much attention as the token math. A retry after a tool call that timed out but actually succeeded does not only cost tokens. It can apply the same change twice. So the order matters: make side effects idempotent first, then decide how many retries you can afford.
challan116's suggestion to budget retries as their own line item matches what we do on client builds. When a feature is scoped, it gets an expected cost per run and a separate allowance for failures, written down next to the acceptance criteria. If the failure line starts growing, that is a signal about the prompt or the model choice, and it shows up within days.
When you route to the next model and carry context forward, how do you stop the second model from repeating a side effect the first one already triggered?
One distinction I think matters here is between a failed response and a failed operation. A timeout doesn't prove the side effect didn't happen.
For state-changing operations, I'd separate retrying the transport from retrying the operation itself. If the outcome is ambiguous, preserve the original operation reference and reconcile its status before deciding whether another execution is safe. Idempotency helps, but it doesn't automatically resolve partial completion or providers that don't support idempotency keys.
Good distinction, and it's the one our R4 sketch glossed: retrying the transport (the call didn't arrive) is safe to repeat; retrying the operation (did the side effect land?) needs the reconcile-first step you describe. The trap is treating a timeout as a transport failure when it might be an operation that already committed — re-firing then double-applies. The piece I'd add: the ledger you mention has to be readable by any model in the chain, not just the one that started it, or the handoff to model B reintroduces the same ambiguity you just closed. Question: when the status is still ambiguous after reconcile, do you fail the task or escalate to a human/queue?
The ordering point is the crux — idempotency has to be a precondition, not a retry setting. You can't retry your way out of a non-idempotent side effect; the retry just multiplies the damage.
On the cross-model case: the same "retry controller owns the tool result" boundary from earlier helps here too. If model B's attempt is a transport failure (not a decision failure), the cached result from model A's already-successful call short-circuits — so B never re-fires the side effect, it just receives the result as if it came back clean. The nasty case is partial: A triggered the side effect but the result was lost, and B is a different model that can't see A's cache. There you need a side-effect ledger keyed by task id, not by model — so any model in the chain can check "did this task's effect already land?" before acting.
Question: do you run that side-effect ledger as a separate service, or bake it into the orchestrator? Separate is cleaner to reason about but adds a hop on every call.
I'd lean toward keeping the ledger logically separate from the orchestrator, but not necessarily deploying it as a separate service from day one.
For state-changing financial operations, I'd initially persist the operation record, provider reference, and execution status alongside the transaction state in a durable database. The orchestrator can consult that record before deciding whether an operation is safe to execute again.
The important distinction for me is between a transport failure and an ambiguous operation outcome. If a provider may have completed the side effect but the response was lost, I'd reconcile the original operation before considering another execution.
I'm exploring this in Agentic Switcher, where the focus is deterministic orchestration, ownership fencing, and safe recovery across provider integrations. A separate ledger service could make sense later, but I'd first establish the consistency and recovery semantics.
One question I'd be interested in exploring is how you'd handle a side effect that succeeded externally but whose ledger update failed. That seems to be the hardest boundary to get right.
This matches what I see — logical separation, physical colocation early. The trap is making the ledger a synchronous dependency of the retry path; then a ledger hiccup stalls exactly the operation it's supposed to protect. Keeping it as a side-write (operation record + provider ref + status) that the retry reads but never blocks on is the version that actually ships. The real job of the ledger isn't audit, it's decision — "have I already committed this signature?" has to be answerable in microseconds or the retry just fires anyway. How are you gating the retry on the ledger read — optimistic fire-then-check, or a hard gate?
One correction and one number.
The correction: "context paid once" doesn't hold, and stefan's question above is the right one. Model B has no memory of model A. It receives the context and pays for it. What routing changes is the rate on that send, cache read or full ingestion, never the number of sends. Every call re-sends the transcript, so the unit that matters is turns, not attempts.
The number: on 4,451 token-bearing turns across 53 sessions, the median turn pushed 135,267 tokens. Across the corpus 92.9% of the tokens were cache reads, 6.7% fresh input, and the model generated 0.41% of what it read. Priced at 0.1 for a cache read against 1 for fresh input, losing the cache on a turn multiplies that turn's cost by a median of 8.1x (IQR 6.5 to 9.0).
The part that changes the advice is where that cache is lost. Appending the error text to the end of the prompt is cheap, because the prefix still matches and only the new suffix gets billed fresh. What's expensive is changing anything before the end. Editing the system prompt, re-summarizing history, or moving to a model with its own cache namespace reprices everything from the first changed token. We measured the same shape in context compaction: after a rebuild, 69.4% of first turns were more than half fresh input.
So a same-model retry against a warm cache costs roughly what the failed attempt cost, not four times it. The tax arrives when the fix rewrites the front of the prompt.
The instrument I'd add to the billing join above is cache reads divided by total input, per turn. A retry that appends barely moves it. A rewrite drops it, and the drop is the bill.
The correction lands harder than mine did — and the number reframes the whole thing. "Turns, not attempts" is the right unit; the retry that re-sends the transcript isn't free even at a better rate, because the cost that moves is the cache, not the count. Your 92.9% cache-read corpus is the part I under-weighted: a slim handoff helps for exactly the reason your data shows — it keeps the prefix stable, and a stable prefix is what keeps the cache warm. Re-attaching the full original on retry is what blows it, and at your median that's an 8.1× turn. So "slim handoff" and "where the cache is lost" are the same lever seen from two sides; I'd just add that the retry budget should be counted in cache-lost turns, not attempt counts, because the bill isn't the retry — it's the re-warm. The part I'm still unsure on: when you hand off to model B that needs different framing, how do you keep the prefix from breaking — carry a frozen prefix, or accept the cache loss as the price of the switch?
You're right that the bill is the re-warm, and to your question I'd answer the boring way: we accept the loss.
A frozen prefix doesn't survive a switch. The receiving model has never seen that text, so whatever the sender cached can't be spent on it, and the first turn pays full input on however much you carry. What stays warm is the second turn onward, and only if nothing before the end of the handoff changes.
The number that made me stop treating a rebuild as free came out of compaction, which is the same event from a different direction. Across 62 rebuilds in two long sessions, 69.4% had a first turn that was more than half fresh input. Those rebuilds did pay themselves back in about 14 turns, but only because the rebuild also halves what gets re-sent afterwards. A model switch has no shrink, so it's the cost without the refund. It has to earn back on rate alone.
So the rule I'd give you: switch late, and switch on a turn where the prefix was going to break anyway. If the retry was already re-summarizing the history, the switch is close to free because you were paying for the rebuild regardless.
I'm not following the "context paid once" part.
model.call(task.handoff_context)still sends input to the next model. Is your gateway absorbing that cost, or is the handoff context smaller? Switching providers alone doesn't explain the drop from 4x to 1x.Fair challenge — "paid once" was sloppy framing on my part. Model B absolutely pays for what it receives; routing doesn't make tokens free. What it actually changes is three things: (1) the rate — cheap steps hit cheap models; (2) the handoff carries the slimmed context (results + deltas), not the full original replay; (3) a retry that doesn't re-attach the full original. The 4×→1× lives mostly in (2) and (3), not in "paying once." You're right that switching providers alone barely moves the number — the win needs the smaller handoff and a retry that doesn't re-bloat. How are you scoping what travels in the handoff today — full context, or a trimmed carry?
The audit trick that worked for us: log
cache_read_input_tokensnext to each attempt id, then join retries by task id — the replay tax falls out of the billing data as input-tokens-minus-cache-reads per attempt, no app instrumentation needed.One counterpoint to the routing fix: "equivalent" drifts. A fallback model with a different tool-calling format silently degrades the handoff context, so you trade a 4x bill for a quiet quality drop. Worth logging which tier actually served each attempt.
Does your handoff context carry tool definitions too? If model B doesn't expose the same tools, the task may need a re-plan rather than a rerun.
The audit trick is exactly right, and the part I like is that it needs zero app instrumentation — the replay tax just falls out of billing data once you join by task id.
On the format-drift counterpoint: yes, that's the silent killer. The tool schema isn't "data" you carry, it's part of the context the model reasons over. When model B speaks a different tool dialect, replaying the same tool_calls doesn't just degrade — it breaks, and the failure looks like a model problem, not a routing one. The split that's worked for us: the routing layer validates tool compatibility before handoff. If B can't express the same tools, you don't rerun — you either re-plan against B's toolset or you translate the schema. Rerunning the identical call is the worst of both.
Question for you: do you pin tool schemas per-model in a registry up front, or infer compatibility at call time? The up-front pinning catches the drift before it ships, but it rots the moment a provider changes a format.
Answering the closing question from our own incidents: the retry replays more than the first attempt did. The failed attempt's scratch reasoning and error-handling turns get folded back in, so attempt 3 costs noticeably more than attempt 1, and the dashboard still showed one tidy "task succeeded" line either way.
What fixed it for us was two boring rules. First, retry by token budget, not attempt count — when the budget's gone, the task fails loudly instead of succeeding expensively. Second, the retried payload is frozen: no new reasoning, no re-planning, same call against a healthy endpoint. Retries still cost extra, but the number is bounded, predictable, and it's a line item in the dashboard now.
These are the two rules most teams skip, and "frozen payload" is the one that separates a real retry from a quiet re-plan. A retry that re-reasons doesn't just cost more — it can change the outcome, so "attempt 3 succeeded" and "attempt 1 would have succeeded the same way" are not the same claim. The token-budget cap is what makes "fail loudly" safe instead of "succeed expensively." One addition from our side: when the budget's gone, what do you return? A structured "aborted at step N, last-good state X" beats a generic timeout, because the caller can branch instead of blindly re-issuing. Question: do you surface the budget consumption per attempt on the caller's dashboard, or only at the final failure?
The dashboard point is the one that generalizes furthest: any metric averaged per successful request will hide the retry tax, because the failures get folded into the cost of the eventual success. Retries also concentrate twice over — they fire at peak load, and they fire on the longest, messiest inputs — so the unbudgeted path is expensive in exactly the conditions where the system can least afford it. Budgeting retries as their own line item, capped by cost rather than by attempt count, is the version of this most teams could adopt without changing their architecture at all.
This is the most adoptable version of the whole argument — no architecture change, just accounting, which is why it'll actually ship in teams that won't touch their retry logic.
The "capped by cost, not attempt count" refinement is the key detail most people miss. An attempt-count cap lets a single huge context blow the budget while a cheap call gets throttled for no reason — the two failures have opposite shapes. Cost-capping flips it so the expensive replay is the thing that gets stopped.
One addition that makes it land sooner: surface the retry budget at the dashboard level, not just the invoice. You said it shows up "within days" when it's scoped next to acceptance criteria — it shows up within hours if the dashboard itself turns red when the retry line crosses its allowance. Visibility is what converts this from a post-mortem into a guardrail.
Question: do you budget retries per-route or per-feature? Per-route catches the one expensive endpoint; per-feature catches the feature that's quietly retry-heavy across routes.
"You don't make retries cheaper by retrying harder" is the core insight here, and it's the opposite of what most teams do.
The default response to "the model is down" is "retry with the same model, hope it comes back". But if the model is down, retrying it doesn't help — routing to a different model does.
Curious how you handle the "no equivalent model available" case. If all the models in the registry are down, does the task fail gracefully, or does it queue and wait?
"Retrying harder" is just re-paying the 4× tax with more determination — the insight holds, and it's the opposite of instinct.
On the no-equivalent case, it depends entirely on the task's latency contract, and the trap is treating both the same:
The mistake is a global "retry N times then give up" that applies the synchronous logic to async work (wasted waits) or the async logic to sync work (timeout blowups).
Question: do you split failure modes by task class (sync vs async, read vs write), or by a single global timeout? Class-based is more code but each path actually fits its caller.
"Class-based is more code but each path actually fits its caller" is the right tradeoff, and it's the one most teams avoid because it's harder to explain in a design doc.
The split isn't just sync vs async. It's also read vs write, and inside writes, idempotent vs non-idempotent. A sync read can fail fast and serve a cached result. A sync write needs to know whether it committed. An async read can queue. An async write can queue, but only if it's idempotent — otherwise you're queuing a duplicate.
Four classes, four different retry policies. It's more code, but each one is simpler than a global policy trying to handle all four.
The prefix cache tax is where this bites hardest. On long contexts, the first attempt hits warm KV cache and costs fractional input rates. The retry that fires during an outage or failover lands on a cold replica or a fallback model without cached state, paying full prompt ingestion prices four times in a row. A failure doesn't just multiply tokens; it drops you back to cold-start unit economics on the largest inputs in the pipeline.
This is the sharpest extension of the 4× thesis, because the tax isn't 4× the same price — it's 4× a higher unit price. You lose the warm cache, so each retry pays full ingestion on the largest context in the pipeline. The failure doesn't just multiply tokens, it multiplies the rate.
Two mitigations that have helped:
Question: have you measured the cache-hit-rate drop specifically on retry paths, or only on the aggregate? The aggregate hides it — the retry path can be near-zero hit rate while the overall number looks healthy, which is exactly why it slips past the dashboard.
This is a massive blind spot for so many teams running long-context models.
Beyond the immediate 4× token cost, another nightmare scenario happens when teams combine uncapped Exponential Backoff with In-Memory Caching (Prompt Caching). If the retry delay exceeds the provider's cache TTL window (e.g., 5 minutes for Anthropic or OpenAI), you not only repay the context tokens, but you also lose the 50–90% prompt caching discount on every subsequent retry.
Treating model failures as a routing problem rather than an HTTP retry problem is the exact mindset shift backend engineers need to make when moving from traditional APIs to LLM infrastructure. Great write-up!
This is the sharpest addendum to the cache point, because the retry schedule is itself a cache-killer. Uncapped exponential backoff doesn't just add latency — if the gap crosses the provider's cache TTL (your 5-minute Anthropic/OpenAI window), every subsequent retry repays the whole prefix at full price AND loses the 50–90% discount. So the backoff ceiling isn't a reliability knob, it's part of the cost model. Pairs directly with theagentloop's 8× number: the multiplier isn't only from re-sending, it's from re-sending cold. Question: do you pin the backoff ceiling to the provider's cache TTL, or just hope retries land inside the window?
Tokenade is my project, and a retry is not a cheap second try: it replays the whole context, so each attempt pays the full prefix again. Cost per successful task shows that, where cost per call hides it.
the 4x holds for a timeout, but where the call dies changes the bill a lot. a provider that errors before the first token usually costs you nothing, one that dies mid stream already charged you for the full prefill. we log failure stage next to attempt count, otherwise the dashboard averages the two together.
one gap in the failover version: switching models drops the prompt cache, so the next model pays full price on that 50k context anyway. we retry the same provider first while the cache is warm and only fail over on hard errors.
did you compare cache hit rate on retries versus failovers?
This is exactly the risk I designed around in a Node agent I run. Two things helped: a per-tool timeout (45s), so a hung data source fails once instead of stalling the whole turn, and falling back to a different model provider instead of retrying the same one. Retrying an identical long-context request against a provider that's already slow mostly buys you the same failure, at full context price.
This matches what I see running LLM traffic behind an API gateway. Retries replay the full context and the bill lands on the shared budget, not the caller. Making the task travel with its context instead of replaying it is the right call.