A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the model client and never put it in the budget.
That retry path only fires when the model is slow or down — i.e. at peak load. And every retry replays the entire context. So the one part of the system nobody accounts for is the part that costs the most, exactly when you can afford it least.
The number that surprises people
Take a support agent for an ASEAN e-commerce shop. A hard ticket pulls a 50k-token context (order history, policy docs, prior turns) and emits ~1k tokens.
- One attempt ≈ 51k tokens.
- With
retries: 3and the task still failing, you pay for 4 attempts: 51k × 4 ≈ 204k tokens for a task you budgeted at 51k.
That's a 4× tax — and it isn't spread evenly. The tasks that fail are the expensive ones: long context, heavy tool chains, messy inputs. Retries concentrate exactly where cost is highest.
retry_policy:
naive:
attempts: 3
on_failure: "replay full context to same model"
hidden_cost_multiplier: 4 # 1 attempt + 3 full-context replays
shows_up_in_dashboard: false
cost_aware:
attempts: 3
on_failure: "route to next equivalent model"
carries_context: true
hidden_cost_multiplier: 1 # context travels, never replayed
shows_up_in_dashboard: true
Why the bill stays invisible
Most cost dashboards report average cost per successful request. A task that fails three times and finally succeeds doesn't appear as "4× cost" — it appears as one success with a latency blip. The retry is a hidden fourth axis of spend:
- Latency triples → someone notices.
- Cost triples → almost nobody measures it, because it's buried in "failed then retried" traffic.
So teams tune the happy path and ignore the one code path that runs under stress and multiplies the bill.
The fix is routing, not retrying
You don't make retries cheaper by retrying harder. You stop replaying context: when a model is unavailable, advance the task to the next equivalent model that can take the same handoff context. The context travels with the task; you don't re-pay for it as a "retry."
With 25+ models behind one OpenAI-compatible endpoint, "equivalent" stops being theoretical — there's a tier you can actually route to without rewriting prompts or adapters. For teams under data-residency rules (PDPA-aligned, SG-hosted), this also matters: re-running context on a different provider just to "retry" raises compliance questions, not just cost.
flowchart LR
subgraph Naive[Naive retry: context replayed]
N1[Task · 50k ctx] --> N2[Attempt 1: 50k in]
N2 -->|fail| N3[Replay 50k in]
N3 -->|fail| N4[Replay 50k in]
N4 -->|fail| N5[Replay 50k in]
N1 -.paid 4x.-.-> N5
end
subgraph Smart[Cost-aware: context travels]
S1[Task · 50k ctx] --> S2[Model A call]
S2 -->|unavailable| S3[Model B · same ctx]
S3 -->|unavailable| S4[Model C · same ctx]
S1 -.ctx paid once.-.-> S4
enddef step(task, model_registry):
# cost-aware: context is paid ONCE, then travels with the task
for model in model_registry.ordered_for(task.capability):
try:
return model.call(task.handoff_context) # no replay
except Unavailable:
continue # next equivalent, same context
raise NoCapableModel
# for contrast — the naive version re-sends the FULL context every attempt
def naive_retry(task, model, attempts=3):
for _ in range(attempts):
try:
return model.call(task.full_context) # full context RE-SENT each time
except (Timeout, Unavailable):
continue
raise RetryExhausted
The router doesn't retry the model. It advances the task to the next model that can take its context. Same uptime, a fraction of the bill.
The question I'll leave you with
Before you ship retries: 3: for the tasks that fail, how many tokens does the retry actually replay — and is that number in any dashboard you look at?
No tool to sell, no link to click. Just this: most LLM systems that survive a model outage are just retry loops with better logging — and the loop is the most expensive thing they do.
Top comments (110)
The 4× tax number is the one I'll quote. One addition from the ops side: the retry path also needs idempotency on tool-call results — replaying the full context against the same model isn't just expensive, it can double-apply a tool call if the first attempt actually succeeded server-side and only the response was lost. So it's the router you describe plus an idempotency check; either one alone still leaves a nasty failure mode.
Great addition — the silent-success case is exactly the one people miss when they treat "retry" as just re-running the loop. If the tool call already executed server-side but only the response was dropped, a naive context replay re-derives the same decision and fires it again. Double-apply.
One refinement I'd push: idempotency keys only help when the tool can take one. The cleaner split is to treat the tool layer as its own retry boundary, separate from the LLM context replay. You don't actually need to replay the whole context to retry a failed tool call — re-issue the call with the same idempotency key, and the context-replay loop (the expensive 4× part) only triggers when the model's decision failed, not the tool transport. Two different retry loops, two different cost profiles.
Which raises the messy case: tools with no native idempotency (a bare POST that won't accept a key). Do you wrap them behind an idempotency proxy, or just accept the double-apply risk on the low-stakes ones and only guard the destructive ones?
"Two different retry loops, two different cost profiles" is the cleanest way to frame this, and it's the part most agent frameworks get wrong.
They treat "tool call failed" and "model call failed" as the same thing, so they replay the whole context for both. But a tool call doesn't need the model's context — it just needs the same idempotency key.
Curious how you handle the "tool call succeeded but the result was lost" case. If the model never sees the result, it might decide to call the tool again with different arguments. Does the retry loop re-inject the original result, or does it let the model re-decide?
Good question — and it's the exact failure mode that breaks naive retries.
The fix is ownership: the retry controller, not the model, owns the tool result. On a silent-success (timeout, but the call executed server-side), you don't let the model re-decide. You key the tool call by signature — tool name + args hash — and cache the result; on retry you short-circuit and hand the model the cached result as if it came back clean. The model never gets a second pass at picking different arguments, because it's not in the loop for transport failures.
The trap is when retries live inside the model's own loop — then you can't stop the re-decision. So it's less "retry the tool" and more "the orchestrator replays the cached outcome." That only holds if the tool is idempotent-confirmed; otherwise you're replaying a side effect.
Which exposes the messier case: partial success. If the tool wrote something but you're unsure how much — a batch insert that committed half — the cached-result shortcut is wrong. You need either true idempotency or a fresh failure. How do you signal "ambiguous success" back to the orchestrator without forcing a full re-run?
"Ambiguous success" is the exact state that most retry systems don't have a word for.
You're right that the cached-result shortcut is wrong here, because replaying a partial write is worse than not replaying at all. The only honest answer is to treat "ambiguous success" as a first-class state, not as a failure or a success.
The orchestrator needs a way to say: "I don't know if this fully committed. Here's what I do know: the call was sent, the connection dropped, the tool reports partial completion." The model can then decide whether to retry, compensate, or escalate — but it decides with the actual state, not with a lie.
Treating "ambiguous success" as a first-class state is the part most retry code skips — and you're right that the cached-result shortcut is actively dangerous here, because a partial write replayed is worse than no replay. The line I'd draw: the retry controller owns the tool result, so "ambiguous success" is exactly the one state where it must refuse to auto-retry and instead surface the observed facts (sent / dropped / partial) the way you describe. That keeps the model deciding with real state instead of a guessed one.
The open question for me is where "ambiguous success" lives in the state machine — its own explicit node, or folded into a terminal "needs-human / escalate" bucket? The former gives finer recovery (compensate vs retry), the latter ships simpler. Which way did you land?
I landed on the explicit node (the former).
Folding it straight into a terminal "needs-human" bucket is tempting for simplicity, but it throws away the opportunity for automated compensation. If a tool call returns "ambiguous success" (e.g., a timeout on a write operation), the system shouldn't immediately wake up a human.
Instead, the explicit node lets the agent run a read-only "state check" or "idempotency check" tool to figure out if the write actually succeeded. Only if that check also fails do we route to the terminal escalation bucket. It's a bit more work on the state machine, but it saves a massive amount of human interruption!
This is the right call — the "needs-human" default is what quietly burns the most expensive resource on problems a machine could settle. The part I like is that the state check is itself a tool the agent calls, so the human stays out of the loop until there's a real judgment to make. The risk I'd watch is the check becoming its own retry loop: if the read is flaky, "ambiguous success" can bounce between the check and escalation forever. How do you bound the number of state-check attempts before it's allowed to wake a human?
The risk of the check becoming a retry loop is real. In my experience, the best way to bound it is a strict state machine with a retry counter.
If the agent hits the 'ambiguous' state, it increments a counter. Once that counter hits a hard limit (e.g., 3), the agent is physically forbidden from calling the check tool again and must yield to the human. If the read is flaky, no amount of agent retries will fix the underlying system — escalation is the only sane path.
The hard-limit-and-yield-to-human part is the piece most teams skip, and it's the part that actually stops the loop. One thing we learned the hard way: the counter has to live somewhere the agent process can't reset. We kept it in agent local state at first, and a restart silently zeroed it — suddenly the bound was gone. Once we moved it out, the 'ambiguous' state actually meant something.
Curious whether you persist the counter in the ledger too, or keep it in the agent's own state?
This is the difference between 'retry with backoff' and 'retry with a leash.' The leash only works if the agent can't slip it off by restarting.
The leash metaphor is perfect — backoff alone has that gap where a restart just slips the leash off and the budget resets. The interesting failure is exactly the one you named: an agent that respawns a process to dodge the ceiling. Is your leash enforced outside the agent (process-level hard cap) or inside the framework? And how do you stop the agent from spawning a fresh process that doesn't inherit the counter?
Process-level hard cap, enforced by a supervisor. The agent can spawn whatever it wants, but the supervisor checks the shared counter before allowing a new process to start. If the budget is exhausted, the spawn is refused. The counter isn't in the agent's memory; it's in a store the agent can't touch.
That's the leash living outside the agent, which is the only version that actually holds — a clever respawn can't silently reset a counter it doesn't own. The detail that matters: is the counter per-task or global across tasks? If it's per-task, how do you handle a task that legitimately fans out — does each branch get its own budget, or does the root task budget cover the whole tree?
Exactly. The leashing has to live outside the agent's control to actually hold. To answer your question: it's global across the tree. The root task budget covers everything, and branches draw from it. If we gave each branch its own budget, a task that fans out into hundreds of sub-tasks could silently multiply the total spend.
We do track per-branch consumption for observability, but the hard cap is always on the root.
Root-level cap is exactly right, and the fan-out multiplier is the silent killer — a task that explodes into 100 subtasks multiplies spend unless the cap sits above all of them. The trap: if your “root” is itself a sub-branch that re-roots on retry, the budget resets and the cap silently disappears. So the hard cap has to live in the orchestrator, not the agent — which is your “outside its control” point made concrete. When a single request legitimately needs more than the root allows, do you reject it, or route it to a cheaper path?
The re-rooting trap is brutal—budget resets are a classic bypass. To answer your question: we try routing to a cheaper path first (like a smaller model tier), and if it still exceeds the hard cap, we reject and require manual approval. Silently expanding the cap defeats the whole purpose.
Reject-with-approval is the part I'd underline — it makes the cap a policy boundary instead of a tuning hint, which is the only version that survives contact with a production incident.
The cheaper path has a trap though: it usually buys a lower price per call by spending more calls. Smaller tiers retry more, produce longer outputs, and sometimes need an extra verification step, so a request that was going to breach the cap can breach it faster after being downgraded. If the cap is absolute, it's worth asking whether the remaining budget can actually fund the longer path before you route to it — otherwise you've paid for a few extra steps just to reject anyway. The other half is making the rejection legible: log the remaining budget, the step index and the reason, or the human approving it gets "over cap" with nothing to act on.
When you downgrade, do you re-evaluate the cap against the expected call count, or keep the same absolute number?
Spot on about the hidden cost of cheaper paths. To answer your question: we re-evaluate the cap against the expected call count. If the projected cost of the downgraded path exceeds the remaining budget, we skip straight to rejection and log the math for the human reviewer. Otherwise, you're just paying to watch it fail slowly.
Skipping the downgrade when its projected cost exceeds the remaining budget is the right instinct, but there's a loop in it: the expected call count belongs to the downgraded path you haven't taken yet. The estimate is least reliable exactly for the paths you'd most want to reject — new tiers, unusual fan-out, low-volume traffic — so the decision to bypass can rest on the number with the widest error bars. And if the estimate errs optimistically, the request you rejected would have fit.
An alternative is to stop predicting and check the remainder at each step. Remaining budget is a known quantity, not a forecast; before each call you only need the worst case for that one step, and if the remainder can't cover it, you stop. That turns "projected cost" into an upper bound, keeps the cap absolute, and takes the human-review dependency off the hot path — rejection becomes a consequence of the counter rather than of a forecast. The tradeoff is that you can't decline up front, so part of the budget may be spent before stopping; in practice that's usually cheaper than a conservative pre-rejection.
One thing worth checking in what you log: is the math kept for the reviewer the predicted call count, or the realised one? If it's only the prediction, there's no way to learn afterward whether skipping the downgrade was right — the counter that would have settled it never ran.
Do you ever replay a rejected request against the counter to see whether the projection held?
"You can't decline up front" — that's the exact trade my version was hiding, and you're right that it's usually the cheaper one. A conservative pre-rejection based on a wide-error-bar estimate is just a guess with extra confidence.
On logging: it's the prediction, which is the flaw you're pointing at. Replay is the missing piece — without it the counter that would have settled the question never runs, so the gate can't learn. We don't replay today; that's the change I'd make first. Log the realised count alongside the projection for every request the gate lets through, and the accepted ones become the training set for whether the projections were ever trustworthy.
Thanks for this — it's the first pushback that makes me want to rewrite the decision rather than defend it.
Logging the realised count next to the projection is the right first change, and it is worth doing before touching the decision itself. There is a bias in the resulting training set, though: only requests the gate let through ever produce a realised count. The requests you declined, skipped, or sent down a different path are precisely the ones where the projection had the widest error bars — and they never enter the set. A model fitted on survivors will report that projections are trustworthy on easy traffic, which is the one case where you did not need the estimate.
The harder version is that for a request rejected before the budget was spent, no realised count exists to collect. Re-running it for real is not a calibration strategy for anything with side effects. What can be replayed is the decision: record remaining_budget, the worst case for the next step, the projection, and the gate's decision as an event, then re-evaluate that predicate offline against a corrected estimate without executing anything. A small shadow sample that actually runs to the cap supplies the ground truth the rejected branch can never give you.
That gives a concrete check rather than a general hope. Order the bypass decisions by projection minus realised on the shadow sample and look at where the error concentrates. If it clusters in new tiers, unusual fan-out or low-volume traffic, the estimate is least reliable exactly where the pre-rejection is most confident — which is the loop that made the up-front decline shaky in the first place.
The caveat is cost and safety: shadow calibration is only honest for calls that can be replayed without external effects, and for those that cannot, the event replay is all you get. Does the training set you have in mind contain any sample from a request the gate refused, or will it only ever see the requests the gate already approved?
You nailed the survivorship bias exactly. To answer your question directly: right now, it only sees the requests the gate approved. The rejected ones are completely invisible, which is exactly where I need the calibration most.
Logging the decision event is the only way to fix that without triggering side effects, and using a shadow sample for side-effect-free calls to get the actual ground truth is a brilliant way to close that loop.
That is the right place to put the shadow sample, and I would add one condition before trusting its ground truth: the expected labels for those side-effect-free calls need to be fixed before the run. Otherwise the calibration set is still drawn from the distribution the gate already approved, and the survivorship bias you described comes back one step later with better tooling.
The same paired reading applies as with the main metric — a low error rate on the shadow sample says little unless the approve rate on the positive set is reported alongside it, because declining everything is also calm.
One structural detail: the shadow call is a second execution path, not a second label on an existing one. If both write into one event stream, the calibration rows need their own writer field, or a downstream report cannot tell which rows were allowed to have side effects and which were observed without them.
If none of the shadow calls rejected by the gate carry a correct label, which measurement tells you whether the calibration set is skewed or the gate is simply too wide?
Great point about the writer field. To answer your question: you compare the gate's rejection against the realized cost on the shadow sample. If the shadow proves the rejected call would have fit inside the budget, the gate is too wide; if it confirms the projected overage, the gate was right. The calibration set is only truly skewed if it contains zero borderline cases to test against.
One thing I'd push on in the cost_aware block: hidden_cost_multiplier 1 only holds if "context travels" means the new model doesn't have to read it again. It does. Routing to the next model still bills the full 50k input tokens on that model, so you've swapped a replay on model A for a first read on model B, and you've thrown away any prompt cache you'd built on A. With caching, a retry against the same provider can actually be the cheaper path, since the replayed prefix is billed at the cache-read rate rather than full price, as long as the retry lands inside the cache window. So the real fix probably depends on why it failed. A timeout or 5xx from an overloaded endpoint is where routing helps, while a malformed output is better retried in place with the cache warm. Did your team separate those two failure types in the budget, or was it one retry counter?
This is the nuance I wish I'd put in the article. We defaulted to 'route on any failure' and only later separated the cache-warm in-place retry from the route-away retry — and they have completely different cost shapes, like you said. Routing a malformed-output case away just throws away the prompt cache you'd built and re-bills the full input on the next model.
We ended up tracking cache-hit-rate per step as its own line item. Once you split the two failure types, did it change your actual cost numbers much, or mostly just change where the spikes showed up?
I don't have a measured before/after to give you, my comment was reasoning from how the billing works rather than from a measured split. I'd expect the total to actually drop, but only by the malformed-output share, since the timeout side doesn't change. Timeouts and 5xx still pay a full first read on the next model, so that part of the bill doesn't really change. The malformed-output retries are where it shifts, because a retry that lands inside the cache window bills the replayed prefix at the cache-read rate, which is a fraction of the full input price on the providers that offer caching. The regenerated output is still full price though, so there's a ceiling on it. How much that saves overall depends on how many of your failures were malformed output in the first place. Tracking cache-hit-rate per step is a good call, it's the number that tells you whether the in-place retries are actually landing warm. What share of your retries turned out to be malformed output vs timeouts?
Your split between malformed-output retries and timeout/5xx is the useful one — they bill completely differently, and the cache-window retry only helps the first kind. For us, malformed output is roughly 30-40% of retries, with timeouts dominant. That ceiling you mention (regenerated output is always full price) is real, so the save is bounded by how many failures were malformed to begin with. Did you design the 'retry inside the cache window' explicitly — i.e. control retry timing to stay in-window — or did it just fall out of how your loop is shaped?
Neither, honestly. It's not a loop I run with that design, I was reasoning from how the billing works. If I were building it I'd do it explicitly, though: keep the retry for malformed output immediate and in the same provider, so the prefix is still warm, and not put it behind a long backoff the way you would for a 5xx. A backoff on timeouts is sensible, but it's also the thing that would push a retry outside the window if you applied it to both kinds. Since your malformed share is 30-40%, that split sounds like where the savings are.
The 'explicit beats accidental' point is the one I'd underline — a retry that lands in-window only because the loop happens to be shaped that way is fragile; designing for it is what makes the cache win repeatable. On same-provider immediate retry: do you size the window statically (a fixed timeout) or dynamically off the provider's cache TTL? We found a fixed window drifted out of sync with the real TTL as traffic patterns shifted.
Dynamically if I were building it, though anchored on the cache-hit fields in the response rather than on a TTL timer. A fixed window has a second problem beyond traffic: on Anthropic the 5-minute lifetime is measured from the start of the request that wrote or read the cache, not from when the response finished. So a call that streams for 4 minutes leaves you about 1 minute for the retry, and a fixed timeout counted from the end of the response will drift with output length. It's refreshed on every hit, so a warm prefix keeps extending. If your retries regularly land outside that, Anthropic also has a 1-hour cache option, which costs 2x base input on the write instead of 1.25x but gives a static window real headroom. OpenAI's in-memory retention is looser still, around 5 to 10 minutes of inactivity and up to an hour, and it's counted from last use rather than from the start of a request, so a static number won't line up with it either. What I'd trust is cache_read_input_tokens on Anthropic or cached_tokens on OpenAI from the failed attempt and the retry: if the retry reads zero cached tokens, it missed the window, and that's the number to tune against.
Anchoring on the response fields is the bigger shift here: it turns cache affinity from a prediction problem into an observable one. The streaming detail makes it worse in a way that's easy to miss — a long generation doesn't just consume latency, it consumes the retry's cache budget, so output length becomes a routing constraint rather than a post-hoc tuning knob.
One failure mode worth separating from a missed window: a retry often isn't byte-identical to the attempt it replaces. Add a retry counter, switch the model tier, or prepend "the previous attempt failed with X" and the prefix hash changes, so cached_tokens comes back zero even well inside the TTL. That zero has two very different causes — an expired window versus a drifted prefix — and they need different fixes. Keeping the retry prefix frozen and putting the failure context in a separate field or log line is usually the cheaper of the two.
Do you keep the retry prefix byte-identical, or does your retry path inject failure context into the prompt?
I don't run a retry loop like this in production, so I can't tell you what mine does. I was reasoning from the billing docs. If I were building it, though, I'd freeze the prefix, and your split of "expired" versus "drifted" is the part I hadn't separated.
Anthropic's docs say a cache hit needs the prompt identical up to the cache breakpoint. So a "the last attempt failed with X" line injected in front of the breakpoint breaks the match from that point on, and everything after it gets written again. The docs also say that if cache_creation_input_tokens and cache_read_input_tokens are both zero, nothing was cached at all, often because the prefix is under the minimum length. That's a third cause worth ruling out.
So I'd keep the shared prefix ahead of everything that varies, put an explicit cache breakpoint at its end, and add the failure context after it as new user content. I haven't measured what that does to retry quality, so I can't say the model behaves as well with the note that late in the prompt. Have you compared the two?
Appreciate the honesty about reasoning from the billing docs — that boundary is worth stating explicitly, and the third zero-value case is a real one: if creation and read tokens are both zero, nothing was ever cached, usually because the prefix is under the minimum length. That belongs ahead of the expired-versus-drifted split as a triage branch, since the other two both assume a cache once existed.
I don't have a clean comparison on the placement question to cite, so I'd treat it as something to measure rather than assert. Putting the failure note after an explicit breakpoint preserves the shared prefix, but it also changes what the model sees as the task: a new user turn late in the prompt can read as a follow-up instruction rather than a retry of the same request, which is a different failure than losing the cache hit. If the goal is to fix one failure without re-planning the whole task, that distinction matters more than the token saving.
The comparison I'd want is on a fixed set of failed attempts: for each layout, does the retry repeat the same failure, and does the tool-call sequence or output length change — not just whether the final answer is right.
When you put failure context after the breakpoint, is it a separate user turn, or the tail of the same message?
Neither yet, I haven't built it. For the cache it doesn't matter. The docs count the prefix in content blocks, "up to and including the block designated with cache_control", not in turns. Same message or new turn, the prefix holds.
Automatic caching copes too. The first attempt writes at the end of the shared prompt, and the retry's lookback finds that entry one block back. You just pay the write rate on the note each time. An explicit breakpoint at the end of the static prefix avoids even that.
There's one catch. The API combines two user turns in a row into one, so a separate turn only differs when the failed answer sits between them as an assistant turn. That's the layout to test against the tail, on your fixed set.
The tail probably reads more like a retry. That's still a guess.
platform.claude.com/docs/en/build-...
The blocks-not-turns distinction is the useful correction here: it moves the question out of prompt wording and into what actually gets serialised on the wire. If two consecutive user turns collapse into one, then my earlier separate-turn-versus-tail framing was measuring the wrong variable. What differs is whether the failed assistant turn is replayed between them, not where the note sits. I'd hold that fixed before running the comparison, otherwise a behaviour change gets credited to a layout the API never saw as a distinct message.
There's a second cost the cache fields won't show. Keeping the failure note behind an explicit breakpoint does protect the shared prefix, but the note is exactly the part that changes on every attempt — error text, attempt count, sometimes the previous output pasted in for context. cache_read_input_tokens can stay flat while the uncached tail grows with each retry, so a clean cache-hit metric and a growing bill are entirely compatible. That's the same split as before: the billing field tells you the prefix matched, not that the model treated the call as the same task.
Which is why I'd want the layout recorded in the trace rather than left implicit in whatever builds the prompt. If a replay can't tell which layout was used, a divergent retry isn't attributable, and "the model behaved differently" stays unfalsifiable. Honest caveat: I haven't run this comparison either, so all of the above is about what the measurement has to control, not a result.
When you log a retry, do you record the prompt layout as part of the node's input, or treat it as ambient state the replay is expected to reproduce?
I'd record it as part of the node's input. If a replay has to rebuild the layout from ambient state, a divergent retry can't be pinned on the layout, which is your unfalsifiable case.
The nearest thing I have is the request log in Elzi, our local agent app. Each model call writes the serialised request (model config plus the messages) as its first line, then the response lines with token usage. So the layout is recoverable by reading the file. It isn't a named field, nothing links a retry to its first attempt, and only the last ten requests are kept. That's fine for debugging, but for an experiment I'd store the layout as its own field.
Recovering the layout by reading the serialised request is better than leaving it implicit, but it quietly adds a dependency: the parser that infers the layout from that first line is itself versioned client code. So ambient state moves one layer down rather than going away. Naming the layout as its own field fixes "can I read it", which is the part you already flagged — but layout only matters comparatively, so the other half is whether two attempts can be paired at all.
That is why "nothing links a retry to its first attempt" is the more load-bearing gap of the three you listed. A divergent retry is not an observation about one request; it is a difference between a baseline and a later attempt. With no chain, a named layout field gives you many well-described requests and no pair to compare. The minimal change I'd make first is three keys on each event — run_id, attempt_index, parent_attempt — and only then the layout field. The chain is what makes the field evidence rather than metadata.
The ten-request window is the second truncation, and it is easy to miss because it only bites on the interesting runs. On a long chain, or when other runs interleave, the first attempt is the one evicted, so "layout is recoverable" holds only for short chains. Retaining by run rather than by recency, or pinning the first attempt's layout as a baseline that cannot be evicted, keeps the comparison intact without keeping everything forever.
To be clear about the scope: this is all about what a measurement has to control. I have not run the layout comparison either, so nothing here says which layout is cheaper — only that without a chain and a surviving baseline, a divergent retry stays unfalsifiable. On a twelve-attempt chain, could you still reconstruct the layout of attempt one from today's log, and if not, from which attempt would you trust the comparison?
No, I couldn't. The log keeps ten files and rotates them by finish order, so on twelve agent-level re-asks with nothing else running, attempts one and two are already gone, and with other requests interleaving even more are lost. A finished file also keeps no id at all, only a slot number that shifts every time another request finishes, so the files that survive can't be paired either.
HTTP retries are worse for your purpose. On the providers I checked they run inside one logged call, so only the final error lands in the request log, and each retry is just a warning in the general log with nothing tying it to the request.
So from which attempt would I trust the comparison? Honestly, none as it stands, because pairing the survivors means reading request bodies by hand. You're right that the chain comes first. I haven't built those keys yet, and a layout field without them just labels requests nobody can pair.
"You don't make retries cheaper by retrying harder" is the core insight here, and it's the opposite of what most teams do.
The default response to "the model is down" is "retry with the same model, hope it comes back". But if the model is down, retrying it doesn't help — routing to a different model does.
Curious how you handle the "no equivalent model available" case. If all the models in the registry are down, does the task fail gracefully, or does it queue and wait?
"Retrying harder" is just re-paying the 4× tax with more determination — the insight holds, and it's the opposite of instinct.
On the no-equivalent case, it depends entirely on the task's latency contract, and the trap is treating both the same:
The mistake is a global "retry N times then give up" that applies the synchronous logic to async work (wasted waits) or the async logic to sync work (timeout blowups).
Question: do you split failure modes by task class (sync vs async, read vs write), or by a single global timeout? Class-based is more code but each path actually fits its caller.
"Class-based is more code but each path actually fits its caller" is the right tradeoff, and it's the one most teams avoid because it's harder to explain in a design doc.
The split isn't just sync vs async. It's also read vs write, and inside writes, idempotent vs non-idempotent. A sync read can fail fast and serve a cached result. A sync write needs to know whether it committed. An async read can queue. An async write can queue, but only if it's idempotent — otherwise you're queuing a duplicate.
Four classes, four different retry policies. It's more code, but each one is simpler than a global policy trying to handle all four.
The read-vs-write and idempotent-vs-not split is the part most "one global retry policy" designs quietly collapse. Four small policies that each fit their caller beat one clever policy that tries to branch on all four — and the branching logic is usually where the bugs hide. The async-write-only-if-idempotent rule is the one people skip and then discover via duplicate side effects. When your async-write path hits a flaky queue, does it ever downgrade to sync, or does the queue failure just become a different retry class?
Definitely a different retry class. Downgrading to sync is a trap — it breaks the whole point of having an async boundary.
If the queue is flaky, the async-write path should have its own Circuit Breaker. After X failed publishes, the circuit trips, the async path fails fast (returning a 202/503 to the user), and the system just retries the queue publish with exponential backoff. Keeping the failure isolated to the queue is much safer than letting a sync fallback take down your primary database.
Exactly — downgrading to sync is the trap because it quietly moves the queue's failure into your primary path. The 'keep the failure isolated to the queue' instinct is the right one. We made the opposite call once and let the sync fallback touch the database; one flaky publish turned into a cascade we felt everywhere.
Did you put a separate budget on the circuit-breaker's own retries, or do they still count against the same per-task cost you're tracking?
The trap of downgrading to sync is that it feels like a resilience measure, but it's actually a blast-radius amplifier. You're not 'handling' the failure; you're just relocating it to a place where it can do more damage.
Downgrading to sync being a blast-radius amplifier is the part almost nobody catches — it feels like 'handling' the failure and it's really just relocating it somewhere with worse observability. We've watched a sync downgrade turn a recoverable async blip into a blocking one. Worst case you've seen where the sync downgrade cascaded — and did you keep any observability on it, or did the move-to-sync hide it?
The worst case I saw was a payment service that fell back to a synchronous DB write when the queue was backed up. The queue was slow, so the fallback fired, and suddenly every request was hitting the database directly. The database couldn't handle the load, so it started timing out, which made the queue even slower. The whole thing cascaded in about ninety seconds. We only caught it because we had a dashboard showing DB connection count, and it spiked. The sync path didn't have its own metric — it was invisible until the DB started screaming.
That's the sync-downgrade blast-radius amplifier in the wild — the queue slows, the fallback fires, and suddenly the database is taking the traffic the queue existed to protect it from. The scary part is it feels like 'handling' the failure. When you found it, did you have any observability left on the sync path, or did the move-to-sync hide it until something actually fell over?
Exactly. That "silent failover" is the worst kind of failure mode because it looks like resilience on the dashboards. To answer your question: we had almost no visibility. The sync fallback didn't have its own metrics, so we only found out when the database latency spiked and took down the whole service. Lesson learned the hard way.
The dashboard staying green while you burn the degraded path is the whole danger of silent failover — and the DB spike taking down everything is the classic “the failover became the incident” story. The fix isn't only metrics on the fallback; it's making the fallback's cost part of the same budget and observability as the primary, so a silent switch shows up at the routing layer in real time, not in a post-mortem. A labeled expensive secondary is fine; an unlabeled one isn't. Did you end up giving the fallback its own budget line, or folding it into the root cap?
Spot on. "The fallback became the incident" is exactly what happened. To answer your question: we folded it into the root cap, but slapped a strict rate limit and specific tagging on the fallback path. It shares the overall budget, but if it eats more than 20% of the total allowance, it triggers a page. An unlabeled expensive secondary is a nightmare, but a throttled one is manageable.
Folding it into the root cap plus a 20% tripwire is a cleaner answer than a separate fallback budget, because a separate line is exactly how a degraded path disappears from the dashboard again.
One thing I'd watch: a percentage threshold changes meaning with traffic. At low volume, 20% can be three calls and a page at 3am; at high volume, 20% is a large absolute spend before anyone hears about it. Pairing it with an absolute floor, or computing it over a sliding window rather than the current task, tends to cut the noise without hiding the real breach. The rate limit has a quieter interaction too — if fallback calls queue, the task may time out rather than fail, and a timeout re-enters the retry path, which goes straight back to the throttled fallback. Sharing one counter between the rate limiter and the retry budget closes that loop.
Is the 20% measured against the full root allowance, or against what's been spent so far?
Great catch on the timeout loop! That's a nasty piece of recursion for the budget. To answer your question: it's measured against the full root allowance, but we added an absolute floor (e.g., 50 calls) to kill that low-volume noise you mentioned.
As for the loop, if a fallback times out, it burns the counter but explicitly bypasses the retry path—it fails fast to avoid exactly the spiral you described. Sharing the counter is definitely the way to go.
One correction and one number.
The correction: "context paid once" doesn't hold, and stefan's question above is the right one. Model B has no memory of model A. It receives the context and pays for it. What routing changes is the rate on that send, cache read or full ingestion, never the number of sends. Every call re-sends the transcript, so the unit that matters is turns, not attempts.
The number: on 4,451 token-bearing turns across 53 sessions, the median turn pushed 135,267 tokens. Across the corpus 92.9% of the tokens were cache reads, 6.7% fresh input, and the model generated 0.41% of what it read. Priced at 0.1 for a cache read against 1 for fresh input, losing the cache on a turn multiplies that turn's cost by a median of 8.1x (IQR 6.5 to 9.0).
The part that changes the advice is where that cache is lost. Appending the error text to the end of the prompt is cheap, because the prefix still matches and only the new suffix gets billed fresh. What's expensive is changing anything before the end. Editing the system prompt, re-summarizing history, or moving to a model with its own cache namespace reprices everything from the first changed token. We measured the same shape in context compaction: after a rebuild, 69.4% of first turns were more than half fresh input.
So a same-model retry against a warm cache costs roughly what the failed attempt cost, not four times it. The tax arrives when the fix rewrites the front of the prompt.
The instrument I'd add to the billing join above is cache reads divided by total input, per turn. A retry that appends barely moves it. A rewrite drops it, and the drop is the bill.
The correction lands harder than mine did — and the number reframes the whole thing. "Turns, not attempts" is the right unit; the retry that re-sends the transcript isn't free even at a better rate, because the cost that moves is the cache, not the count. Your 92.9% cache-read corpus is the part I under-weighted: a slim handoff helps for exactly the reason your data shows — it keeps the prefix stable, and a stable prefix is what keeps the cache warm. Re-attaching the full original on retry is what blows it, and at your median that's an 8.1× turn. So "slim handoff" and "where the cache is lost" are the same lever seen from two sides; I'd just add that the retry budget should be counted in cache-lost turns, not attempt counts, because the bill isn't the retry — it's the re-warm. The part I'm still unsure on: when you hand off to model B that needs different framing, how do you keep the prefix from breaking — carry a frozen prefix, or accept the cache loss as the price of the switch?
You're right that the bill is the re-warm, and to your question I'd answer the boring way: we accept the loss.
A frozen prefix doesn't survive a switch. The receiving model has never seen that text, so whatever the sender cached can't be spent on it, and the first turn pays full input on however much you carry. What stays warm is the second turn onward, and only if nothing before the end of the handoff changes.
The number that made me stop treating a rebuild as free came out of compaction, which is the same event from a different direction. Across 62 rebuilds in two long sessions, 69.4% had a first turn that was more than half fresh input. Those rebuilds did pay themselves back in about 14 turns, but only because the rebuild also halves what gets re-sent afterwards. A model switch has no shrink, so it's the cost without the refund. It has to earn back on rate alone.
So the rule I'd give you: switch late, and switch on a turn where the prefix was going to break anyway. If the retry was already re-summarizing the history, the switch is close to free because you were paying for the rebuild regardless.
The "switch on a turn where the prefix was going to break anyway" rule is the cleanest framing I've seen — the rebuild cost is already in the budget, so the model switch just rides it instead of adding a new one. And the 69.4% number is the gut-check that keeps "rebuild is free" from becoming a habit. One thing I'd push on: across those 62 rebuilds, did any carry a partial prefix — only the stable middle — into the new model, or is it always all-or-nothing re-warm? A partial carry could save part of the hit without the full refund — have you ever measured whether the stable middle of the prefix survives the switch and actually reduces the re-warm?
Good push, and the honest answer splits in two.
The switch case I can't measure. My database holds exactly one model-switched message in its entire history, so the scenario you are asking about, where a rebuild turn is also the moment the model changes, has n=1 here. Anything I said about it would be a story about one turn.
The carry case I can measure, and it isn't all-or-nothing. Across 62 compactions with a readable turn on each side, I compared the cache-read tokens on the first turn after the rebuild against that turn's total context. Median carry is 33.5%, range 0 to 96.8%. 21 of 58 events came back over 50% cached, 12 came back under 5%, and 3 came back at exactly zero. So a partial prefix does survive a rebuild, and the share is not stable at all.
What varies most is the session, not the event. Two sessions with 26 and 10 rebuilds each have median carries of 47.7% and 67.1%. A third with 14 rebuilds sits at 11.1%. One average over all of them would've hidden exactly the thing you are asking about.
Here is the limit, though. A cache-read count tells you how many tokens the provider served from cache. It doesn't tell you which part of the prompt they came from. So "the stable middle survives" is not something my record can show. I can show how much survived on each rebuild. I can't show that it was the middle, and I would be guessing if I said the summary stayed cold and the tail stayed warm. To answer that you need a provider that reports cache hits per block, or a controlled test where you change one region at a time and watch the number.
This is the right instinct — measuring per rebuild instead of averaging is exactly the discipline the article argues for, and the 'session varies more than event' finding is the counterintuitive one worth sitting with. Your 3 near-zero / low sessions make me wonder if those are just high-rebuild-frequency sessions where the prefix is always shifting. Did you ever try pinning a sticky prefix (freeze the stable middle across rebuilds) to see if the carry rate stabilizes?
Short answer: no, and the table is small enough to print.
Turns between rebuilds against median carry, per session: 422 turns, 2.0% carry. 51 turns, 3.8%. 80 turns, 24.2%. 49 turns, 65.6%. 38 turns, 14.9%. 25 turns, 31.7%.
The two sessions that rebuild most often sit in the middle. The session that rebuilds least often by a wide margin carries almost nothing. Frequency doesn't predict it.
What does line up is the model serving the first turn after the rebuild. 123 of the 142 first turns ran on one model and carry 19.5% at the median. 14 ran on another and carry 64.8%. The other five are too few to say anything about. That's an association, not a cause, and cache_read gives a quantity rather than a location, so I can't tell a preserved prefix from a provider that caches hard on its own.
One correction I owe you. Yesterday I told you 62 events, a 33.5% median and three per-session medians. That came from a slice pinned on 10-24 and a script I didn't save, so I can't rebuild it. Same question, one definition, every compaction in the table at a cutoff of 05:10Z today: 143 events, 142 with a readable turn after them, median 21.2%, range 0 to 95.5, 45 over half, 34 under 5%, 3 at exactly zero. Same shape, different number. The definition now lives in a script that exits non-zero if a figure moves.
On pinning the stable prefix: I can't test it here. The client assembles the prompt, so the only thing I can observe is what it sends, and pinning happens upstream of that. What the data does say is that pinning would matter most where the first post-rebuild turn is mostly fresh, and those turns sit on the model that caches least. That's a hypothesis with no lever attached on my side.
That table is the 'session varies more than event' finding made concrete — 2% and 65% on different sessions with similar turn counts means rebuild frequency alone doesn't explain carry. The 65.6% session is the one I'd dig into: was it longer, or did it have a more stable prefix that survived the rebuilds? We saw the same pattern — two sessions with similar rebuild counts but wildly different carry, and the difference was always prefix stability, not the count.
Not longer, and that's what makes it interesting. That session rebuilt 10 times with a median gap of 49 turns between rebuilds. The two deepseek sessions nearest it ran 55 and 60 events at gaps of 38 and 25, so the 65.6% one rebuilt less often and still carried three to four times more context forward.
The column that separates them is the model. Grouped by whatever served the first turn after each rebuild: mimo-v2.6-flash-free reads 64.8% median over 14 pairs, deepseek-v4.1-flash-free 19.5% over 123. The 10-event session is mimo; the 55 and 60 event ones are deepseek. Within this log, rebuild frequency and session length both fail to explain the split, and the serving model explains more of it than either.
I can't call that prefix stability, though, and I don't want to borrow your conclusion because it fits. cache_read gives a quantity, not a location: we see how many tokens were billed at the cached rate, never which ones. A provider that caches harder on its side looks identical from here to a client re-sending a stable prefix. Same number, different mechanism.
Separating those needs one session, one model, two rebuilds with the prefix pinned. I can't run it, because the client assembles the prompt and the log only shows the bill.
One caveat: these come from a receipt pinned on 10-30, not a live query. The v1 corpus has since been lost in a wipe, so that receipt is the primary source now, not a convenience.
You're right to refuse the word stability, and I should withdraw it from my earlier reply. What the table supports is a difference in carry associated with the serving model; it doesn't identify a mechanism. "We saw the same pattern, and the difference was always prefix stability" was a claim from our side, not something your receipt can establish — borrowing it here would be exactly the error you're guarding against.
Your quantity-not-location point goes further than I'd taken it. Two mechanisms produce the same billed number: a provider caching more aggressively on its side, and a client re-sending a shorter or more stable prefix. But there's a third ambiguity even if you could locate the cached tokens — a byte-identical prefix can play a different role after a rebuild. Same bytes, different position in the task; the cache counter doesn't know the difference. So the honest status of carry rate is an observation awaiting explanation, not evidence for either mechanism.
The controlled run you describe — one session, one model, two rebuilds, prefix pinned — needs something the bill can't give you: the input token sequence on both sides of each rebuild. That has to be captured where the prompt is assembled. If the pinned receipt still holds per-pair total input tokens, there's a cheaper first cut: provider-side caching keeps the total roughly fixed while the cached fraction moves, whereas a client that re-sends less changes the total itself. It won't separate every case, but it separates the two you named without re-running anything. And with the v1 corpus gone, the receipt being the primary source is a constraint on what can be concluded, not a footnote.
Does the receipt carry per-rebuild input totals, or only the cached / uncached split?
All four token fields, per row: input, output, reasoning and cache.read. So the prompt total is
input + cache.readon any pair you pick, and your cut needed no re-run. I've just taken it.On 102 rebuild pairs from the only surviving corpus, a backup taken before the wipe with a cutoff of 25 October:
The bulk behaves the way you predicted for provider-side caching. The total sits near 96k however the split lands, and inside a session the correlation is indistinguishable from zero, so the high-carry pairs are not sending less. That kills "the prefix got smaller" as a general account.
The top bucket is the exception and I won't round it away. At 75% carry and above the prompt is about a fifth smaller than at the bottom. That's six pairs, and the buckets in between aren't ordered at all, with 25-50% the largest at 118,972, so I read it as noise rather than a modest trend. If it holds it's the one place where the client might be sending less, and six pairs can't tell you.
Which is where your third ambiguity lands, and I think it's stronger than either of the first two. If the same bytes can play a different role after a rebuild, then carry stops being a usable variable rather than an unexplained one, because two pairs with the same share can mean different things and nothing in the record separates them. That's a different status from "mechanism unknown", and I'd state it that way.
Two corrections to what I gave you earlier. The 143 events, 142 pairs, 21.2% reading came off the live database and can't be reproduced, because it was wiped on 2 November. The durable backup reads 103 events, 102 pairs, median 25.5%, range 0 to 95.5, with 33 pairs over half, 27 under 5% and 3 exactly zero. The first-turn-with-a-cache-hit comparison, which I print alongside and don't pin, is 28.7%. Use 102 and 25.5%, and treat 143 and 21.2% as withdrawn.
On the receipt being the primary source, I made the tool enforce that instead of promising it.
--checknow tests the corpus fingerprint before the pins and exits 5, "CORPUS CHANGED", rather than 1, "DRIFT", when it's a different corpus. A pin that moves looks identical whether the definition moved or the data did, and that ambiguity is what produced the wrong number in the first place.One correction back at you: the v1 corpus isn't gone. The live tables are empty, but the same backup holds all 19,518 v1 rows. It's the live copy that went.
I accept the withdrawal and will use 102 pairs and 25.5% as the durable reading; the earlier figures were from a database that no longer exists, so there is nothing to recompute against. Putting the corpus fingerprint ahead of the pins is the right fix, and exit 5 is a better failure than a plausible-looking drift number.
The distinction you drew between "definition moved" and "data moved" suggests a third case your record does not yet carry. If the same bytes can play a different role after a rebuild, then cache.read is not one variable — it is reused prefix on one pair and freshly sent input on another, with the same number attached. Adding a role field at capture time makes those two rows incomparable in a useful way; deriving the role afterwards from the share is the ambiguity all over again.
The withdrawal itself is worth an event. A correction in prose does not travel with the corpus, so the old number can reappear in any downstream report that still has the earlier text. A tombstone keyed to the corpus fingerprint is replayable; a sentence is not.
I would keep your reading of the top bucket as noise rather than a trend — six pairs cannot carry it — and I would keep the near-zero within-session correlation scoped to the surviving backup and its cutoff. If two pairs show the same cache.read bytes but one is reused and one is freshly sent, which field in the current receipt separates them?
The observation that dashboards show cost per successful request is the uncomfortable one. Failed attempts disappear into the average, so the feature looks cheap right up until the invoice arrives.
SkepAgent's point deserves as much attention as the token math. A retry after a tool call that timed out but actually succeeded does not only cost tokens. It can apply the same change twice. So the order matters: make side effects idempotent first, then decide how many retries you can afford.
challan116's suggestion to budget retries as their own line item matches what we do on client builds. When a feature is scoped, it gets an expected cost per run and a separate allowance for failures, written down next to the acceptance criteria. If the failure line starts growing, that is a signal about the prompt or the model choice, and it shows up within days.
When you route to the next model and carry context forward, how do you stop the second model from repeating a side effect the first one already triggered?
The ordering point is the crux — idempotency has to be a precondition, not a retry setting. You can't retry your way out of a non-idempotent side effect; the retry just multiplies the damage.
On the cross-model case: the same "retry controller owns the tool result" boundary from earlier helps here too. If model B's attempt is a transport failure (not a decision failure), the cached result from model A's already-successful call short-circuits — so B never re-fires the side effect, it just receives the result as if it came back clean. The nasty case is partial: A triggered the side effect but the result was lost, and B is a different model that can't see A's cache. There you need a side-effect ledger keyed by task id, not by model — so any model in the chain can check "did this task's effect already land?" before acting.
Question: do you run that side-effect ledger as a separate service, or bake it into the orchestrator? Separate is cleaner to reason about but adds a hop on every call.
I'd lean toward keeping the ledger logically separate from the orchestrator, but not necessarily deploying it as a separate service from day one.
For state-changing financial operations, I'd initially persist the operation record, provider reference, and execution status alongside the transaction state in a durable database. The orchestrator can consult that record before deciding whether an operation is safe to execute again.
The important distinction for me is between a transport failure and an ambiguous operation outcome. If a provider may have completed the side effect but the response was lost, I'd reconcile the original operation before considering another execution.
I'm exploring this in Agentic Switcher, where the focus is deterministic orchestration, ownership fencing, and safe recovery across provider integrations. A separate ledger service could make sense later, but I'd first establish the consistency and recovery semantics.
One question I'd be interested in exploring is how you'd handle a side effect that succeeded externally but whose ledger update failed. That seems to be the hardest boundary to get right.
This matches what I see — logical separation, physical colocation early. The trap is making the ledger a synchronous dependency of the retry path; then a ledger hiccup stalls exactly the operation it's supposed to protect. Keeping it as a side-write (operation record + provider ref + status) that the retry reads but never blocks on is the version that actually ships. The real job of the ledger isn't audit, it's decision — "have I already committed this signature?" has to be answerable in microseconds or the retry just fires anyway. How are you gating the retry on the ledger read — optimistic fire-then-check, or a hard gate?
I'd favour a durable gate before the external side effect, not optimistic fire-then-check.The important distinction is that the gate shouldn't become a remote synchronous dependency. I'd keep the ledger logically separate but co-located initially, with an atomic state transition or idempotency constraint protecting the operation before the provider call.If the provider outcome is uncertain, the orchestrator should reconcile using the provider reference or idempotency key rather than blindly retrying. A ledger outage should fail closed for potentially duplicate financial operations, while allowing safe reads and reconciliation to continue where possible.
The goal is to make retry safety a property of the state machine and its invariants, not just a fast ledger lookup.
How do you handle the ambiguous-outcome case when the provider has accepted a request but the orchestrator times out before recording the result?
The fail-closed-on-ledger-outage distinction is the right one — safe reads keep flowing, duplicate financial writes stop cold. On the ambiguous case you raised: we treat "provider accepted, orchestrator timed out before recording" as a third state, not success and not failure, and we reconcile by idempotency key before any reissue. That third state is where most "duplicate charge" bugs actually live. Do you keep that pending state inside the orchestrator, or do you push it down to the ledger as a row the reconciler owns?
I'd push the durable pending state into the ledger and make the reconciler responsible for resolving it. The orchestrator owns execution; the ledger owns the authoritative operation state; the reconciler owns closing the gap between intent and confirmed outcome.Before the provider call, we persist the operation identity and establish an atomic execution claim. If the call times out, we transition to an outcome-unknown state rather than treating it as either success or failure. The reconciler then checks the provider using the idempotency key or provider reference before any reissue.The orchestrator may hold transient execution context, but recovery must not depend on that process surviving. One caveat: where the provider supports neither idempotency nor reliable status lookup, we cannot guarantee safe automatic retries; that case needs an explicit uncertainty and escalation path.For me, that separation is what makes reconciliation a recovery mechanism rather than just another retry loop.
The outcome-unknown transition is the key move and most teams miss it — they jump straight to success/failure and then can't tell a timeout from a dropped call. Persisting the operation identity before the provider call is what makes the reconciler safe to run at all.
The one place it got hairy for us was idempotency-key reuse across a flaky gateway — same key, different semantically-meaningful attempt. For providers with no status lookup, do you escalate every time, or keep a manual 'confirmed-lost' bucket someone drains on a weekly pass?
One distinction I think matters here is between a failed response and a failed operation. A timeout doesn't prove the side effect didn't happen.
For state-changing operations, I'd separate retrying the transport from retrying the operation itself. If the outcome is ambiguous, preserve the original operation reference and reconcile its status before deciding whether another execution is safe. Idempotency helps, but it doesn't automatically resolve partial completion or providers that don't support idempotency keys.
Good distinction, and it's the one our R4 sketch glossed: retrying the transport (the call didn't arrive) is safe to repeat; retrying the operation (did the side effect land?) needs the reconcile-first step you describe. The trap is treating a timeout as a transport failure when it might be an operation that already committed — re-firing then double-applies. The piece I'd add: the ledger you mention has to be readable by any model in the chain, not just the one that started it, or the handoff to model B reintroduces the same ambiguity you just closed. Question: when the status is still ambiguous after reconcile, do you fail the task or escalate to a human/queue?
Routing-the-task-not-the-model is the right instinct, and the token math is brutal once you see it. One thing I'd put before the retry-budget discussion: idempotency.
A timeout doesn't tell you the call failed — it tells you that you didn't hear back. If the side effect already ran (the payment went, the email sent, the row was written), "carry context forward and advance to the next model" can apply it twice. The cheapest retry policy in the world still double-charges someone if the thing it retried had a side effect.
So I'd gate it: reads retry freely; writes retry only behind an idempotency key, or re-grade the outcome (did the action actually happen?) before re-running. Otherwise the retry tax you measure in tokens comes back later as a reconciliation bill — same lesson as your dashboard point: the real cost is on the path you're not looking at.
Glad this one traveled over from your local-LLM piece — and yes, idempotency first is the right ordering. The retry-budget discussion is moot if a retry can double-apply a side effect; you've capped the cost of the call but doubled the damage. The split I landed on: the retry controller owns the tool result keyed by signature, so a silent-success (heard nothing, but it ran) returns the cached outcome instead of re-firing. That's what makes a cost cap safe to actually enforce. You put it better than I did: a timeout is a communication failure, not an operation failure. How are you surfacing that distinction to the caller — same error type, or a distinct "unknown" status?
The averaged-dashboard point is the one that gets people. A retry that eventually works just looks like a slow success, so the extra cost never shows up anywhere you'd think to look. I've started treating spend per task, retries included, as the number that matters instead of cost per call. Curious if you see the same thing with agents that never error at all. They just keep calling the model and going nowhere. No failure, no retry counter, still a bill.
The no-error loop is the scarier one, and our dashboard hid it the same way — latency looked fine, nothing went red, so it read as healthy. We only caught it when we ranked by tokens-per-resolved-task and a 'simple' agent was quietly 4x a 'hard' one that actually failed fast.
For the looping-but-never-failing agents, did you end up bounding them by a max-steps budget, or by watching tokens-per-task drift over time?
Both, but the hard stop does the real work. Watching drift tells you what happened after the money's already gone. I'd rather have a per-task ceiling that lives outside the agent, so a restart or a clever retry can't reset it. Drift is how I find out the ceiling was set wrong.
Hard stop plus an external ceiling is the combo that actually holds — and keeping the ceiling outside the agent is the only way to stop a clever retry from resetting it, same as you said. We put ours outside too for exactly that reason. Is your per-task ceiling a budget (tokens or spend) or a time bound? They behave very differently when a failover is what's eating the budget.
Spend, not a clock. A failover can replay a large context and burn through the budget while finishing comfortably inside a timeout. In Warden, limits are API spend per agent over a window like 24 hours, not per task. There's also a loop stop for an agent repeating the same tool calls with the same results. Loops that vary are what the spend limit is for.
Spend rather than clock is the right unit — a failover replaying a large context can exhaust the budget while looking perfectly healthy on latency. The timeout and the budget are measuring different failure modes and neither substitutes for the other.
The part I'd watch is the grain of the window. A per-agent 24h ceiling turns one runaway task into a degraded day for every later task on that agent, and the record then says "window exceeded" without answering which task consumed it. A window limit works well as a circuit breaker but poorly as an attribution unit; keeping per-task accounting alongside it is what lets you explain the breach afterwards instead of just stopping it.
The loop stop has a similar blind spot. Matching identical tool calls with identical results catches the literal loop, but a retry that varies an attempt counter, a timestamp or an equivalent query produces different results with the same semantics. "The acceptance criterion hasn't advanced across N calls" tends to be a more robust signal than comparing outputs byte-for-byte.
When the window trips, do you reject new tasks or downgrade them — and does the rejection carry an attribution back to the task that filled it?
This is exactly the risk I designed around in a Node agent I run. Two things helped: a per-tool timeout (45s), so a hung data source fails once instead of stalling the whole turn, and falling back to a different model provider instead of retrying the same one. Retrying an identical long-context request against a provider that's already slow mostly buys you the same failure, at full context price.
The "different provider, not the same one" point is the one people skip and then pay for twice — a retry against a stalled provider is mostly just buying the same failure at full context price. The per-tool timeout is what makes the fallback actually fire: a hang becomes one clean failure instead of a slow bleed across the whole turn. When you switch providers mid-task, do you carry the prompt cache or do you accept the full re-warm on the new one? That's the part that decides whether the fallback saves money or just moves the cost to a different line item — do you measure the re-warm explicitly, or only see it in the per-provider bill?
We accept the full re-warm. The cache stays with the provider, so it's lost when we switch. At our volume that cost only shows up in each provider's bill, and we don't track it separately. The fallback is for reliability, not saving money.
Honest answer — if the fallback exists for reliability and not savings, eating the re-warm is just the price of staying up, and that's a fair trade. The number I'd still want is how much that re-warm costs as a share of the whole trajectory, because at high retry counts it can quietly dominate. Do you ever measure the failover re-warm, or is it explicitly out of scope once reliability is the goal?
the 4x holds for a timeout, but where the call dies changes the bill a lot. a provider that errors before the first token usually costs you nothing, one that dies mid stream already charged you for the full prefill. we log failure stage next to attempt count, otherwise the dashboard averages the two together.
one gap in the failover version: switching models drops the prompt cache, so the next model pays full price on that 50k context anyway. we retry the same provider first while the cache is warm and only fail over on hard errors.
did you compare cache hit rate on retries versus failovers?
Logging failure stage next to attempt count is the fix for the averaged-dashboard problem — a pre-token error is free and a mid-stream death is full prefill, and a mean hides that gap completely. Retry-the-same-provider-while-warm before failing over is exactly the cache-aware order most people get backwards. When you do fall over, do you re-order the prefix so the new model sees the cacheable part first, or do you just eat the full re-warm on that 50k context? That choice is most of the remaining bill on a 50k context — do you re-order the prefix on failover, or just accept the full re-warm?
we reorder on failover. the cacheable prefix (system prompt plus tool defs) goes first and the volatile turns last, so the new provider at least writes a cache entry that pays off on the next retry. the first call still eats the full cold read though, so we only fail over once the retry budget on the warm provider is gone.
are cache write surcharges eating the savings for you on shorter contexts?
Prefix-first reordering is the right reflex — put the cacheable part (system + tool defs) up front so even a failover writes a cache entry the next retry can use. On your surcharge question: we saw the cache-write surcharge eat most of the win on short contexts (under ~4K), so for short tasks we now prefer 'retry in place on the same provider' over cross-provider failover that rebuilds the cache cold. Does your failover trigger on a budget threshold, or on error type — do you treat a 5xx differently from a timeout?
I'm not following the "context paid once" part.
model.call(task.handoff_context)still sends input to the next model. Is your gateway absorbing that cost, or is the handoff context smaller? Switching providers alone doesn't explain the drop from 4x to 1x.Fair challenge — "paid once" was sloppy framing on my part. Model B absolutely pays for what it receives; routing doesn't make tokens free. What it actually changes is three things: (1) the rate — cheap steps hit cheap models; (2) the handoff carries the slimmed context (results + deltas), not the full original replay; (3) a retry that doesn't re-attach the full original. The 4×→1× lives mostly in (2) and (3), not in "paying once." You're right that switching providers alone barely moves the number — the win needs the smaller handoff and a retry that doesn't re-bloat. How are you scoping what travels in the handoff today — full context, or a trimmed carry?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.