A few weeks back I watched an agentic pipeline burn a month of budget in three days. The instinct everyone reached for was the obvious one: "the frontier model is too expensive, switch to a smaller one." That's the wrong diagnosis, and it's worth writing down why.
When we finally tagged every token to the (step, tool, model) tuple that spent it, the picture was uncomfortable. The expensive model wasn't the problem. The problem was that every step — the trivial classifier, the summary, the hard reasoning call — silently defaulted to it, and nobody could see which step was responsible. We'd been arguing about which model to buy when we should have been arguing about which step was allowed to call it.
Cost in an agent isn't a number. It's a tree.
In a single LLM call, cost is one line item. In an agentic loop it's a tree: each step calls a model, maybe a tool, maybe a retry, maybe a sub-step that calls again. The bill at the end is the sum over that whole tree.
If you only ever see the total, you are optimizing blind. "Make it cheaper" with only a total in front of you means guessing. And the guess is almost always "use a smaller model everywhere," which throws away quality on the steps that actually needed the frontier model while leaving the real waste untouched.
The silent default that inflates everything
Most agent frameworks let you set "the model" once, at the top. So a routing decision that should have been per-step gets frozen into a single global choice. A classification step that a small model handles fine? Frontier. A summarization of text the model just produced? Frontier. The one genuinely hard reasoning call? Also frontier — but now it's lost in the noise.
The waste here isn't that frontier models are pricey. It's that ~80% of those calls never needed them, and without per-step attribution you can't even prove which 80%. You're paying a premium on steps whose output quality wouldn't have moved.
Per-step attribution is boring infrastructure, not a feature
There's no clever trick. You log, for every call: step id, tool, model, input/output tokens, latency, retry count. That's it. Once that data exists, the waste shows up in minutes — usually a handful of steps dominate the bill, and they're rarely the ones you'd have guessed.
I've lost count of how many times the "expensive" step turned out to be a retry loop quietly re-running a hard call five times because the tool returned a shape the parser rejected, not because the model was wrong. That's not a model-cost problem. That's an un-instrumented step.
Failover is where the blind spots explode
Here's the part that actually blows budgets. A model errors, or times out, or returns garbage. The naive fix re-runs the whole trajectory — or at best the whole step — against the same model. Cost spikes exactly when you can least afford it, on the run that was already failing.
A planned failover routes that single step to a fallback at the step boundary: same tool contract, different model, the rest of the run continues. The gap between "retry the universe" and "swap the model for this one call" is most of the cost story nobody talks about, because you only see it when things break — and by then you're not measuring.
The lever lives at the routing layer, not in prompts
The point that took me too long to internalize: you cannot prompt your way to per-step cost governance. The layer that decides which model serves each step is the only place that can both measure the spend and act on it. Put the attribution and the failover decision anywhere else and you're asking a component that doesn't see the calls to fix calls it can't see.
This is also where the compliance constraint and the cost lever happen to land on the same component. In Southeast Asia, PDPA-aligned handling means prompts and outputs can't wander out of region — so the routing layer has to be SG-hosted (Tencent Cloud) and keep data resident. That same in-region gateway is exactly where per-step attribution naturally lives: the router is already on the path of every token, so measuring them costs nothing extra. A gateway that exposes 25+ models behind one OpenAI-compatible endpoint isn't just a convenience for failover — it's the only place the measurement and the swap can share state.
The question I keep coming back to
When your agent's bill doubles, can you point to the three steps that caused it — or do you just know the total went up? If you can't attribute per step, the fix was never a cheaper model. It was instrumenting the routing you already had.
Top comments (13)
"Cost in an agent isn't a number, it's a tree" is exactly right, and the
(step, tool, model)tuple is the unit that finally makes the bill legible. We went through the same arc — the reflex is "switch to a cheaper model," and it's almost always the wrong lever, because the waste isn't the price per token, it's the count of unnecessary frontier calls no one attributed.Two things I'd add from doing this in anger. First, retries are the sneakiest line item — a step that silently retries twice triples its cost and doesn't show up unless you log attempt number, and agent frameworks love to hide retries inside the tool layer. Second, once you have per-step attribution, the interesting metric isn't total cost, it's cost-per-successful-outcome, because a "cheap" step that fails and forces a re-plan upstream is more expensive than the expensive step it replaced. The logging you describe is genuinely the whole trick, but the payoff is being able to ask "which step has the worst cost-per-success," not just "which step is priciest." Did tagging change which steps you actually cut, versus what you'd have guessed?
The retries-inside-the-tool-layer point is the one most teams miss, because their observability stops at the tool boundary — the retry never shows up as a distinct call. And cost-per-successful-outcome is the right reframe: the router's job becomes "cheapest path that still succeeds," not just "cheapest path." On your question — tagging changed what we cut, because it exposed steps that looked cheap per-token but were net-negative once re-plan cost counted. Did you keep any step that got cheaper on paper but failed the success bar?
The (step, tool, model) tuple is doing a second job you have not cashed in yet, and it is the one I would care about more than the bill.
Everything here treats a retry as a spend line. That holds as long as the step is a read. The moment a step has an effect — writes a row, sends a message, moves money — the retry loop quietly re-running a hard call five times because the parser rejected a shape is not expensive. It is five effects. The cost dashboard is the cheapest detector anyone has for that, and nobody reads it as one, because the number it prints is dollars and the thing it found is a duplicate.
Payments has one rule here and it is older than any of this: the unit you retry is the unit you make idempotent. Same unit, same key. Your step boundary is already the right boundary for the failover decision, so it is already the right boundary for the idempotency key — you just have to admit the step has a class as well as a cost. Effectful steps retry against a key. Pure steps retry freely. Right now both are "a step" in the tuple.
Failover sharpens it. Routing a single step to a fallback at the step boundary is right, and it means the fallback runs against whatever the first model already did. If the first call timed out after the tool committed, the fallback is a second commit with a different model's name in the log. Not a more expensive run — a wrong one. Per-step attribution is what makes it visible at all: same step id, two models, one of which you would never have guessed was in the trajectory.
Genuine question since you have the data: when you tagged every token, did a retry show up under the same step id or as a new one? Whether the schema collapses retries into the step is the difference between "this step cost 5x" and "this step ran 5 times", and only one of those sends you looking for duplicates.
This is the sharpest extension and the one I'd want to build on. The (step, tool, model) tuple quietly assumes read-only steps — the moment a step mutates state, cost attribution and retry semantics split apart. A retry of a write isn't a duplicated cost, it's a duplicated action. The routing layer should know whether a step is idempotent before allowing a blind retry; otherwise "cheaper path" can mean "double side effect." Have you found a clean way to tag effect-vs-read at the tuple level?
“Swap the expensive model” is usually the wrong first diagnosis.
Once cost is attributed to the (step, tool, model) tuple, the bill often lives in repeated cheap steps and tool chatter that never had a success gate — not in the one frontier call everyone blames. Cost in an agent is a tree; a single sticker price is a guess.
The check I’d put next to task success: tokens (or $) per complete success, broken by step. If a cheaper route wins on spend but loses on completion, that is not a savings — it is a different, worse contract.
Exactly — the frontier call is the visible scapegoat; the real leak is the long tail of cheap steps, each firing with no success gate. Model-swap treats the symptom. The fix is attaching a success gate to the cheap steps so the router can collapse the chatter instead of echoing it. That only works if the gate is the tool's own signal, not just downstream task outcome. Do you gate tool-call retries on the tool's success signal, or on the final task result?
The “cost is a tree” framing makes a lot of sense.
I think there’s a similar observability problem with agent context too. We usually know what context was available, but not always which pieces actually influenced a decision or which assumptions became stale along the way.
Per-step attribution for cost feels useful; I wonder if long-running agents eventually need something similar for context.
The context problem is the same tuple wearing a different mask. We log what context was retrieved, but rarely what was attended — so "available context" and "used context" diverge silently. The defensible move is to attribute context cost to the step that requested it and measure outcome when it's pruned, the same way we gate a cheap tool call. Do you trace attention back to the retrieval step, or treat the prompt as one undifferentiated blob?
Per step attribution is the difference between a cost story and a guess. I would add one guardrail here, record quality outcomes beside tokens so a cheaper route is judged by task success rather than spend alone. That makes failover policy measurable and keeps the router from quietly optimizing for low cost at the expense of the hard step.
The guardrail is what turns failover from a reflex into a policy — without outcome data the router silently optimizes the easy 80% and starves the hard 20%. Recording quality beside tokens is the missing column most cost dashboards don't have. Do you gate failover on a per-step success threshold, or a rolling task-level one? Per-step catches a bad route faster, but task-level is less twitchy on noisy single steps.
The retry point is very real. I’ve seen a cheap step become the expensive one just by retrying downstream work. Tracking attempts separately from the step cost makes that obvious fast.
Yes — the "attempts" dimension is what flattens the illusion that a step is cheap. Plot cost per (step, attempt) and the 3rd retry of a $0.001 step that triggers a $2 downstream re-run stops hiding. The trap is letting attempts ride on the caller's retry loop instead of the router's budget. Do you cap attempts per step, or let the router decide when an attempt budget is exhausted and route the next one differently?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.