Last quarter we finally stopped trusting the LLM bill. We wired real per-step token attribution into 11 production agents — not the default "total tokens" the provider hands you, but a tracer that tags every call, every tool, every cache lookup with the trajectory it belonged to.
The number that broke our mental model:
Across those 11 agents, the single most expensive step — measured per resolved task, not per API call — was the model in exactly 2 of them. In the other 9, something we'd filed under "negligible overhead" was quietly eating more than the LLM.
Here is the trap, in one picture. Same agent, same steps — two ways of ranking them.
WHAT THE DASHBOARD SHOWS (per-call mean) WHAT THE BILL ACTUALLY IS (per-trajectory total)
Step $/call rank Step $/task rank
LLM completion $0.0028 #1 PII / guardrail $0.0042 #1
PII / guardrail $0.00035 #4 Embedding loop $0.0036 #2
Embedding (in loop) $0.00002 #5 LLM completion $0.0028 #3
Semantic-cache chk $0.00090 #3 Semantic-cache chk $0.0009 #4
Tool call $0.00010 #2 Tool call $0.0008 #5
Left column: what every dashboard reports. Right column: what one task actually costs. The model drops from #1 to #3. A step you pay two-hundredths of a cent per call becomes the #2 cost driver. The ranking is not the same list.
And across all 11 agents, the #1 cost step by trajectory-total:
#1 cost driver (per trajectory), across 11 production agents
Embedding / retrieval loop ██████████ 5 / 11
PII / guardrail model ██████ 3 / 11
Model (LLM) ████ 2 / 11
Cache-verify model ██ 1 / 11
Nine of eleven. The model isn't even in the lead most of the time.
The hidden axis: mean-vs-trajectory
The reason dashboards mislead is structural. They report mean cost per call, because that's what one request easily exposes. But an agent step isn't called once — it's called inside loops, per chunk, per re-plan.
The real cost of a step is:
step_cost_per_task = calls_per_task × cost_per_call
A step at $0.00002/call looks free on the mean chart. Call it 180 times in a retrieval loop and it's $0.0036/task — more than the LLM, which runs once. The "free" step wins by 28%, on exactly the hard tasks you care about.
The invisible cost axis is trajectory-total vs call-mean. Every dashboard I've seen only draws one of them.
Three cases we actually shipped
These aren't hypotheticals — they're line items from the 11 agents.
Case 1 — The "free" embedding step that wasn't.
We had a retrieval step the finance team literally labeled free in the cost model (embeddings at $0.00002/call). It lives inside the agent's re-planning loop, which on hard tasks re-retrieves up to 180 times. 180 × $0.00002 = $0.0036/task. The LLM completion, called once, is $0.0028. So the step we scored as zero out-costs the model by 28% — on every hard task, which is the only tasks that matter.
Case 2 — The cache hit that cost more than the miss.
Our semantic cache doesn't just return the cached answer on a hit. It runs a reranker plus a 70B "is this still relevant?" check, to avoid serving stale advice. That verification costs $0.0009. A genuine miss regenerates from the 8B model for $0.0006. So a hit — the thing caches exist to save you — costs 50% more than a miss, across the 40% of traffic that hits. We were paying a "savings" tax and calling it a win.
Case 3 — The guardrail that 4×'d the bill.
A PII-scrubber runs a 7B model on every message chunk before the LLM sees it: 12 chunks × $0.00035 = $0.0042/task. We added it for a compliance ask and never costed it per-trajectory. It became the #1 line item on 3 of the 11 agents. Removing it wasn't on the table — but routing it to a 0.5B model dropped it to $0.0003, a 14× cut, with no detectable quality change on our eval set.
How you should actually measure
The fix is almost never "pick a cheaper model." It's accounting per trajectory so the loop multiplier can't hide, then routing each step to the right-sized model. We sit in front of 25+ models behind one OpenAI-compatible gateway; the interesting savings came from moving the scrubber and the cache-verify down, not from swapping the completion.
Three changes that took an afternoon:
-
Attribute every token to a
trajectory_id, not a request. If you can't say what one task cost, you don't know your cost. - Rank by total, sample real tasks. Pull N resolved tasks, sum per step, sort by total. The mean chart will lie to you; the total won't.
-
Put a
cost_per_resolved_taskSLO in place. Alert on the trajectory, not the call.
A minimal version of what we run:
cost_attribution:
unit: trajectory # not 'request' and not 'call'
trace_id: trajectory_id # tag every step, tool, cache lookup
rank_by: total_cost # never mean_cost
alert:
metric: cost_per_resolved_task
threshold_usd: 0.02
exclude_steps: [] # never exclude the 'cheap' steps
And the five lines that surface the buried step:
from collections import defaultdict
def rank_steps_by_task(events):
per_step = defaultdict(float)
for e in events:
per_step[e.step] += e.cost_usd # already trajectory-attributed
return sorted(per_step.items(), key=lambda kv: -kv[1])
# prints the REAL ranking — usually not what your dashboard shows
for step, cost in rank_steps_by_task(events):
print(f"{step:24} ${cost:.5f}")
The question I'll leave you with
The dashboards aren't lying. They're reporting the axis that's easiest to measure. We spent a year optimizing model choice while a loop multiplier and a "free" safety step ran away from us — invisible the whole time, because nothing on the screen was drawing the trajectory-total axis.
So: when you last looked at your agent's cost, were you ranking by what one call costs — or by what one task actually costs? Because those are not the same list.
What's the step you've been treating as free?
Top comments (19)
The same mean-versus-trajectory trap shows up on the input side. An MCP tool definition looks like a one-off setup cost, but it rides along on every turn of the trajectory. In the 20 tool definitions I measured this week, the heaviest was 32.7K tokens, nearly all of it an output schema inferred from sample rows, and nobody sees it on a per-call dashboard.
The MCP tool-definition tax is one of the best-hidden ones — it reads as a one-time setup cost and then rides along on every single turn. 32.7K on a single def is wild; the schema-inferred-from-sample-rows part is the bit nobody sees on a per-call dashboard. We hit a similar wall where tool schemas dwarfed the actual step output. Did you ever try caching or pre-computing those output schemas instead of rebuilding them each turn, or is the inference cost just accepted as fixed?
Not tried. The schema is inferred by the hosted server from sample rows, so on our side the only lever is the rows themselves: flat rows give a small schema. Whether a declared dataset schema replaces the inference I have not tested.
Cost hides in the intermediate steps, so a total per task can still hide where it went. Report cost per step, and report the cache-hit path's own overhead — reranking and verification work on a hit — as its own line, separate from miss cost. A hit is not automatically a saving until that full hit path is measured against the miss it replaces.
This is the 'treat the hit path as a first-class cost' move the article was pushing for — and splitting reranking from verification on a hit is the right cut. We found verification (confirming the hit is actually valid) was the quietly expensive half, because it's work you pay whether or not the cache 'saved' you anything. A hit only becomes a saving once you measure that full hit path against the miss it replaced, exactly as you said. In your numbers, does verification or reranking dominate the hit-path overhead?
Good cut. I won't guess a split — we don't publish a reranking-vs-verification breakdown. What we do publish is the whole hit path: a 14-step trajectory replay at 47ms against 3.9s live, so the hit-path cost is already inside that 47ms. The reason I'd bet on verification as the heavier half is structural, not measured: in our design verification is a gate, not a ranking — a write-bearing trajectory never serves on a fuzzy match, an abstract match is refused as an exact repeat with context diffed, and a context change with no taint info is a miss — so the hit pays the full eligibility check before anything is served. Splitting that check out as its own line, the way you split reranking from verification, is exactly how you'd find out whether the bet is right.
The mean-vs-trajectory table is the part that rewired how I read agent cost dashboards. I have been ranking steps by $/call too — and writing off the embedding loop as noise — until you put the 180× re-retrieve case next to a single LLM completion. Per-task totals make the "free" step look expensive in a way the provider bill never will.
The 180x re-retrieve case is exactly the kind of number that stays invisible until you line it up against a single completion. We saw the same shape with embedding loops that read as 'noise' on a per-call average and only show their weight once you sum them per task. The part I keep coming back to: a per-task total can make a 'free' step look expensive in a way the provider bill never will, because the bill has no idea what your step boundaries are. Curious whether your 180x was a cache miss triggering a re-fetch, or a deterministic loop that just kept going — and how you account for steps like that once they're inside a per-task id.
real token accounting on 11 production agents. that's the part most agent posts skip.
That's the gap, yeah — most agent posts stop at 'which model we picked' and skip the accounting entirely. The instrumentation is boring work and it's where the interesting numbers hide. Out of curiosity, when you run agents, what cost item surprised you most once you actually measured?
Retries and context reloads add up faster than the headline model call. That 9 out of 11 split is why the boring accounting matters.
the cache hit costing 50% more than a miss is the line that stuck with me. we saw the same shape when a reranker sat behind our retrieval cache: it cost more per hit than just regenerating on a small model, and it only showed up once we summed cost per task instead of per call. the per call dashboard looked healthy the whole time. how are you handling trajectory_id when a task fans out into parallel sub agents, one id per root task or per branch?
The reranker-behind-retrieval-cache shape is the exact one we hit too — a hit costing more than a small-model regen, invisible until you sum per task instead of per call. On trajectory_id: we anchor on the root task id as the trajectory key, and branches get a composite key (root, branch_index) so we can both aggregate across the tree and not smear one branch's cost onto another. When a branch retries, it reuses the same composite key rather than minting a new one. Is your fan-out a DAG or a tree — and if a branch can re-enter, how do you keep the re-entry from double-counting?
Data versioning is the part everyone skips until it bites them.
Data versioning really is the 'ignore it until it bites you' category, same as cost attribution — fine while nothing breaks, painful the moment it does. When it bit you, was it training-data drift or an eval set that had silently changed under you?
The instrumentation part is where most teams stop. Did the token data change which model you picked?
Good question, and it did change some picks. Once per-step cost was visible, a few steps we'd been running on a frontier model quietly moved down — they were doing cheap, stable work that never needed the heavy model. The flip side was a couple of 'cheap' steps that were secretly the most expensive once retries stacked up. Did you instrument in a staging environment first, or go straight to production instrumentation?
Data versioning is the part everyone skips until it bites them.
The eval-set approach is the one I've actually stolen for our own models.