A team I talked to last month had a 4x jump in inference spend over one quarter. Not a crisis — they had shipped an agent, usage grew, revenue grew. The problem was smaller and more annoying: when finance asked which part of the product had gotten more expensive, nobody could answer.
Three engineers, three different guesses. Every dashboard was green. Every invoice was correct. And still, no one could name the step.
That is the failure mode I keep running into. Not "the bill is too high." The bill is correct and unreadable.
I want to argue this isn't a finance problem or a pricing problem. It's an instrumentation problem, and it has a specific cause: the unit of work changed, and the unit of attribution didn't.
Per-token prices fell. The bill still went up.
Every model you depend on has gotten cheaper per token over the last two years. And yet the bill went up. Both things are true, which should tell you that price is not the variable that matters.
What changed is the number of calls behind a single unit of user-visible work.
In the chat era, "summarize this customer's last quarter" was one request: a prompt, a completion, done. In the agent era, the same sentence is a plan call, a handful of retrieval or embedding calls, two or three tool calls against internal systems, a compression step so the tool output fits, a context rebuild, a generation, a validation pass, and possibly a second generation because validation failed.
Twenty to forty calls for one thing the user perceives as a single action.
And here's the part that makes the arithmetic non-linear: context is a multiplier, not an addend. Call #30 doesn't carry the cost of call #30. It carries the accumulated results of calls #1 through #29. As a loop gets deeper, input tokens grow roughly with the square of the step count while useful output stays flat.
So the shape of the bill changed. It used to be a small number of expensive things. Now it's a large number of individually cheap things, where cost concentrates in the plumbing between the things everyone is looking at.
The attribution problem
Most teams instrument cost along axes that made sense for a chat product:
- by API key
- by endpoint
- by team or environment
These answer who. They do not answer why.
The conversation I described above usually goes like this. Someone pulls up spend by API key and says: platform team is 60% of the bill. Okay — but the platform team runs eleven features. Which one? And inside that feature, was it the retrieval step, the tool calls, or the retries? The attribution stops one layer above where the money is actually being spent.
When the unit of work becomes a task chain, attribution has to move with it. In practice that means you need to answer three different questions at three different altitudes:
- Total. How much this month, split by model and by outcome (200 vs. 429 vs. timeout — a status code is a cost signal, not just a health signal).
- Attribution. Which feature, which tenant, which task type.
- Single step. Inside this one trace, which of the 34 calls ate 60% of the cost.
Almost every team I've talked to has level 1. Some have level 2. Very few have level 3 — and level 3 is the only one that changes behavior, because it's the only one that tells you what to fix.
There's a name for this in the vendor world now: an observability center with global, attribution, and per-step views. It's being sold as an enterprise feature. It shouldn't be a feature. It's the minimum for operating an agent at all.
Why the application layer can't answer this alone
Here's the part that's genuinely structural, and it's the reason this doesn't get fixed by "just add better logging."
Your application knows business semantics. It knows this is task tsk_8f21, for tenant acme, in the support_draft feature, at step retrieve_compress. What it does not know is the cross-model reality of that step: how many tokens the provider actually billed, whether the prompt cache hit, whether a silent retry tripled the input, or whether the same step got routed to a different model than last week.
The call layer knows all of that and knows none of the semantics. It sees POST /v1/chat/completions, 18,432 prompt tokens, 412 completion tokens, cache miss, 340ms. It has no idea this is a support draft for a customer named Acme.
So each side holds half the picture, and the split is exactly along the boundary where cost decisions get made. That's why "we'll join the logs later" fails: by the time you join them, the fields you need were never emitted.
The fix is unglamorous. Put the business coordinates on the request as it goes out, and let the layer that sees the tokens record them alongside the usage row.
# Business coordinates travel with the call, not in a side table.
COORDS = {
"X-Task-Id": "tsk_8f21",
"X-Task-Step": "retrieve_compress",
"X-Tenant": "acme",
"X-Feature": "support_draft",
}
resp = client.chat.completions.create(
model=model_for_this_step,
messages=messages,
extra_headers=COORDS,
)
# What comes back is not just a completion. It's a cost line item
# that already knows where it belongs.
That's it. One dict on the way out. Everything downstream — the usage table, the aggregation, the per-step breakdown — becomes possible; without it, nothing downstream can be reconstructed later.
Then the query you actually want is boring SQL instead of an archaeology project:
SELECT task_step,
SUM(prompt_tokens + completion_tokens) AS tokens,
SUM(cost_usd) AS cost,
COUNT(*) AS calls,
SUM(CASE WHEN status <> 200 THEN 1 ELSE 0 END) AS failures
FROM model_usage
WHERE task_id = 'tsk_8f21'
GROUP BY task_step
ORDER BY cost DESC;
Six lines. If your current stack can't answer that query, you don't have a cost problem yet — you have a cost problem you can't see, which is worse.
Three blind spots that show up in almost every audit
When teams do get per-step visibility, the same three things light up. It's predictable enough that I now look for them first.
1. Context rebuild. The loop re-sends conversation history, tool schemas, and retrieved documents on every iteration. Nobody budgeted for this because nobody counted it as a feature. A single page snapshot can be 20k+ tokens; do that inside a five-step loop and it dominates your bill while your completion tokens look healthy.
2. Silent retry. A retry that re-runs the whole step doesn't just cost 2x — it usually re-sends a larger context (the failed attempt got appended) and quietly resets whatever spend cap you set at the task level. Your guardrail was on the task; the cost was incurred on the step.
3. Tool output inflation. A tool returns 400KB of JSON. Two lines of it matter. It enters the context raw, gets re-sent on every subsequent call, and is paid for on each one. Trimming at the tool boundary is usually the single highest-leverage change available — higher than switching models.
None of these are model-quality problems. All three are invisible without step-level attribution and obvious with it.
Put the cap where the cost is incurred
One more thing that only becomes visible once you have step-level data: most teams define their spending limits in the wrong place.
A task-level cap is the intuitive choice — "this job may cost at most X." But it answers the wrong question. It tells you a task blew its budget; it doesn't tell you which step did it, and by the time it fires the task has usually already failed in a way the user noticed. Meanwhile the pathological call keeps happening: the context rebuilt for the twelfth time, the tool output that inflated 40x, the retry chain that never terminated.
A step-level cap fires on the specific call that went wrong, while the task can still degrade gracefully instead of dying.
There's a second, quieter problem. Caps implemented in application code are per instance. Three workers, three counters, three times the budget — and the counter resets on every deploy. A cap enforced in the layer that every call already passes through is global by construction, and it survives a rolling restart.
That's the same lesson as the attribution one, just from the other direction: if the layer that sees the tokens isn't the layer that enforces the policy, your guardrail is an estimate.
The self-test
I'll leave you with the question I now ask every team, because it separates the ones who can operate an agent from the ones who are still guessing:
Take the most expensive task your product ran last week. Can you name its three most expensive steps, and the reason each one cost what it did?
If yes, you have the instrumentation and the rest is tuning. If no, the bill isn't the problem — the missing step-level attribution is. And no amount of model shopping fixes a number you can't decompose.
Top comments (2)
the 20-40 calls per action bit is what gets people, cost per token looks great on the pricing page and then retries and context rebuilds eat it. tagging spend by task chain instead of api key seems like the only way finance ever gets an answer
A run total can't say which step moved it. Log input and output tokens per tool call, tagged with the step, and the step that grew usually shows within a day of data. Did the jump come from one step, or from all of them?