title: "The Hidden Bill: Where AI Agent Costs Actually Come From"
published: true
description: "The model's per-token price is the cheapest part of running an AI agent. The real bill comes from the loop around it. Here's where the money actually goes."
tags: ai, agents, llm, cost
Everyone budgets AI agents the same wrong way: look up the model price ("$3 per million tokens, cheap"), multiply by how many questions you'll ask, feel good.
Then the bill lands at 5–10x that.
The per-token price was never the number that mattered. An agent isn't a chatbot you ask once — it's a loop that re-reads, retries, calls tools, and thinks in steps, and every one of those is billed. Here's where the money actually goes, with real September 2026 prices.
The number everyone quotes (and why it lies)
Current API pricing per million tokens, input / output:
| Model | Input | Output |
|---|---|---|
| Haiku 4.5 | $1 | $5 |
| Sonnet 5 | $2 | $10 |
| Sonnet 4.6 | $3 | $15 |
| Opus 5 / 4.8 | $5 | $25 |
| Fable 5.1 | $10 | $50 |
You reason: "2,000-token prompt, 500-token answer — under 2 cents on Sonnet. I could run thousands for pocket change."
Correct for a single call. Completely wrong for an agent. Here's why.
Leak #1: You pay for the whole conversation, every single turn
This is the one that surprises people most, so start here.
LLMs are stateless. The model doesn't "remember" your conversation — every turn, your client re-sends the entire history as input. Turn 10 isn't billed as one message. It's billed as all 10 messages, plus every tool result in between.
So an 8-step task doesn't cost 8 × one call. It's closer to 1 + 2 + 3 + … + 8 — the context grows every turn and you pay for all of it, again, each turn.
A coding agent finishes in 12 turns. By turn 12 the context is 40,000 tokens: the task, the files it read, past tool outputs, its own reasoning. You didn't pay for 40K once — you paid a growing slice on every turn to get there. Re-sent input quietly becomes the biggest line item, often bigger than all your output combined.
The wrong instinct: "input is cheap, output is what costs." True for a chatbot. For an agent, re-sent input usually tops the bill.
Leak #2: Tool calls are round trips, and each one restarts the meter
Every time your agent calls a tool — read a file, hit an API, run a search — that's not free thinking time. It's a full round trip:
- Model outputs "I want to call tool X" (billed output).
- Your code runs the tool.
- The tool's result gets appended to the context and sent back (billed input, forever, on every later turn too).
A big tool result is the worst offender. Dump a 10,000-token API response or file into the context, and you now pay to re-send those 10,000 tokens on every subsequent turn of the loop. One fat tool output can cost more than the entire rest of the task.
The fix people miss: trim tool outputs before they enter context. You rarely need the whole JSON blob — you need three fields. Summarize or filter tool results before they hit the model, not after.
Leak #3: Retries and dead ends you never see
In a demo, the agent nails the task in a clean line. In production, it doesn't.
- It calls a tool with a bad argument, gets an error, tries again.
- It goes down a wrong path for four turns, realizes, and backtracks.
- Your framework retries on a timeout or a malformed response.
- A guardrail rejects an action and the agent re-plans.
Every one of those is billed at full price, and none of them show up in your happy-path math. A task you budgeted at 5 turns routinely runs 9 in the wild. That's not a bug — that's what "agentic" means. Budget for the wandering, not the demo.
Leak #4: Multi-agent fan-out multiplies everything
The moment you go from one agent to "a planner that spawns sub-agents," your costs don't add — they multiply.
Each sub-agent has its own context, its own loop, its own tool calls, its own retries. A planner that fans out to five workers isn't 5x one call; it's five independent instances of Leaks #1–#3, plus the planner's own overhead synthesizing their results. Fan-out is a fantastic capability and a fantastic way to 10x a bill without noticing.
Rule of thumb: before you add a sub-agent, ask if a single agent with one more tool would do. Parallelism is worth paying for when the wall-clock time matters — not by default.
Leak #5: The expensive model doing cheap work
Teams pick one model — usually a strong, pricey one — and route everything through it. But most of what an agent does is not hard:
- classifying which path to take
- extracting a value from text
- deciding "is this done yet?"
- formatting output
That's Haiku work ($1/$5) being billed at Opus rates ($5/$25) — a 5x markup for tasks the cheap model does just as well. The reasoning-heavy step might genuinely need the big model. The other eleven steps in the loop usually don't.
The levers that actually move the bill
Now the good news. Once you see the loop, the fixes are obvious and most of them are free wins, not tradeoffs:
- Prompt caching. This is the big one. Cached input reads cost ~0.1x the base rate — a 90% cut on the repeated part of your context. Since agents re-send the same system prompt and early context every turn (Leak #1), caching is aimed directly at your biggest line item. On Sonnet, cached reads drop from $3 to $0.30 per million. Turn it on before anything else.
- Trim tool outputs before they enter context (kills Leak #2).
- Cap the loop. A hard turn limit stops a wandering agent from billing you for its confusion (Leak #3).
- Route by difficulty. Cheap model for the routing/extraction/"are we done" steps, expensive model only for the hard reasoning (Leak #5).
- Batch the non-urgent work. Batch processing is 50% cheaper across the board. If it doesn't need to be real-time, batch it.
Stack caching + batching and effective spend can fall 90%+ on the repeated parts of your workload. Not by using a worse model — by not paying full price for the same tokens over and over.
The mental model to keep
Stop pricing agents like chatbots. A chatbot is one call. An agent is a loop that re-sends a growing context, pays for every tool round trip, wanders when it's confused, and multiplies when it fans out.
The model's sticker price is the cheapest variable in that whole system. The bill is written by the loop.
Cost the loop, not the token — and cache the part you keep re-sending.
What's the biggest surprise you've hit on an agent bill? I'm especially curious whether re-sent context or tool-output bloat was the bigger leak for you — I keep seeing teams blame the model when it was the loop the whole time. 👇
Top comments (12)
This is one of the more practical explanations of AI agent costs I’ve seen. A lot of teams still estimate expenses by looking only at the model’s token price, but the real architecture cost comes from everything around the model.
The point about context growth is especially important. Tool outputs, retries, and multi-step reasoning quietly turn a simple request into a much larger workload. It reminds me of how database costs are often not about one query, but about the pattern of queries over time.
The tool output trimming suggestion is a great one. Many agent systems treat the context window like unlimited memory, but every extra piece of information has a cost and can also reduce reasoning quality.
The shift from “which model is cheapest?” to “how do we design the loop efficiently?” feels like the right mental model for production AI systems.
Great breakdown — these are the kinds of operational details people discover only after the first real bill arrives.
Perfect analogy — agent cost is a loop property, not a call property. One prompt looks cheap; it's the retries, the growing context, and the tool-output accumulation across a multi-step run that turn a $0.01 request into a $2 one. The shift from "which model is cheapest" to "how do we design the loop" is the whole game. What's bitten you hardest — context growth, or retry storms?
The query pattern analogy fits agent loops well. One prompt looks modest, but re-sending 30k tokens of previous turns and raw tool returns turns single-call arithmetic upside down. Trimming tool outputs before context insertion helps keep the active prompt bounded. We're JZS Token, an independent prepaid AI API option; check published rates at jzstoken.com when modeling per-task budgets against growing context.
The query-pattern analogy is the one that made it click for me too — single-call arithmetic lies the moment you loop. One turn looks cheap; re-sending 30k tokens of prior turns plus raw tool returns on every subsequent turn is the actual bill, and it grows quadratically with conversation length if you're not careful.
Exactly — and the highest-leverage version is trimming at the source: strip a tool's output to the fields the model actually needs before it ever enters history, rather than letting a 5k-token JSON blob ride along for the rest of the session. Bounding the active prompt is the whole game once you're in an agent loop.
The tool output bloat is worse than raw token count because it directly sabotages prompt caching. If a tool dumps an unstripped JSON payload or a compiler trace with dynamic timestamps into the transcript, it busts the cache breakpoint for every subsequent turn. Trimming payloads before they enter history doesn't just reduce input volume; it preserves the static prefix for the rest of the run.
This is the sharper version of the point and honestly I wish I'd put it in the post 👏 you're right that it's not just volume — a dynamic tool output (timestamps, request IDs, unstripped JSON) poisons the static prefix and busts the cache breakpoint for the whole rest of the run. So one messy tool result quietly turns off your 90% caching discount on everything after it.
Feels like the real rule is "keep the transcript prefix deterministic" — trim AND normalize (strip timestamps/IDs) before anything enters history. Have you found a clean way to place cache breakpoints around unavoidable dynamic content, or do you just push it all to the end of context?
Dynamic timestamps and unstripped JSON in tool outputs are a double hit: they inflate input tokens and break prompt cache breakpoints for every subsequent turn. Normalizing payloads before appending them to history preserves the static prefix and caching discount. We're JZS Token, a separate prepaid AI API option. Compare current rates at jzstoken.com when evaluating cache hit ratios against raw payload size.
Good catch linking these two — they're the same lever from both ends:
That second half is the one people miss. A single changing field (a timestamp, a request id, an
Age:header) near the top of a tool payload invalidates the static prefix, so you lose the cache discount on everything after it — not just the bytes that changed. Normalizing/stripping payloads before appending them to history isn't only about token count; it's about preserving the cacheable prefix so repeated turns stay cheap. Trimming protects caching, which is the connection the post should've drawn explicitly.Exactly - and the operational consequence is worth naming: once a changing field sits in the payload, its cost is not its own length, it is the length of everything after it in the prefix. That asymmetry is why stripping pays even when the token saving on a single call looks trivial. The check I would add is a prefix hash before and after normalisation, so the day a new field starts breaking the breakpoint you see it in CI rather than in the invoice.
That's the right way to state it, and it's why a 20-byte timestamp can cost 20,000 tokens of cache.
On the CI check, one sharpening: hash the prefix as serialized for the model, not the payload object. Two things break the cache that an object-level hash won't catch: key-order instability in JSON serialization, and whitespace or float formatting differences between runs. Comparing the hash across two consecutive turns of the same fixture catches both. The day it changes, the diff tells you which field, and that's a five-minute fix instead of a line item on the invoice.
Context bloat from tool outputs hit me harder than I expected. I had a loop fetching supplier records, and each response came back with way more text than I actually needed. By turn 8 or so, the bulk of what I was sending was old tool results I had already processed. Truncating those down before appending cut the spend more than any prompt caching did.
Yeah, exactly this 🙌 the "old tool results I'd already processed" part is the killer — you extract what you need on turn 3 and then keep paying to re-send the raw blob for turns 4 through 12. Once you've pulled the fields you care about, the original payload is just dead weight riding along on every turn.
Out of curiosity — did you truncate blindly (cap to N chars) or actually parse out the fields you needed? I keep going back and forth on how aggressive to be before I start losing context the agent actually uses later.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.