title: "The Hidden Bill: Where AI Agent Costs Actually Come From"
published: true
description: "The model's per-token price is the chea...
For further actions, you may consider blocking this person and/or reporting abuse
This is one of the more practical explanations of AI agent costs I’ve seen. A lot of teams still estimate expenses by looking only at the model’s token price, but the real architecture cost comes from everything around the model.
The point about context growth is especially important. Tool outputs, retries, and multi-step reasoning quietly turn a simple request into a much larger workload. It reminds me of how database costs are often not about one query, but about the pattern of queries over time.
The tool output trimming suggestion is a great one. Many agent systems treat the context window like unlimited memory, but every extra piece of information has a cost and can also reduce reasoning quality.
The shift from “which model is cheapest?” to “how do we design the loop efficiently?” feels like the right mental model for production AI systems.
Great breakdown — these are the kinds of operational details people discover only after the first real bill arrives.
Perfect analogy — agent cost is a loop property, not a call property. One prompt looks cheap; it's the retries, the growing context, and the tool-output accumulation across a multi-step run that turn a $0.01 request into a $2 one. The shift from "which model is cheapest" to "how do we design the loop" is the whole game. What's bitten you hardest — context growth, or retry storms?
The query pattern analogy fits agent loops well. One prompt looks modest, but re-sending 30k tokens of previous turns and raw tool returns turns single-call arithmetic upside down. Trimming tool outputs before context insertion helps keep the active prompt bounded. We're JZS Token, an independent prepaid AI API option; check published rates at jzstoken.com when modeling per-task budgets against growing context.
The query-pattern analogy is the one that made it click for me too — single-call arithmetic lies the moment you loop. One turn looks cheap; re-sending 30k tokens of prior turns plus raw tool returns on every subsequent turn is the actual bill, and it grows quadratically with conversation length if you're not careful.
Exactly — and the highest-leverage version is trimming at the source: strip a tool's output to the fields the model actually needs before it ever enters history, rather than letting a 5k-token JSON blob ride along for the rest of the session. Bounding the active prompt is the whole game once you're in an agent loop.
The tool output bloat is worse than raw token count because it directly sabotages prompt caching. If a tool dumps an unstripped JSON payload or a compiler trace with dynamic timestamps into the transcript, it busts the cache breakpoint for every subsequent turn. Trimming payloads before they enter history doesn't just reduce input volume; it preserves the static prefix for the rest of the run.
This is the sharper version of the point and honestly I wish I'd put it in the post 👏 you're right that it's not just volume — a dynamic tool output (timestamps, request IDs, unstripped JSON) poisons the static prefix and busts the cache breakpoint for the whole rest of the run. So one messy tool result quietly turns off your 90% caching discount on everything after it.
Feels like the real rule is "keep the transcript prefix deterministic" — trim AND normalize (strip timestamps/IDs) before anything enters history. Have you found a clean way to place cache breakpoints around unavoidable dynamic content, or do you just push it all to the end of context?
Dynamic timestamps and unstripped JSON in tool outputs are a double hit: they inflate input tokens and break prompt cache breakpoints for every subsequent turn. Normalizing payloads before appending them to history preserves the static prefix and caching discount. We're JZS Token, a separate prepaid AI API option. Compare current rates at jzstoken.com when evaluating cache hit ratios against raw payload size.
Good catch linking these two — they're the same lever from both ends:
That second half is the one people miss. A single changing field (a timestamp, a request id, an
Age:header) near the top of a tool payload invalidates the static prefix, so you lose the cache discount on everything after it — not just the bytes that changed. Normalizing/stripping payloads before appending them to history isn't only about token count; it's about preserving the cacheable prefix so repeated turns stay cheap. Trimming protects caching, which is the connection the post should've drawn explicitly.Exactly - and the operational consequence is worth naming: once a changing field sits in the payload, its cost is not its own length, it is the length of everything after it in the prefix. That asymmetry is why stripping pays even when the token saving on a single call looks trivial. The check I would add is a prefix hash before and after normalisation, so the day a new field starts breaking the breakpoint you see it in CI rather than in the invoice.
That's the right way to state it, and it's why a 20-byte timestamp can cost 20,000 tokens of cache.
On the CI check, one sharpening: hash the prefix as serialized for the model, not the payload object. Two things break the cache that an object-level hash won't catch: key-order instability in JSON serialization, and whitespace or float formatting differences between runs. Comparing the hash across two consecutive turns of the same fixture catches both. The day it changes, the diff tells you which field, and that's a five-minute fix instead of a line item on the invoice.
Context bloat from tool outputs hit me harder than I expected. I had a loop fetching supplier records, and each response came back with way more text than I actually needed. By turn 8 or so, the bulk of what I was sending was old tool results I had already processed. Truncating those down before appending cut the spend more than any prompt caching did.
Yeah, exactly this 🙌 the "old tool results I'd already processed" part is the killer — you extract what you need on turn 3 and then keep paying to re-send the raw blob for turns 4 through 12. Once you've pulled the fields you care about, the original payload is just dead weight riding along on every turn.
Out of curiosity — did you truncate blindly (cap to N chars) or actually parse out the fields you needed? I keep going back and forth on how aggressive to be before I start losing context the agent actually uses later.