The last piece argued that comparing a $250K engineer to a $20K token bill is a trap, because the two numbers measure different things and the cheap one hides its real cost. Fine. But "the token bill is not the whole story" leaves a follow-up hanging: what is actually in that $20K? Where does the money go once the invoice clears?
It goes to waste, mostly. Not fraud, not overpriced models. Ordinary, invisible waste that nobody instruments because the bill arrives as one number and one number tells you nothing. A $20K monthly spend on production AI is not a cost. It is a pipeline with leaks at four specific joints, and every one of them is measurable.
The bill is not one line, it's four leaks
Start with the model. On Claude, Opus 4.8 runs $15 per million input tokens and $75 per million output, and Sonnet lands at roughly a fifth of that. Those are the posted numbers, and they are the part of the bill you can't argue with. What you can argue with is how many tokens you send, how many times you send them, and how many of those sends did no useful work.
Break a real production bill apart and the same four categories show up every time: retries and stalls, context and prompt bloat, tool-schema overhead from MCP servers, and redundant eval or judge passes. Each one has a fix. None of the fixes require a better model.
Leak one: retries and stalls
An agent loop that hits a rate limit, a malformed tool call, or a timeout doesn't fail cleanly. It retries. And the retry re-sends the entire context that got you to the failure point, so a stall at turn nine costs you turns one through nine again, in full, at output prices if the model already started generating.
This is the quiet killer in agentic workloads. A single user request that should cost one round trip can spawn six because the agent got confused, called a tool wrong, read the error, and tried again. In production agent loops, error-recovery and re-planning routinely consume more tokens than the productive reasoning does. The model isn't thinking harder. It's redoing work.
The fix is boring and it works: cap retries explicitly, log every retry with the reason, and treat a high retry rate as a bug in your tool definitions, not a cost of doing business. If your agent retries a tool 30% of the time, the tool's description is unclear or its schema is wrong. Fix the schema and the retries disappear. Instrument this first, because it's usually the biggest single line and the one teams never look at.
Leak two: context and prompt bloat
Every request carries a system prompt, a set of instructions, skill definitions, and whatever conversation history you've accumulated. Most teams have no idea how big that payload is. They wrote the system prompt eight months ago, bolted three more skills onto it, and never measured the total.
Measure it. A tool like token-baseline run across your prompt, skill, and command corpus gives you a real number per component, so you can see that the 4,000-token "helpful preamble" nobody has read since launch is riding along on every single call. At Opus input rates, 4,000 wasted tokens on 100,000 daily calls is real money, and it buys you nothing.
The structural fix is prompt caching. Anthropic's prompt caching charges a 25% premium to write a cache entry and then serves cache reads at 10% of the base input price. If your system prompt is stable across calls, and it should be, caching turns a repeated 4,000-token tax into a one-time write plus pennies per read. Teams that cache their stable prefix routinely cut input spend by half or more. Teams that don't are paying full freight to re-send the same unchanging text thousands of times a day.
Leak three: MCP tool-schema overhead
This one is new enough that most teams haven't caught it yet. Connect an agent to a handful of MCP servers and every request now carries the full JSON schema for every tool those servers expose, before the model reads a word of the actual task. GitHub's server alone ships around 35 tools. Stack a few servers and you can burn 50,000-plus tokens on schemas the model will not use on this particular request.
You can audit this directly. A tool like mcp-token-audit measures the token cost of each connected server's schema payload, and the results are usually ugly. Half your context window can be tool definitions for capabilities the current task doesn't touch.
The fix is on-demand tool loading: expose tool names to the model, and load the full schema only when the model decides to call the tool. I wrote about the mechanics of this separately, but the short version is that you should never pay to describe a tool the request won't use. Audit your schema payload, then defer everything you can.
Leak four: redundant eval and judge passes
The last leak comes from good intentions. You added an LLM judge to grade outputs, then an eval pass to check the judge, then a second judge for confidence. Now every production response triggers three extra model calls, and two of them are asking nearly the same question.
Judge calls are output-heavy and they compound. If your judge re-reads the full input plus the candidate answer plus a rubric on every call, you're paying to re-process the same context three times to answer one quality question. Sample instead of grading everything. Grade 5% of production traffic continuously and the full set only when you ship a prompt change. Collapse redundant judges into one call with a structured multi-field output. The goal is confidence in your quality, not a receipt for every token you can spend proving it.
Instrument the pipeline, then fix the joints
Here's the position: stop treating the token bill as a price and start treating it as telemetry. The number on the invoice is the sum of four measurable subsystems, and every one of them leaks in a way you can see the moment you point a tool at it.
Run token-baseline against your prompt and skill corpus this week. Run mcp-token-audit against your connected servers. Turn on prompt caching for your stable prefix. Put a counter on retries and set an alarm when the rate climbs. None of this is exotic, and none of it needs budget approval. It needs someone to accept that "$20K in tokens" is a diagnosis waiting to happen, not a fact to shrug at.
The trap in the first piece was believing the token number was small. The trap in this one is believing it's a single number at all. It isn't. It's a pipeline. Instrument it, and half of it turns out to be leak.
Top comments (0)