The metric that quietly decides your agent bill isn't the input price of your model. It's how much you pay to read what you already sent.
If you run multi-step agents, you've probably had this moment: a task you expected to cost pennies comes back as a small surprise on the invoice. You didn't change models. You didn't prompt more. So where did the tokens go?
Most of the time, they went to paying for the same context, again and again.
A 2-minute task can fire 40+ billable calls
Here's a shape I keep seeing. An agent does a "2-minute" job:
- reads a file
- drafts a plan
- calls a tool
- reflects on the result
- retries
- summarizes
Each step is an LLM call. And each call re-sends the same scaffolding: the system prompt, the task description, and — critically — the growing conversation history. A step that adds 200 new tokens of thinking can still carry 4,000 tokens of context it already paid for once.
Multiply that across 40 steps and the math stops being about "model price." It's about how many times you re-pay for context you already have.
What cache-hit pricing actually is
Normal input pricing charges you per token you send, every time. Cache-hit pricing changes the unit: if the provider already has your prefix cached (because you sent it recently and it hasn't expired), the read of that cached prefix is billed at a deep discount instead of full input price.
On a unified gateway this is visible per model. Two concrete examples from the model catalog:
- DeepSeek-v4-Pro: ¤10 / 1M tokens for a cached read (¤ is the platform's billing unit)
- Qwen3.5-plus: ¤20 / 1M tokens for a cached read
Those aren't the full input rates — they're the cached-read rates, and that's the number that matters for agentic workloads, because agentic workloads are mostly repeats.
The takeaway isn't "this model is cheaper." It's: for any loop that re-sends context, cached-read price is the real marginal cost, and most teams optimize for the wrong number.
The math that actually moves the bill
Skip the exact figures and look at the shape. Say a loop runs 40 calls, and each carries ~4k tokens of repeated context plus ~200 tokens of new content.
- Pay full input on every repeated token: you're billed for 4k × 40 = 160k "new" tokens that were actually old.
- Pay cached-read on the repeated prefix: that 160k drops to a fraction — the cached-read rate instead of full input.
On the two models above, cached reads land at ¤10–¤20 / 1M versus full input that's multiple times higher. The loop's cost doesn't go to zero, but the repeat tax collapses. In agentic systems, the repeat tax is most of the bill — so this is where the 70%+ savings actually live, not in "pick a cheaper model."
You can't optimize what you can't see
There's a trap here. Cache-hit pricing only helps if you can see hits and misses. A black-box API that just returns text hides the one number you need: was this token a cache hit or a fresh charge?
A request trace makes it observable. Each call should expose its stages:
- REQUEST — what came in
- AUTH — who/what called
- ROUTE — which model served it
- RESPONSE — what came back
- METER — what it cost, including cache hit/miss
When every call shows its cache hit/miss and per-stage cost, the loop stops being a mystery. You can see which step blows the budget and whether your prefix is actually staying warm.
Routing + cache affinity is the real lever
Cache-hit pricing is necessary but not sufficient. The other half is keeping the cache warm, and that's a routing problem.
The 80/20 pattern holds: route the easy 80% of calls to a small/fast model, keep frontier for the hard 20%. But the part people miss is cache affinity — if you keep the same system prompt and stable prefix across the loop, the cache stays warm and the cheap reads keep hitting. Change the prefix on every step (reformat the history, rewrite the system prompt, shuffle the order) and you silently evict your own cache. You pay full input forever.
With model: "auto" style routing behind one OpenAI-compatible endpoint, the loop doesn't have to think about which model serves which step — but it still has to respect cache affinity, because that's what turns "cheap model" into "cheap and cached."
A practical cache-friendly checklist
- Keep the system prompt and stable prefix byte-identical across loop steps
- Append new content; don't rebuild the whole context each time
- Prefer providers/models that expose cache-hit pricing and a request trace
- Watch cache hit rate per step, not just total spend
- Route by task, but preserve prefix stability so the cache survives
The point
Agentic cost isn't a model-selection problem. It's a repeat-tax problem: how many times you pay to read context you already sent, and whether you can see it happening.
Cache-hit pricing + request trace + cache-friendly routing is the combination that turns a scary agent bill into a boring one. Most teams optimize the first and ignore the other two — which is why their invoices still surprise them.
If you want the routing playbook that pairs with this (the 80/20 split and how to keep the cache warm), it's at https://tokenlat.com — but the idea stands on its own: stop counting input price. Start counting cache hits.
Top comments (4)
I hit this in my own system recently, after a comment on one of my articles sent me looking. Two things were wrong and only one is on your checklist: my system prompt was assembled fresh each turn, so the prefix moved on every call in a long run, and on Anthropic I was not caching at all, because there you have to mark the cache breakpoints yourself and without a marker nothing caches and nothing warns you. A hit-rate number would have told me I was at zero, but not why.
Switching models is a cold cache too. On Anthropic the first call after a switch costs more than normal input, because a cache write is 1.25x the base rate, 2x if you want it to live an hour. So the cheap model isn't cheap on the step where it takes over. How does
model: "auto"know a step is worth that?Exactly the failure mode that pushed us to write this. Both traps you hit are ones we kept hitting too: re-serializing the system prompt every turn silently moves the prefix and evicts the cache, and on the Anthropic side there's no way to mark a breakpoint manually — you're at the mercy of their auto-bucketing, no warning when it misses. The sneakiest part is the model switch: cold cache plus the 1.25x–2x write upcharge means the step where "the cheap model takes over" is often the most expensive one. What we ended up doing is pinning system + first user turn as a fixed prefix and forcing every call to return a hit/miss marker, otherwise the money leaks and you never see where.
I think the deeper implication is that cache-hit rate is really a measure of architectural determinism, not just cost optimization. If identical intent produces different prompt prefixes because each agent mutates context differently, you've already lost before pricing comes into play. The teams that consistently reduce inference costs aren't just optimizing prompts they're designing agent workflows where context is treated as a stable protocol. Once you start measuring prefix stability alongside cache-hit rate, routing efficiency and cost become much more predictable. That's a much more useful KPI than average cost per request.
I love reframing cache-hit rate as a measure of architecture determinism rather than just a cost tactic. That was the same instinct behind the post: if the same intent produces a different prefix every time, you've already lost before pricing even enters the picture. Once prefix stability and hit rate are both measurable, routing stops being folklore and becomes engineering. Your line about "treating context as a stable protocol" deserves its own post.