The metric that quietly decides your agent bill isn't the input price of your model. It's how much you pay to read what you already sent.
If you run multi-step agents, you've probably had this moment: a task you expected to cost pennies comes back as a small surprise on the invoice. You didn't change models. You didn't prompt more. So where did the tokens go?
Most of the time, they went to paying for the same context, again and again.
A 2-minute task can fire 40+ billable calls
Here's a shape I keep seeing. An agent does a "2-minute" job:
- reads a file
- drafts a plan
- calls a tool
- reflects on the result
- retries
- summarizes
Each step is an LLM call. And each call re-sends the same scaffolding: the system prompt, the task description, and — critically — the growing conversation history. A step that adds 200 new tokens of thinking can still carry 4,000 tokens of context it already paid for once.
Multiply that across 40 steps and the math stops being about "model price." It's about how many times you re-pay for context you already have.
What cache-hit pricing actually is
Normal input pricing charges you per token you send, every time. Cache-hit pricing changes the unit: if the provider already has your prefix cached (because you sent it recently and it hasn't expired), the read of that cached prefix is billed at a deep discount instead of full input price.
On a unified gateway this is visible per model. Two concrete examples from the model catalog:
- DeepSeek-v4-Pro: ¤10 / 1M tokens for a cached read (¤ is the platform's billing unit)
- Qwen3.5-plus: ¤20 / 1M tokens for a cached read
Those aren't the full input rates — they're the cached-read rates, and that's the number that matters for agentic workloads, because agentic workloads are mostly repeats.
The takeaway isn't "this model is cheaper." It's: for any loop that re-sends context, cached-read price is the real marginal cost, and most teams optimize for the wrong number.
The math that actually moves the bill
Skip the exact figures and look at the shape. Say a loop runs 40 calls, and each carries ~4k tokens of repeated context plus ~200 tokens of new content.
- Pay full input on every repeated token: you're billed for 4k × 40 = 160k "new" tokens that were actually old.
- Pay cached-read on the repeated prefix: that 160k drops to a fraction — the cached-read rate instead of full input.
On the two models above, cached reads land at ¤10–¤20 / 1M versus full input that's multiple times higher. The loop's cost doesn't go to zero, but the repeat tax collapses. In agentic systems, the repeat tax is most of the bill — so this is where the 70%+ savings actually live, not in "pick a cheaper model."
You can't optimize what you can't see
There's a trap here. Cache-hit pricing only helps if you can see hits and misses. A black-box API that just returns text hides the one number you need: was this token a cache hit or a fresh charge?
A request trace makes it observable. Each call should expose its stages:
- REQUEST — what came in
- AUTH — who/what called
- ROUTE — which model served it
- RESPONSE — what came back
- METER — what it cost, including cache hit/miss
When every call shows its cache hit/miss and per-stage cost, the loop stops being a mystery. You can see which step blows the budget and whether your prefix is actually staying warm.
Routing + cache affinity is the real lever
Cache-hit pricing is necessary but not sufficient. The other half is keeping the cache warm, and that's a routing problem.
The 80/20 pattern holds: route the easy 80% of calls to a small/fast model, keep frontier for the hard 20%. But the part people miss is cache affinity — if you keep the same system prompt and stable prefix across the loop, the cache stays warm and the cheap reads keep hitting. Change the prefix on every step (reformat the history, rewrite the system prompt, shuffle the order) and you silently evict your own cache. You pay full input forever.
With model: "auto" style routing behind one OpenAI-compatible endpoint, the loop doesn't have to think about which model serves which step — but it still has to respect cache affinity, because that's what turns "cheap model" into "cheap and cached."
A practical cache-friendly checklist
- Keep the system prompt and stable prefix byte-identical across loop steps
- Append new content; don't rebuild the whole context each time
- Prefer providers/models that expose cache-hit pricing and a request trace
- Watch cache hit rate per step, not just total spend
- Route by task, but preserve prefix stability so the cache survives
The point
Agentic cost isn't a model-selection problem. It's a repeat-tax problem: how many times you pay to read context you already sent, and whether you can see it happening.
Cache-hit pricing + request trace + cache-friendly routing is the combination that turns a scary agent bill into a boring one. Most teams optimize the first and ignore the other two — which is why their invoices still surprise them.
If you want the routing playbook that pairs with this (the 80/20 split and how to keep the cache warm), it's at https://tokenlat.com — but the idea stands on its own: stop counting input price. Start counting cache hits.
Top comments (0)