DEV Community: TokenLat

Cache affinity in practice: 5 patterns that keep your agent's prompt cache warm

TokenLat — Tue, 11 Aug 2026 07:46:05 +0000

You shipped an agent. Tokens/month exploded. You switched to a "cheaper" model and the bill barely moved.

Here's the thing nobody prints on the pricing page: in a loop, the model price is often a rounding error next to the repeat tax — and the repeat tax is, at its core, a cache-affinity problem.

This is the practical follow-up to why agentic systems should care about cache-hit pricing. That post argued the cost lives in cache behavior, not in raw model price. This one is about what you can actually do about it.

What prefix caching actually rewards

Most providers (OpenAI, Gemini, Anthropic, and the OpenAI-compatible gateways on top of them) offer automatic prefix caching: if the start of your prompt is byte-for-byte identical to a previous request, the cached tokens cost a fraction of a fresh input token.

The keyword is byte-for-byte identical, and it only helps if the identical part sits at the front of the prompt. Reorder one line, inject a timestamp above the system prompt, or re-serialize history with a fresh UUID, and you've evicted your own cache. The model never sees the hit — you just pay full input, every step.

So "cheaper model" optimizes the wrong number. The lever is cache affinity: how stable is your prefix across the loop?

5 patterns that keep the cache warm

Stable system prompt + fixed tool schemas at the very top. Tools definitions rarely change between steps. Put them first, verbatim, and stop touching them.
Append-only history. Don't re-serialize the whole conversation each step. Keep a canonical transcript and append; let the unchanged prefix stay cached.
Inject volatile context after the stable prefix. Scratchpads, retrieved docs, and tool results are fine — as long as they sit below the system prompt + history, not above it.
Route the easy 80% to a small model, but preserve the shared prefix. Routing by scenario is smart. Just don't let the small model re-format the prefix; a different tokenizer can silently break the cache.
One gateway, one canonical formatter. When three sub-agents each format "the context" their own way, you get three incompatible prefixes and zero cache reuse. Centralize prompt assembly.

5 patterns that evict your cache (the O(n²) traps)

Re-sending the entire conversation every step. Frameworks that replay full memory each turn pay O(n²) tokens over a session. The prefix can never stabilize.
Re-serializing history with a timestamp or UUID in the prefix. A new updated_at each call = a new prefix = no cache, forever.
Putting volatile content above the stable prefix. Current time, request id, trace id — if it's before the system prompt, it poisons every cache hit downstream.
Mixing tokenizers across models without a stable canonical form. Switching models mid-loop without canonicalizing the prefix resets the cache and doubles your input cost.
No observability. If you can't see cache hit vs miss per request, you can't tell which of the above you're doing. You're flying blind on the single biggest cost lever.

The trace you're missing

The fix that makes everything else measurable is boring but decisive: a per-request trace that reports, for every call, whether the prefix hit cache and how many tokens were charged vs cached.

Once you have that, the O(n²) traps show up as a line item. You stop guessing and start watching the cache-hit ratio the way you watch p99 latency. That's the difference between "we cut model cost" and "we cut agent cost."

Landing AI means making this the default

"AI 落地" (getting AI into production) is sold as a model-access problem. It isn't. Access is solved. The hard part is the boring operational layer: warm caches, visible traces, sane routing — by default, not as a heroic refactor after the bill arrives.

If you want a gateway that surfaces cache-hit/miss per request and keeps one canonical prefix across models, that's the whole point of what we're building at TokenLat: https://tokenlat.com

Why agentic systems should care about cache-hit pricing

TokenLat — Fri, 07 Aug 2026 08:32:46 +0000

The metric that quietly decides your agent bill isn't the input price of your model. It's how much you pay to read what you already sent.

If you run multi-step agents, you've probably had this moment: a task you expected to cost pennies comes back as a small surprise on the invoice. You didn't change models. You didn't prompt more. So where did the tokens go?

Most of the time, they went to paying for the same context, again and again.

A 2-minute task can fire 40+ billable calls

Here's a shape I keep seeing. An agent does a "2-minute" job:

reads a file
drafts a plan
calls a tool
reflects on the result
retries
summarizes

Each step is an LLM call. And each call re-sends the same scaffolding: the system prompt, the task description, and — critically — the growing conversation history. A step that adds 200 new tokens of thinking can still carry 4,000 tokens of context it already paid for once.

Multiply that across 40 steps and the math stops being about "model price." It's about how many times you re-pay for context you already have.

What cache-hit pricing actually is

Normal input pricing charges you per token you send, every time. Cache-hit pricing changes the unit: if the provider already has your prefix cached (because you sent it recently and it hasn't expired), the read of that cached prefix is billed at a deep discount instead of full input price.

On a unified gateway this is visible per model. Two concrete examples from the model catalog:

DeepSeek-v4-Pro: ¤10 / 1M tokens for a cached read (¤ is the platform's billing unit)
Qwen3.5-plus: ¤20 / 1M tokens for a cached read

Those aren't the full input rates — they're the cached-read rates, and that's the number that matters for agentic workloads, because agentic workloads are mostly repeats.

The takeaway isn't "this model is cheaper." It's: for any loop that re-sends context, cached-read price is the real marginal cost, and most teams optimize for the wrong number.

The math that actually moves the bill

Skip the exact figures and look at the shape. Say a loop runs 40 calls, and each carries ~4k tokens of repeated context plus ~200 tokens of new content.

Pay full input on every repeated token: you're billed for 4k × 40 = 160k "new" tokens that were actually old.
Pay cached-read on the repeated prefix: that 160k drops to a fraction — the cached-read rate instead of full input.

On the two models above, cached reads land at ¤10–¤20 / 1M versus full input that's multiple times higher. The loop's cost doesn't go to zero, but the repeat tax collapses. In agentic systems, the repeat tax is most of the bill — so this is where the 70%+ savings actually live, not in "pick a cheaper model."

You can't optimize what you can't see

There's a trap here. Cache-hit pricing only helps if you can see hits and misses. A black-box API that just returns text hides the one number you need: was this token a cache hit or a fresh charge?

A request trace makes it observable. Each call should expose its stages:

REQUEST — what came in
AUTH — who/what called
ROUTE — which model served it
RESPONSE — what came back
METER — what it cost, including cache hit/miss

When every call shows its cache hit/miss and per-stage cost, the loop stops being a mystery. You can see which step blows the budget and whether your prefix is actually staying warm.

Routing + cache affinity is the real lever

Cache-hit pricing is necessary but not sufficient. The other half is keeping the cache warm, and that's a routing problem.

The 80/20 pattern holds: route the easy 80% of calls to a small/fast model, keep frontier for the hard 20%. But the part people miss is cache affinity — if you keep the same system prompt and stable prefix across the loop, the cache stays warm and the cheap reads keep hitting. Change the prefix on every step (reformat the history, rewrite the system prompt, shuffle the order) and you silently evict your own cache. You pay full input forever.

With model: "auto" style routing behind one OpenAI-compatible endpoint, the loop doesn't have to think about which model serves which step — but it still has to respect cache affinity, because that's what turns "cheap model" into "cheap and cached."

A practical cache-friendly checklist

Keep the system prompt and stable prefix byte-identical across loop steps
Append new content; don't rebuild the whole context each time
Prefer providers/models that expose cache-hit pricing and a request trace
Watch cache hit rate per step, not just total spend
Route by task, but preserve prefix stability so the cache survives

The point

Agentic cost isn't a model-selection problem. It's a repeat-tax problem: how many times you pay to read context you already sent, and whether you can see it happening.

Cache-hit pricing + request trace + cache-friendly routing is the combination that turns a scary agent bill into a boring one. Most teams optimize the first and ignore the other two — which is why their invoices still surprise them.

If you want the routing playbook that pairs with this (the 80/20 split and how to keep the cache warm), it's at https://tokenlat.com — but the idea stands on its own: stop counting input price. Start counting cache hits.

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

TokenLat — Fri, 31 Jul 2026 04:43:52 +0000

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

Your agent's token bill is probably 5x higher than it needs to be — not because the models are expensive, but because of routing discipline. Most teams wire every call to one frontier model and call it a day. This post is the practical fix: how to send the easy 80% of agent calls to cheap models, keep frontier for the hard 20%, and not lose a point of quality doing it.

Everything below uses real numbers from a gateway that exposes 22 models across 8 providers behind one OpenAI-compatible API.

The default that's costing you

Here's the shape of a typical agent loop:

classify the request
extract structured fields
pick a tool
call the tool, parse the result
summarize what happened
decide the next step
draft the reply

Of those, maybe one or two steps actually need frontier-level reasoning. The rest are classification, extraction, and formatting — work that a fast, cheap model does indistinguishably from a flagship. Yet most setups send all of it to gpt-5.5 (or whatever their default is) because it's the path of least resistance.

That's the tax. You're paying flagship prices for steps that don't need it.

The 80/20 split, with the math

Route by task difficulty, not by model loyalty:

Easy 80% — classification, extraction, summarization, short tool-formatting, routing decisions → a fast China-model tier
Medium 15% — coding, planning, multi-step reasoning → a strong reasoning tier
Hard 5% — the genuinely gnarly frontier cases → flagship, reserved

A blended-cost estimate (input tokens, per 1M):

80% × ¤160   (cheap tier input)   = ¤128
15% × ¤560   (reasoning tier)     = ¤84
 5% × ¤7000  (flagship input)     = ¤350
---------------------------------------
blended input cost               ≈ ¤562 / 1M
vs. all-flagship                 = ¤7000 / 1M
savings                          ≈ 70%+ in this example

Even with a more conservative split, 70%+ cheaper is the floor — and quality doesn't move, because the hard calls still go to frontier.

The trap: why "just use the cheapest model" backfires

Naive cost-cutting picks one cheap model for everything. That fails for two reasons:

Quality cliffs. A ¤160/1M model is great at extraction and terrible at planning a 12-file refactor. Push hard tasks down and users notice.
Cache invalidation (the silent tax). This is the one almost nobody budgets for. Long agent conversations re-send the system prompt + tool definitions on every turn. If you switch providers mid-conversation, you reset prompt-cache affinity and re-pay the full prefix each turn. A conversation that should be cheap suddenly isn't.

The win isn't "use a cheap model" — it's routing discipline with cache affinity intact: pin the easy 80% to a stable cheap model so the cache stays warm, and only escalate to frontier for the calls that earn it.

What a real routing layer looks like

With a unified gateway, this is roughly ten lines. One API, models swapped by model: — no per-provider SDK juggling:

def route(task):
    if task.complexity == "easy":        # classify, extract, summarize
        return "TokenLat-deepseek-v4-Flash"   # ¤160/1M input
    if task.needs_reasoning:             # code, plan, multi-step
        return "TokenLat-deepseek-v4-Pro"     # ¤560/1M input, ¤10/1M cache read
    return "chatgpt-5.5"                 # frontier, only the hard 5%

resp = client.responses.create(
    model=route(task),
    input=task.prompt,
)

Or skip the router entirely and let the gateway decide per call:

resp = client.responses.create(
    model="auto",          # gateway picks the right tier per request
    input=task.prompt,
)

model: "auto" is the zero-config version of the same idea — the gateway applies routing policy so you don't hand-roll it.

Real numbers from a 25-model gateway

A unified gateway should show you its pricing up front. Per 1M tokens, in credits (¤), here's a slice (input / cache-read):

TokenLat-deepseek-v4-Flash — cheap / fast tier → ¤160 / ¤40
TokenLat-qwen3.5-plus — general tier → ¤130 / ¤20
TokenLat-deepseek-v4-Pro — reasoning tier → ¤560 / ¤10 (lowest cache-read)
TokenLat-glm-5.1 — Chinese-stable → ¤950 / ¤160
TokenLat-kimi-k3 — flagship (China) → ¤3000 / ¤300
gemini-3.1-pro — frontier → ¤2800 / ¤300
chatgpt-5.5 — frontier → ¤7000 / ¤700

The headline gap: cheap-tier input is ~44x cheaper than flagship input. That's the entire 80/20 argument in one line.

Cache-hit pricing — the agentic loophole

For agentic systems, the input price is almost a distraction. What matters is the cache-read price, because the static prefix (system prompt, tool schemas, retrieved context) gets re-sent every turn.

TokenLat-deepseek-v4-Pro caches reads at ¤10/1M
TokenLat-qwen3.5-plus at ¤20/1M

So a long-running agent loop that pins a stable model and keeps its prefix cached can serve most turns at the cache-read rate — a fraction of the input rate. The providers that reset your cache on every switch quietly erase this saving. Cache affinity is a routing decision, not an afterthought.

Make it visible

You can't optimize what you can't see. Every request through a proper gateway should leave a trace:

REQUEST → AUTH → ROUTE(auto→text-pro) → RESPONSE → METER
region: SEA | status: 200 OK | latency: 842ms | tokens: 1,284 | cost: ¤0.0048

Five stages, one line, per call. When a bill spikes, you see which model, which route, and which request — instead of staring at a monthly total and guessing.

Don't guess — measure quality

The one thing this playbook assumes: you actually check that the cheap tier is good enough. Route a sample of real tasks through both tiers, score the outputs on your success criteria, and only then commit the split. Most teams find the quality cliff is far smaller than they feared — and the ones where it isn't, they've already reserved frontier for.

The takeaway: the savings aren't in hunting for a cheaper model. They're in routing the easy 80% to a stable cheap tier, keeping cache affinity, and escalating only the hard calls to frontier. Do that and 70%+ cheaper is routine — with quality intact.

If you want to try this without wiring up eight providers yourself: TokenLat is a unified AI gateway for Malaysia & Southeast Asia — one OpenAI-compatible API across 22 models, with auto routing and per-request cost traces. The model list above is live at tokenlat.com.

What's your current cost per agent-task — single digits or scaling faster than you'd like? Curious how the 80/20 split would land on your workload.

Your agent's token bill is 5x too high — and it's not the model price

TokenLat — Mon, 27 Jul 2026 09:45:27 +0000

Most teams blame their model provider when the inference bill spikes. They're looking at the wrong line item.

The real leak is architecture — and it's the difference between a token bill that scales with value and one that scales with chaos. Here's what we see shipping agentic systems in production.

The hidden multiplier: agent loops

A "2-minute task" is never one call. An agent fires 30–60 tool calls per run, and most frameworks stuff the entire conversation history into every prompt. So a job you'd estimate at ~4K tokens becomes 40 calls × 8K context = 320K tokens — billed at frontier rates.

Frontier pricing per call looks cheap. Multiplied by agent-loop iterations, it quietly becomes the largest line in your cloud bill.

The 80/20 of inference

Not every call needs a frontier model.

~80% of agent traffic is routing, extraction, formatting, classification, summarization. Trivial. Leading efficient models — including top China models — handle these at near-parity.
~20% is genuine reasoning, open-ended generation, ambiguous planning. That's where frontier earns its price.

Route the 80% to efficient models and reserve frontier for the 20%. Same output quality. A fraction of the bill.

A unified gateway beats a drawer of API keys

The trap most teams hit: they wire 4 providers with 4 clients, then let a naïve router "roam" between them. On failover it loses cache affinity, re-embeds context, and your 1.5x cost target drifts back toward ~1x — or worse.

A single OpenAI-compatible endpoint across OpenAI + Gemini + leading China models fixes this:

One client, one code path.
Provider pinning holds cache locality; it only fails over on hard error, not price drift.
Your application code never changes when you swap a model.

In SEA, "PDPA-aligned" is the baseline, not a premium

For Malaysia and SEA teams, inference isn't just a cost question — it's a compliance one.

PDPA requires 72-hour breach notification and a designated DPO.
In-region data residency (SG-hosted) is now the default expectation, not a paid add-on.
The adoption gap is real: ~93% of the workforce uses GenAI, but ~73% are stuck at L1 (experimenting, not in production). The blocker is almost always cost + compliance friction — not the models.

Compliance stops being a tax and becomes table stakes when your gateway is PDPA-aligned by design.

The numbers

In our pricing analysis, frontier output runs on the order of 100x the cost of efficient China-model output per million tokens. Route the easy 80% there and blend:

Up to 90%+ cheaper on the routine 80%
70%+ lower blended bill overall
Quality on the 20% that matters: unchanged

The fix is architecture, not negotiation

You don't fix a 5x token bill by begging your provider for a discount. You fix it by routing, caching, and unifying — then letting compliance be the default instead of the exception.

We break down agent cost architecture weekly. If you're shipping agents and your bill is climbing faster than your usage, come find us:

Discord: discord.gg/rxEtWG897V
Site: tokenlat.com