Cache affinity in practice: 5 patterns that keep your agent's prompt cache warm

You shipped an agent. Tokens/month exploded. You switched to a "cheaper" model and the bill barely moved.

Here's the thing nobody prints on the pricing page: in a loop, the model price is often a rounding error next to the repeat tax — and the repeat tax is, at its core, a cache-affinity problem.

This is the practical follow-up to why agentic systems should care about cache-hit pricing. That post argued the cost lives in cache behavior, not in raw model price. This one is about what you can actually do about it.

What prefix caching actually rewards

Most providers (OpenAI, Gemini, Anthropic, and the OpenAI-compatible gateways on top of them) offer automatic prefix caching: if the start of your prompt is byte-for-byte identical to a previous request, the cached tokens cost a fraction of a fresh input token.

The keyword is byte-for-byte identical, and it only helps if the identical part sits at the front of the prompt. Reorder one line, inject a timestamp above the system prompt, or re-serialize history with a fresh UUID, and you've evicted your own cache. The model never sees the hit — you just pay full input, every step.

So "cheaper model" optimizes the wrong number. The lever is cache affinity: how stable is your prefix across the loop?

5 patterns that keep the cache warm

1. Stable system prompt + fixed tool schemas at the very top. Tools definitions rarely change between steps. Put them first, verbatim, and stop touching them.

2. Append-only history. Don't re-serialize the whole conversation each step. Keep a canonical transcript and append; let the unchanged prefix stay cached.

3. Inject volatile context after the stable prefix. Scratchpads, retrieved docs, and tool results are fine — as long as they sit below the system prompt + history, not above it.

4. Route the easy 80% to a small model, but preserve the shared prefix. Routing by scenario is smart. Just don't let the small model re-format the prefix; a different tokenizer can silently break the cache.

5. One gateway, one canonical formatter. When three sub-agents each format "the context" their own way, you get three incompatible prefixes and zero cache reuse. Centralize prompt assembly.

5 patterns that evict your cache (the O(n²) traps)

1. Re-sending the entire conversation every step. Frameworks that replay full memory each turn pay O(n²) tokens over a session. The prefix can never stabilize.

2. Re-serializing history with a timestamp or UUID in the prefix. A new updated_at each call = a new prefix = no cache, forever.

3. Putting volatile content above the stable prefix. Current time, request id, trace id — if it's before the system prompt, it poisons every cache hit downstream.

4. Mixing tokenizers across models without a stable canonical form. Switching models mid-loop without canonicalizing the prefix resets the cache and doubles your input cost.

5. No observability. If you can't see cache hit vs miss per request, you can't tell which of the above you're doing. You're flying blind on the single biggest cost lever.

The trace you're missing

The fix that makes everything else measurable is boring but decisive: a per-request trace that reports, for every call, whether the prefix hit cache and how many tokens were charged vs cached.

Once you have that, the O(n²) traps show up as a line item. You stop guessing and start watching the cache-hit ratio the way you watch p99 latency. That's the difference between "we cut model cost" and "we cut agent cost."

Landing AI means making this the default

"AI 落地" (getting AI into production) is sold as a model-access problem. It isn't. Access is solved. The hard part is the boring operational layer: warm caches, visible traces, sane routing — by default, not as a heroic refactor after the bill arrives.

If you want a gateway that surfaces cache-hit/miss per request and keeps one canonical prefix across models, that's the whole point of what we're building at TokenLat: https://tokenlat.com