For a few months our chat feature paid full price for tokens it had already sent a minute earlier. Each turn of a conversation re-sent the same large block of context about the user's book, and the provider's prompt cache should have discounted almost all of it. In practice only the system prompt was being cached. The measured hit rate on the first call of each turn was about 26%.
After one structural change it went to 86 to 87%, and a three-turn conversation got about a third cheaper. The change was small, but finding it took longer than it should have, because our mental model of how prompt caching works was wrong. This post is about that model.
What we were building
I'm the co-founder and CTO of Hypercritica, where writers upload a manuscript and an AI editor marks it up and talks with them about it. When a writer opens a chat about their book, every request carries:
- a fixed system prompt (the editor's instructions),
- a large per-book block: title, the editor's memory of the book, earlier decisions the writer made, open requests, reference material,
- a small per-turn block: the current thread, where the writer is in the manuscript, recent edits, and the new message.
The per-book block is big and changes rarely. The per-turn block is small and changes every time. That is the textbook case for prompt caching.
The assumption that cost us
We assumed the cache worked on shared leading tokens. Put the stable content first, the volatile content last, and the provider will reuse the longest common prefix. So we ordered everything carefully: system prompt, then the book block, then the turn block, all in one user message.
When we finally measured it, the stable-first ordering on its own had produced roughly nothing beyond the system prompt.
We wrote probes to find out why. Two results settled it:
- Two requests that shared 8,000 identical leading tokens, then diverged inside the same message, got zero cached tokens on the second call.
- A request that repeated an earlier request byte for byte and then appended new input items after it got a cache hit on everything that matched.
So, at least for the model we use, the cache does not match an arbitrary shared prefix. A stored entry is reused when the whole earlier prompt is a token-level prefix of the new one. If your stable and volatile content live in the same message, no earlier prompt is ever a clean prefix of the next, and you only get the parts that come before that message.
TTL was not the problem. Hits still worked after 90 seconds and more, and interleaved traffic from other users did not evict anything.
The fix: split, then prime
Once we knew the rule, the design followed.
Split the message. The request became three parts instead of two:
[system prompt]
[user: stable per-book block]
[user: volatile per-turn tail]
The per-book block is byte-identical across turns for the same book and persona, and it sits in its own message.
Prime the boundary. Splitting alone is not enough, because the cache only stores prompts that were actually sent. So before the first real turn we send one cheap request that ends exactly at the end of the stable block. That stores [system][stable block] as a complete prompt, and every later turn starts with exactly those tokens, so it hits.
Priming is memoized per cache key. We re-prime when the hash of the stable block changes (the writer made a decision that updates the book's memory, say) or when the last prime is a few minutes old.
Be strict about what counts as stable. We added a rule to the code: any new chat context goes in the per-turn tail unless it is truly per-book and stable. One volatile field slipping into the stable block quietly drops the hit rate back to the old number.
What it bought us
Measured on a real book over three turns:
- First call of each turn: 86 to 87% of input tokens cached, up from 26%.
- Three-turn cost: about 35% lower.
- A single-turn chat costs roughly 10% more, because of the prime. Most chats run longer than one turn, so this is a good trade, but it is a real cost to keep in mind.
We applied the same split to the longer pipeline passes that run over every chunk of a manuscript. One large prime there lets each chunk call hit about 90% cached.
Keeping it from regressing
The scary part of prompt caching is that it fails silently. Nothing errors. Your bill goes up and latency creeps, and nobody notices until someone reads an invoice.
So we added a daily check. For each model route it compares the cached-token share over the last 24 hours with the trailing seven days, and it alerts when the rate drops by 20 points from a baseline of at least 25%, as long as there is enough traffic to mean something. Reordering a prompt builder or adding a field to the stable block is exactly the kind of change that zeroes the discount, and now it shows up the next morning instead of at the end of the month.
What I'd tell someone starting out
- Measure cached tokens per call from day one. Don't trust the ordering you designed.
- Find out how your provider actually matches. Ours needed the full earlier prompt to be a prefix, which is stricter than "shared leading tokens."
- Put stable context in its own message and prime it.
- Treat the stable block as an interface. Anything that changes per turn stays out.
- Alert on the hit rate, because nothing else will tell you it broke.
Yash Wanvari is the co-founder and CTO of Hypercritica, the publishing home for writers.
Top comments (0)