A 4% cache hit rate was costing us money. Here is the arithmetic I should have run first.

We turned on prompt caching for our document-QA service and the invoice went up.

Not dramatically. About 5%. Enough that I assumed it was traffic growth for the first two weeks, and it was not.

This post is the calculation I now run before anyone staffs a caching project. It takes ten minutes, it needs three numbers you already have, and in our case it would have told us what hit rate the cache had to clear before it was worth anything at all, and that the best possible outcome was smaller than anyone in the room believed.

What our contract actually charges

Two terms matter and only one of them gets discussed.

A cache read costs a tenth of the normal input rate. That is the term everybody quotes.

A cache write costs 1.25 times the normal input rate, and a miss writes the entry. So a request that misses pays a surcharge on its prefix for the privilege of storing it for the next one. Not every contract has this term. Ours does, and if yours does not, you should be able to point at the rate schedule that says so.

Put those together and there is a hit rate below which caching is a net loss:

h x 0.10 + (1 - h) x 1.25 = 1

which solves to h = 21.7%. Below a 21.7% hit rate, our cache was more expensive than no cache at all.

We shipped at 4%.

Why 4% and not zero

The system prompt opened with Current date and time: 2026-08-01T09:41:07.284Z, injected per request so the service could answer questions about document deadlines. Prefix caching keys on a prefix, so one variable field at position nine invalidates every token after it. Roughly 1,900 tokens of stable system prompt and tool schemas were being priced as novel, more than 40,000 times a day.

The residual 4% was our own retry logic. A retry re-sends the prompt object the client already built, timestamp string and all, so a retried request is byte-identical to its first attempt and lands on a warm prefix, provided the first attempt got far enough to write one. A retry after a 429 rejected at the edge finds nothing warm, because the attempt it is retrying never processed the prefix. That is why the hit rate sat a little below our retry rate for the month, which ran just under 5%, rather than on top of it. Nothing else was hitting the cache at all.

Moving the timestamp below the tool schemas, into the part of the message that was already variable, took the hit rate to 71%. One line of code.

It did not take it to 100%, and the gap is worth naming because it is the part you cannot engineer away either. Cache entries expire, our traffic has quiet stretches overnight and at weekends, and a request that arrives on a cold shard pays full price whatever the prompt looks like. Seventy-one percent was roughly what the traffic shape allowed.

The fix, and the most it could ever have been worth

A request looks like this for us:

System prompt + tool schemas: tokens 1,900, cacheable yes
Retrieved context: tokens 4,600, cacheable no, different every request
User turn: tokens 120, cacheable no
Total input: tokens 6,620, cacheable 28.7% of it
Output: tokens 350, cacheable no

Three states, all measured against a bill with no caching at all:

4% (where we shipped): effective input price per million $3.18, bill vs no cache 4.8% higher
71% (after the fix): effective input price per million $2.51, bill vs no cache 13.4% lower
100% (unreachable): effective input price per million $2.23, bill vs no cache 21.3% lower

The invoice moved 17.4% between the first row and the second, which is the number our dashboard celebrated. It is larger than the 13.4% we were actually saving, because the starting state was worse than not caching at all. The 17.4% is what the invoice did. The 13.4% is what the work was worth. A business case takes the second.

The row that matters most is the third. A perfect cache saves us 21.3%, and that was knowable on day zero. That number should have opened the planning conversation. We computed it at the end instead.

The ten-minute calculation

Three factors, multiplied:

Cacheable prefix, as a share of input tokens. Ours: 1,900 of 6,620, so 28.7%. Not tokens you feel are repetitive. Tokens that are byte-identical and sit before the first variable byte.

The discount on a cache read. Ours is 90% off, so 0.90.

Input spend as a share of the total bill. Output tokens never cache, and ours are billed at four times the input rate, which makes output 17.5% of our baseline invoice. So input is 82.5% of it, and that is the fraction the caching work can touch at all.

0.287 x 0.90 x 0.825 = 21.3%

That is the ceiling. Nothing in the caching work gets past it, because all three factors are properties of the prompt and the contract rather than of the implementation. If you want more, you have to change one of them: make the stable prefix a larger share of the request, negotiate the read rate, or shift the input/output balance. Adding cache infrastructure does none of those.

For what it is worth, the sprint did land. $1,059 a day became $875, so the invoice fell about $5,500 a month, of which about $4,100 is saving measured against never having cached at all. I would still have staffed it. I would have described it differently, and I would not have had to walk anyone back from a number I had implied but never computed.

The failure I had seen before, and why this one is different

I wrote a post a few weeks ago about a response cache running a 90% hit rate while the bill climbed, where the problem was that the hit rate counted requests and the requests it was hitting were the cheap ones (the write-up is here). Same headline metric, different failure, and the difference is worth spelling out because I initially conflated the two.

I should also own the embarrassing part. A timestamp in the system prompt was on the list of key-busters in that post. I found it in one cache, fixed it there, and shipped it into another one.

The distinction that matters: that post was about which requests hit. This one is about how much of a request can hit. Our prefix is 28.7% of the input on every single request, including the ones that hit perfectly, so a flawless cache still leaves 71.3% of the input tokens at full price. The traffic mix has no say in that; the prompt does. We fixed the first cache and the second one still had the same field sitting in it.

What I'd page on

• Cached-token share of input tokens, dropping more than 10 points below its 7-day median. This is cached_input_tokens / total_input_tokens from the usage payload, not requests. Ours sits at 20.4% at a 71% hit rate, so the alert fires below about 10%. It catches someone putting a variable field back near the top of a prompt, which has now happened to us twice.
• Hit rate falling below the break-even point. For us that is 21.7%, computed from the read and write terms in the contract. Below it the cache is a cost center, and unlike a drifting hit rate this threshold means something specific.
• Have you ever checked what your provider actually billed you, divided by the tokens you actually sent? That is the third alert: effective input price per million above $2.70 for two consecutive hours, against $2.51 healthy and $3.18 broken. It is derived from billing rather than from a feature dashboard, so it catches caching, tier changes, and anything else the provider does.
• Stable-prefix share of input tokens is the fourth thing I watch, and it is the only one that is not really an alert. When retrieval starts returning more chunks, the ceiling drops, quietly and without any deploy. Ours is a weekly review line rather than a page, but somebody should be looking at it before the next caching proposal, because it is the first of the three factors and the one most likely to have moved.

Comments

[Luis Cruz]

I completely agree with the author's approach to calculating the cache hit rate before implementing a caching project, and I've seen similar issues in my own experience with optimizing database queries. The fact that a cache write costs 1.25 times the normal input rate, while a cache read costs only a tenth of it, highlights the importance of considering both terms when evaluating the effectiveness of caching. The equation h x 0.10 + (1 - h) x 1.25 = 1 provides a clear threshold for determining when caching becomes a net loss, and it's surprising that the initial implementation had a hit rate of only 4%, well below the 21.7% threshold. What strategies do you think are most effective for identifying and mitigating cache invalidation issues, such as those caused by variable fields like the timestamp in this example?