Providers cache your prompt by exact token-prefix match. What you get back is the longest common prefix between this request and the cached one, and nothing after the first difference — however stable the rest of it is.
So one volatile line decides the bill. Take a 14-token Current date: …, which is 0.16% of an 8,514-token system prompt, and walk it through all seven insertion points. Placed last, the steady bill is 0.1113× the uncached price. Placed first, it is 1.2474× — above 1.000×, so caching there costs more than not caching at all. That is 11.21× between two prompts with identical tokens in a different order.
👉 Live, runs in your browser: https://dev48v.infy.uk/prompt/day80-prefix-caching.html
What is measured and what is declared
Declared, and stated above the first number: the three price multipliers — a cache write costs 1.25× a fresh input token, a cache read 0.10×, fresh input 1.00× — plus a $3.00/Mtok input rate for turning multiples into money. Change two constants and every figure moves with them.
Measured: the prefix matcher, the cumulative arithmetic, the break-even algebra and its agreement with a request-by-request simulation, and every count. There is no sampling anywhere, and nothing here simulates language — the engine is an ordered list of segments with declared token lengths.
The prompt is invented. The page says so.
The hit is a cumulative sum, and that is the whole mechanism
| volatile field at index | sits before | cache hit | steady bill | break-even N* |
|---|---|---|---|---|
| 0 | tool definitions | 0 | 1.2474× | never |
| 1 | operating rules | 3,120 | 0.8304× | 3 |
| 2 | output schema | 4,960 | 0.5844× | 2 |
| 3 | house style guide | 5,520 | 0.5096× | 2 |
| 4 | worked examples | 5,930 | 0.4548× | 2 |
| 5 | escalation policy | 8,170 | 0.1554× | 2 |
| 6 | — (it is last) | 8,500 | 0.1113× | 2 |
Put the field at index p and the hit is exactly the cumulative token count of segments 0…p−1. That column is measured by running the matcher; the cumulative sum is computed separately; they agree at every position.
The assertion that carries the page is monotonicity: moving the field one segment earlier can never increase the hit. Checked at every insertion point of this fixture and of a 48-segment one — 56 positions, not a few sampled slots.
Break-even is closed-form, and its denominator is the interesting part
cold = Ta*w + Tb*b first request writes everything
warm = H*r + (Ta-H)*w + Tb*b every request after it
cold + (N-1)*warm <= N*(Ta+Tb)*b
cold - warm = H*(w - r) the tail cancels
(Ta+Tb)*b - warm = H*(w - r) - Ta*(w - b) and again
N* = ceil( H(w-r) / ( H(w-r) - Ta(w-b) ) ) denominator <= 0 => never
The user's actual turn cancels out of the ratio entirely, which is why it does not appear. And a negative denominator means no number of requests ever repays the write — placed first, the page checks that out to 1,000,000 requests and it is still dearer.
Placed last, the loan is repaid on request 2. The derived N* matches the request-by-request simulation at all 56 positions, and the check is two-sided: at N* the cached run is no dearer, and at N*−1 it really is — so N* is the smallest such N, not merely some N.
The threshold that does not depend on your prompt
Set that denominator to zero and the fixture drops out. What is left is a property of the price model alone: a cached prefix under (w−b)/(w−r) = 21.74% of the cacheable prompt never repays its own write, at any N. The self-check pins the number printed in the prose to the one the formula derives.
Two fixes that look equivalent; one does nothing
The instinct on a cache that never hits is to make the volatile thing smaller — shorter date format, an id instead of a name, drop the seconds. Hit length does not depend on the volatile field's token count at all. At position 0 the hit is 0 whether the field is 1 token or 800, because the matcher stops at the first differing token and the field's own length is entirely on the far side of that stop.
Same corollary for several volatile fields: the hit is decided by the earliest one. Everything after it is already free to change, and tidying those is worth exactly nothing.
What this is not about
Not breakpoints, TTL or minimum-prefix tiers. All of those are held fixed here — they are a different page's subject. This one varies exactly one thing, where the volatile field sits, and prices the consequence. Also not modelled: cache lifetime and eviction, concurrency, and which prefix a provider actually keeps when several are live. And nothing here is about which instruction wins when two conflict — segment order changes the bill; it is not an argument about precedence.
19 in-page checks, 284 verifier asserts, 0 failures — plus 4 deliberate mutations of the engine, 4 caught.
Top comments (0)