A 30-turn measurement of how the saving amortises — and why the "one-line fix" quietly decays as conversations get longer.
Last week I published a benchmark showing that moving a roughly 30-token volatile header out of the top of a system prompt cut steady-state inference cost by 96%. The setup, the code and the raw API data are here: https://github.com/chenyu520-ai/rp-cache-lab
The most useful response came from Max Quimby, who runs scheduled multi-agent jobs and pushed back on the framing:
The amortization depends heavily on session length: if most of your sessions are only 2-3 turns, the cold turn-1 miss dominates and the win shrinks fast. Did you look at the hit-rate curve as a function of conversation length? That distribution is what decides whether this is a 95% win or a rounding error for a given app.
He was right, and I hadn't measured it. The first run was 8 turns, which is too short to answer the question. So I reran it at 30 turns.
This post is the answer, plus something I did not expect to find.
What was measured
Three ways of building the same request. The stable content — character card, world book, long-term memory, about 20,000 tokens — is byte-identical in all three. Only a small volatile header moves:
Session context: local time <ISO timestamp>, turn <n>, session <uuid>, mood index <0.00>.
| Variant | Where the volatile header goes |
|---|---|
| A — naive | Front of the system message |
| B — minimal fix | End of the system message |
| C — optimized | End of the newest user turn; the system message never changes |
30 turns per variant, deepseek-flash, thinking disabled, 90 requests total. Every number below comes from the API's own usage fields. The whole run cost $0.096.
The curve
The turn-1 miss is unavoidable — there is no cache to hit yet. That cost gets spread across the rest of the session, so short sessions carry it disproportionately.
| Session length | A avg/turn | B avg/turn | C avg/turn | C vs A |
|---|---|---|---|---|
| 1 turn | $0.002573 | $0.002574 | $0.002575 | ~0% |
| 2 turns | $0.002584 | $0.001339 | $0.001342 | −48.0% |
| 3 turns | $0.002590 | $0.000930 | $0.000932 | −64.0% |
| 5 turns | $0.002602 | $0.000609 | $0.000599 | −77.0% |
| 10 turns | $0.002625 | $0.000383 | $0.000350 | −86.7% |
| 20 turns | $0.002673 | $0.000301 | $0.000229 | −91.4% |
| 30 turns | $0.002723 | $0.000301 | $0.000189 | −93.1% |
Read the first row carefully. On a one-turn session there is no saving at all — C is fractionally more expensive, because its prompt is very slightly longer (the header rides in the user turn). There is no history to amortise the cold miss over.
An app where users routinely send one or two messages and leave gets roughly half the headline number. An app with 10+ turn sessions gets the whole thing.
Which means the honest answer to "does this save 96%?" is: not until you know your session-length distribution.
The part I didn't expect
The first run made B and C look equivalent. Over 8 turns their aggregate hit rates were 85.8% and 86.8% — a 1 percentage point gap, well inside noise.
Over 30 turns they are not equivalent at all:
| Turn 2 | Turn 30 | Change | |
|---|---|---|---|
| B cache hit rate | 98.9% | 90.5% | decaying |
| C cache hit rate | 98.7% | 98.7% | flat |
| B cost per turn | $0.000104 | $0.000333 | 3.2× |
| C cost per turn | $0.000110 | $0.000110 | flat |
B starts out nearly as good as C and then gets worse every turn. C does not move.
The mechanism is visible in the raw cache-hit token counts, and it is the most useful single table in this post:
| Turn | B hit tokens | C hit tokens |
|---|---|---|
| 2 | 16,896 | 16,896 |
| 5 | 16,896 | 17,280 |
| 10 | 16,896 | 17,664 |
| 20 | 16,896 | 18,688 |
| 30 | 16,896 | 19,584 |
B's hit count never changes. 16,896 is 264 × 64, and 64 appears to be the cache granularity. Because B's system message changes every turn, only the stable block ahead of the header can ever be reused — conversation history is permanently locked out of the cache. As the history grows, the uncached portion grows with it.
C's hit count climbs, because its system message is frozen and its history is append-only, so each turn's cached prefix includes everything before it.
So the "one-line fix" is real but incomplete. It buys you the stable block and nothing else. If your sessions are short, that is most of the win. If they are long, the gap between B and C becomes the thing that matters.
What this means in practice
- Sessions of 1–2 turns: don't restructure for this. There is nothing to amortise and you will see no meaningful change.
- Sessions of 3–10 turns: real, in the 64–87% range. Worth doing. Don't quote 96%.
- Sessions of 10+ turns: the full effect, ~90%+. This is where it pays for itself.
- Regardless of session length: prefer the append-only architecture (C) over the header move (B). B's advantage decays, and the extra work to freeze the system message is a one-time cost.
Caveats
I'd rather the numbers hold up than look good:
- One provider. These are DeepSeek's numbers. The mechanism should generalise to any prefix-based cache, but I have not measured Anthropic's, which uses explicit breakpoints plus a TTL rather than pure prefix matching. That is a real gap — if you have data there, I would like to see it.
- Best-effort caching. No provider guarantees a 100% hit rate, and repeated runs will vary.
- History compression changes the picture. A real app summarises or truncates history instead of letting it grow. That would cap the divergence between B and C, so the 30-turn numbers are the pessimistic case for C's advantage, not the optimistic one.
- The stable block is synthetic, though its structure and scale are realistic. This measures cache behaviour, not model quality.
- Off-peak pricing throughout. Peak is twice off-peak.
Reproducing it
The benchmark is small and self-contained:
node bench.js --self-test --dry-run # proves the three variants differ, costs nothing
node bench.js --turns 30 --budget 1.00
--self-test diffs the serialised request bodies of turn 1 and turn 2 and reports how many bytes are identical. For variant A it is 75 bytes out of 84,665. That is the whole story in one number.
Full per-turn data for both runs, and the session-length analysis, are in the repository:
https://github.com/chenyu520-ai/rp-cache-lab
Thanks to Max Quimby, Tom Veber, IndianInfraNotes and Rulestack for the comments that turned this into a second measurement rather than a single headline. If your sessions are shorter or longer than the ones here, I would like to know what you see.
Top comments (23)
The fixed 16,896-token hit count for B explains the divergence more clearly than the percentage alone. One additional variable for the session-length analysis is the time between turns: thirty turns sent together and thirty turns spread across a day need not have the same prefix reuse in a best-effort cache.
I would stratify the next run by both length and inter-turn delay, then compute the saving from total spend across the observed session mix rather than averaging each session's percentage. That would preserve the useful B/C comparison while showing how much the append-only design delivers under actual user pacing and cache eviction.
Ran it. Four independent probes, each priming its own prefix — every prime came
back as a full miss, which is what confirms no probe was warmed by an earlier
one.
immediate re-send 99.1% hit
2 minutes 99.1%
15 minutes 99.1%
1 hour 99.1%
3 hours 99.1%
So within three hours eviction isn't the variable that matters; the prefix
survives intact. The whole sweep cost about a cent.
Two caveats I'd rather state than have you find:
Three hours is not the documented ceiling — DeepSeek says entries are cleared
"usually within a few hours to a few days", so the interesting region is past
where I stopped. A cliff may well exist; I just haven't reached it.
And my four probes ended up sharing almost identical body content because of a
seeding bug (I passed a string where the generator wanted a number, so every
probe produced the same text and only the leading marker differed). It doesn't
change the result — a cold prime each time is what establishes independence —
but I'd fix and re-run before treating the numbers as final.
Your point about total spend across the observed mix still stands. If sessions
are spread over days rather than hours, this says nothing about that regime.
The cold prime for every prefix is a useful control, and I agree the result only bounds the interval you actually tested. Thanks for flagging the seeding bug rather than presenting the sweep as final.
For the rerun, I would log a hash of each generated body alongside its seed and leading marker. That makes unintended duplicate bodies visible before the requests are sent. Repeating a few prefixes within each delay would also show whether the observed survival is consistent across requests, without turning the sweep into a claim about real traffic.
Hashing each generated body alongside its seed and leading marker is the fix I should
have had from the start — the duplicate bodies were visible in the output and I didn't
look at it that way until the numbers came back suspiciously identical.
Repeating a few prefixes within each delay is a good addition too. One request per
condition can't distinguish "the cache survives" from "that request happened to hit".
The duplicate-body check is worth making an assertion before any requests are sent. Distinct seeds alone would not catch a generator that silently collapses them, whereas distinct body hashes would expose exactly the issue you found.
For the rerun, separate independent prefixes from repeated requests within each prefix. Reporting both counts prevents a large number of repetitions on one cached body from looking like broad evidence across bodies. The cold-prime outcomes can then remain a control rather than being folded into the warm-hit estimate.
The body-hash assertion is the right shape: distinct seeds are an intention, distinct
hashes are an observation, and only the second one fails loudly. I'll assert it before
the first request rather than checking afterwards.
Separating independent prefixes from repeated requests within a prefix is also right,
and it fixes a reporting problem I'd have walked into — one prefix hit thirty times
would have looked like thirty observations.
The pre-request hash assertion also gives the run a useful identity before any provider response exists. I would store the ordered prefix hashes with the seed and configuration, then keep repeats nested under that identity in the result file.
One remaining check is that the hashed bytes are the bytes actually sent after message serialization. Hashing a template before a wrapper adds a timestamp or reorders fields could prove distinct inputs at the generator layer while measuring different cache keys at the provider boundary.
The serialization point is the one that matters, and it's easy to get wrong in the
direction you describe. Hashing the template proves the generator produced distinct
inputs; hashing what actually goes on the wire proves the provider saw distinct cache
keys. Those are different claims and only the second one is the one the benchmark is
about.
Storing the ordered prefix hashes as the run's identity, with repeats nested under it,
also fixes the reporting problem — repeats become a property of a prefix rather than
independent observations.
The ordered hashes make repeats correctly nested under their prefixes. One qualification I would retain is that a wire-payload hash proves serialization identity; it does not itself reveal the provider's internal cache key or which prefix length was eligible.
Keeping observed cached-token usage with each request would connect the input identity to the measured behavior. That way the report can distinguish distinct serialized prefixes, repeated attempts and actual cache hits without treating the hash as evidence of an undocumented provider implementation.
You're right, and it's a bigger caveat than the post gives it credit for.
The 30-turn run compresses time — turns are seconds apart, so nothing ever
expires. A real session can have the same turn count spread across hours, and
DeepSeek documents the cache as best-effort with entries cleared "usually
within a few hours to a few days". If the cache is gone between turns, every
variant pays the cold price and the saving goes to zero for all three, not just
for B.
So the curve is an upper bound for an active session and says nothing about
paced usage. That's a gap in the post rather than a nuance.
Your stratification proposal is the right experiment. The percentage-average is
precisely the statistic that hides the eviction effect, because it weights a
1-turn session the same as a 30-turn one — total spend across the observed mix
is what a real bill would show.
I'll run it with a few inter-turn delays and report back. If you have a sense of
what gaps dominate in your workload, that would help pick which delays are
worth measuring.
I do not have measured inter-turn gaps from a workload to contribute, so I would treat the delays as an exploratory sweep rather than representative traffic. Comparing continuous activity, a short pause, and a pause across hours should expose whether the apparent benefit depends on uninterrupted use.
It would also help to repeat each delayed condition independently instead of continuing one increasingly old cache. Recording actual hit tokens on every request would then distinguish eviction from the prefix-layout differences in the original experiment.
Agreed on all three, and the run is already going with that shape.
Each delay gets its own freshly primed prefix rather than continuing one
ageing cache, precisely so eviction isn't confounded with how long a single
prefix has been alive. Conditions: an immediate re-send, then 2 minutes,
15 minutes, 1 hour, 3 hours. Cache-hit tokens are logged on every request
including the priming one, so a condition that comes back cold is
distinguishable from one that was never warm.
Treating it as an exploratory sweep rather than representative traffic is the
right framing — I don't have a workload to draw gaps from either, so the output
is a shape rather than a number. If any condition shows a cliff, that at least
bounds where the benefit stops being safe to assume.
Logging the prime as well as the later request makes the cold-versus-expired distinction observable. Your later results on the other branch are useful evidence for this exploratory setup, with the seeding caveat kept explicit.
For reproducibility, I would retain the delay scheduled and the delay actually elapsed between completion of the prime and submission of the probe. That makes it possible to tell a cache survival result from a scheduling deviation when the longer-gap conditions are repeated.
Keeping both the scheduled and the actual elapsed delay is a good call — I only logged
the scheduled one. When the longer gaps get repeated, I won't be able to tell a survival
result from a scheduling slip, which is exactly the failure mode you're describing.
I'll add it to the rerun.
Including actual elapsed time would also let you group results by the delay that really occurred. A request scheduled for three hours but executed much later should contribute to the observed later interval, with the scheduling deviation still visible.
I would keep a monotonic elapsed measurement for the delay and wall-clock timestamps for auditability. That avoids a clock adjustment changing the apparent survival time, while preserving enough context to interpret queueing or runner pauses in the longer probes.
Monotonic for the delay, wall-clock for the audit trail. That's the split I want too,
and it keeps a clock adjustment from silently rewriting the result while still leaving
enough context to explain a queueing pause in the long probes.
Grouping by the delay that actually occurred rather than the one I scheduled is the
part I'd otherwise have got wrong — a three-hour probe that ran at four hours is
evidence about four hours, not three.
The four-hour probe example is exactly why the actual delay belongs in the result. I would preserve both the intended delay and the monotonic start-to-start delay, rather than replacing one with the other.
Their difference becomes a scheduling diagnostic: a cache miss after four hours answers the cache question, while the extra hour explains whether the runner met its experimental plan. Recording request duration separately also avoids calling a three-hour sleep a three-hour probe when the preceding request itself occupied several minutes.
Keeping both rather than replacing one with the other is right, and the scheduling
diagnostic framing is the useful part — a miss after four hours answers the cache
question, and the extra hour answers a different one about whether the runner kept to
the plan.
Recording request duration separately matters more than it looks. My first sweep called
a three-hour gap a three-hour probe while the priming request itself took time, so
"three hours" was really "three hours plus whatever the prime cost". That's exactly the
kind of thing that makes two runs disagree for reasons that have nothing to do with the
cache.
The prime duration is exactly the missing term. I would preserve prime start, prime completion and probe start from the same monotonic clock, then name the reported gap by its endpoints. That removes ambiguity about whether provider request time is included.
A probe delayed by runner scheduling should keep its intended three-hour label as plan metadata, but its actual elapsed gap should drive the cache-age comparison. Showing both makes a cache miss interpretable without asking readers to reconstruct the runner's timing from separate logs.
For the Anthropic gap in your caveats, one data point from the agent benchmark of an MCP server I maintain, where the tool definitions and system prompt are identical on every turn. On gpt-5.4-mini, 48% to 90% of prompt tokens were served from cache depending on the run, with no setup. On Claude Haiku 4.5 and Sonnet 5 the same harness reported 0 cached tokens across 144 runs (about 950,000 prompt tokens), because it never set a cache_control breakpoint. So on Anthropic, layout C is necessary but not enough. Two details for your delay sweep: the default cache lifetime there is 5 minutes (1 hour at a higher write price), so 15 minutes and up would come back cold; and tools render before the system prompt, so a tool list that changes between turns acts like your variant A header. Run records: github.com/arhancanli/canlicapital...
This is the most useful thing anyone has added to either post. Three things in
it I did not know and could not have measured.
The Anthropic zero is the striking part. 144 runs and 0 cached tokens because no
cache_control breakpoint was set — that means on Anthropic the failure mode
isn't "the prefix moved", it's "nobody ever asked for the cache". Layout C is
necessary but not sufficient, exactly as you put it. That's a different bug from
the one I wrote about, and probably a more common one.
The 5-minute default lifetime also reframes my own sweep. I measured 2 minutes
through 3 hours on DeepSeek and got 99.1% at every interval, which now looks
like a provider-specific result rather than a general one.
And "tools render before the system prompt" settles a question PromptAlo raised
in the other thread. If the tool list changes between turns there is nothing
ahead of it to preserve, so it's worse than the header rather than equivalent.
Thanks for the run records link — reading them now. If the harness is public I'd
like to run the same questions against DeepSeek to see whether the entry-point
difference survives a provider change.
It's public: github.com/arhancanli/canlicapital... (run.mjs, and tasks.mjs with the ground truth for each task). Two notes for a DeepSeek run. run.mjs calls the OpenAI and Anthropic endpoints directly, so you'd point the OpenAI path at DeepSeek's OpenAI-compatible base URL. And the cached count comes from each provider's usage field, which isn't named the same everywhere, so check where DeepSeek reports cache hits before comparing numbers across providers. I'd like to see what you get.
Thanks for making it public — I've got the repo open. Two things in your note I'd have
got wrong on my own: the usage field naming, and pointing the OpenAI path at DeepSeek's
compatible base URL rather than rewriting the call.
I'll check where DeepSeek reports cache hits before I trust any cross-provider
comparison, and I'll post the numbers here either way — including if they don't
reproduce.
One thing I should say before running it: my own sweep had a seeding bug that produced
identical bodies across conditions, and Ahmet pointed out I should be hashing the
generated bodies rather than trusting the seed. So if the DeepSeek run disagrees with
your OpenAI and Anthropic numbers, the bug is at least as likely to be mine as a real
provider difference.