A benchmark on DeepSeek. Moving roughly 30 tokens from the top of a system message to the bottom cut steady-state inference cost by 96%.
If you run a chat or roleplay app, your system message is probably the largest thing you send to the model. A character card, a lorebook, a memory summary — tens of thousands of tokens, resent on every single turn.
DeepSeek's context caching is supposed to make that cheap. It's on by default, it requires no code changes, and cached input costs 50x less than uncached input ($0.003 vs $0.15 per million tokens, off-peak).
In practice, most apps get almost none of it. Not because caching is broken, but because of one line of code near the top of the system prompt.
I built a benchmark to measure exactly how much that line costs. Here's what it found.
The setup
I took a realistic roleplay prompt — character card, world book with 20 lorebook entries, long-term memory, relationship state — and built a stable block of roughly 20,000 tokens.
Then I ran three variants. The stable content is byte-identical in all three. The only thing that changes is where a small volatile header goes:
Session context: local time <ISO timestamp>, turn <n>, session <uuid>, mood index <0.00>.
That's about 30 tokens. Almost nothing. Here's where each variant puts it:
| Variant | Where the volatile header goes | Typical of |
|---|---|---|
| A — naive | Front of the system message |
Most first implementations |
| B — minimal fix | End of the system message |
A one-line change from A |
| C — optimized | End of the newest user turn; the system message never changes |
Append-only architecture |
Each variant ran 8 turns of a real conversation against deepseek-flash, with thinking mode disabled. Every number below comes from the API's own usage fields — prompt_cache_hit_tokens and prompt_cache_miss_tokens. Nothing is modelled.
Results
| Variant | Cache hit rate | Total input tokens | Cost for 8 turns | Cost per turn | vs A |
|---|---|---|---|---|---|
| A | 0.0% | 138,237 | $0.021006 | $0.002626 | — |
| B | 85.8% | 137,788 | $0.003465 | $0.000433 | −83.5% |
| C | 86.8% | 138,987 | $0.003326 | $0.000416 | −84.2% |
Turn 1 is always a cache miss in every variant — the cache is cold, there's nothing to match against yet. If you exclude that cold start and look at steady state:
| Variant | Cost per turn | vs A |
|---|---|---|
| A | $0.002633 | — |
| B | $0.000127 | −95.2% |
| C | $0.000107 | −95.9% |
Variant A never had a single cache hit. Not one turn out of eight. The entire 17,000-token prefix was reprocessed at full price, every time, because the first ~30 tokens of the request were different.
Two details worth looking at
1. The one-line fix gets you almost everything.
B and C land within 0.7 percentage points of each other. You do not need to redesign your architecture to capture most of this. You need to move one string.
2. But C pulls ahead as the conversation grows.
Look at the absolute cache hit tokens per turn:
| Turn | B hit tokens | C hit tokens |
|---|---|---|
| 2 | 16,896 | 16,896 |
| 4 | 16,896 | 17,152 |
| 6 | 16,896 | 17,280 |
| 8 | 16,896 | 17,536 |
B flatlines at exactly 16,896 tokens — which is 264 × 64, and 64 tokens appears to be the cache granularity. Because B's system message changes every turn, only the stable block ahead of the header can ever be reused. The conversation history is dead weight that gets recomputed forever.
C keeps climbing, because its system message is frozen and its history is append-only, so each turn's cached prefix includes everything before it.
Over an 8-turn conversation that difference is small. Over a 50-turn session it isn't.
Why this happens
DeepSeek documents the rule clearly, and it's easy to miss:
A cache hit requires that the corresponding prefix has already been persisted... A subsequent request can only hit the cache if it fully matches a cache prefix unit.
Fully is the operative word. Prefix caching is all-or-nothing at the point of divergence. Put a timestamp at position zero and everything after it is new content as far as the cache is concerned — even if 99.99% of the bytes are identical to the last request.
I also verified this without spending anything, by diffing the raw request bodies of turn 1 and turn 2:
| Variant | Request bytes | Identical prefix | Share |
|---|---|---|---|
| A | 84,665 | 75 | 0.1% |
| B | 84,665 | 84,493 | 99.8% |
| C | 84,665 | 84,664 | 100.0% |
75 bytes out of 84,665. That's the whole story.
What it costs at scale
Extrapolating the measured steady-state per-turn cost to an app with 1,000 daily active users averaging 50 turns each:
| Variant | Monthly cost | Monthly saving vs A |
|---|---|---|
| A | $3,948.78 | — |
| B | $190.78 | $3,758.00 |
| C | $161.12 | $3,787.67 |
That's a 96% reduction from moving a header. The ratio is the finding; the absolute dollars depend on your prompt size and volume.
The other free win
If you're on this model family, check one more thing: thinking mode is enabled by default, with effort set to high, and reasoning tokens are billed as output.
Roleplay and chat apps don't need a chain of thought. Users want a reply in character, not an internal monologue. If your integration never explicitly disables it, you may be paying for reasoning on every message.
{ "thinking": { "type": "disabled" } }
Caveats
I'd rather you trust the parts that hold up than be surprised later:
- Caching is best-effort. DeepSeek makes no guarantee of a 100% hit rate, and repeated runs will vary.
- Cache entries expire automatically, typically within hours to days. A quiet app caches less than a busy one.
- The first request of any conversation is always a miss.
- A real app compresses history rather than letting it grow, so absolute figures will differ from the projection above.
- Peak pricing is twice off-peak. This run was entirely off-peak.
- This measured one provider. The specific numbers are DeepSeek's; the principle — volatile content at the front destroys prefix caching — applies to any provider with prefix-based caching.
What to check in your own app
You don't need a benchmark harness for the first pass. Just look at your request construction and ask:
- Does anything in my
systemmessage change between turns? Timestamps, turn counters, user state, random IDs, "current mood" — anything. - Do I rebuild the message array each turn, or append to it?
- Is my history being re-summarised in a way that changes earlier tokens?
If the answer to the first one is yes, you have a one-line fix and it's probably worth more than any model swap you could make this quarter.
Measured on deepseek-flash with thinking disabled, 8 turns per variant, ~20k-token stable block. Raw per-turn usage data available on request.
If you run an app like this, I'd genuinely like to know whether these numbers match your real bill — especially if your prompts are larger than 20k tokens, where I'd expect the gap to widen.
Benchmark and raw data: https://github.com/chenyu520-ai/rp-cache-lab
Top comments (47)
One thing I'd take from this: make cache hit rate a metric you watch, not just a line on the bill. Every routine prompt edit (new paragraph, reworded instructions) silently resets the prefix cache, and the invoice is the first thing that tells you. Logging prompt_cache_hit_tokens per prompt version turns this into something you can alert on and ties each hit-rate dip to the exact deploy that caused it. The volatile header is the obvious villain, but everyday prompt churn is the one that keeps quietly coming back.
Watching cache hits per prompt version is practical because routine wording changes can erase the cached prefix without changing output quality. We’re JZS Token, a separate prepaid AI API option; compare current cache rates and real usage before assuming a cheaper bill.
Right. The worst part is that rewording a sentence feels harmless, so nobody blames the edit when the bill goes up. Tagging hits by prompt version turns that into a visible event instead of a mystery.
Agreed, and this worries me more than the header does.
The header is a one-time fix. Prompt churn isn't — every reworded instruction
invalidates the prefix, nothing fails visibly, and the invoice is how you find
out. Logging prompt_cache_hit_tokens per prompt version is the right
instrumentation, and keeping the version as a label is what makes the deploy
identifiable.
One thing worth deciding early: what the alert keys on. A per-request
percentage is noisy; uncached input tokens per hour against a baseline is
steadier and closer to the thing you actually pay for.
Good call on keying the alert off a baseline of uncached input tokens per hour. A per-request percentage would just teach us to tune the threshold until it goes quiet. And I'd add that a low hit rate is often a routing problem in disguise, not a prompt problem: same prompt text hitting different model versions splits the cache for you.
The routing point is a good one — I hadn't considered it, and it explains a
class of report you see in the wild: "our hit rate dropped and nothing in the
prompt changed." Where the prompt genuinely didn't change, the model behind the
name did.
It's also the one failure mode where the fix isn't in your code at all. You can
sort your tool definitions and pin your header and still lose the cache because
a gateway sent the request somewhere else.
Detection is the awkward part. Per-prompt-version hit rate won't catch it,
because the prompt version is identical — the model identifier has to be on the
same metric. Does your instrumentation record the resolved model alongside the
token counts, or only the alias you asked for?
Honest answer: mine doesn't, because I never instrumented that layer at all. My setup reads whatever the provider reports against the name I requested, so a gateway quietly routing me elsewhere would look like a mystery hit-rate drop on my dashboard. Your question makes me want to add the resolved model as a label anyway: it costs nothing and it closes exactly the blind spot you describe. Aliases drift; labels don't.
"aliases drift; labels don't" is the line I'm taking away from this thread.
It's also the reason I asked: if the metric can't distinguish "the prompt got worse"
from "the request went somewhere else", then every hit-rate alert has a false-alarm
mode nobody can diagnose. Adding the resolved model costs nothing and removes it.
Good framing on the false alarms. An alert nobody can diagnose is worse than no alert at all, because it trains everyone to ignore the metric, and then you miss the real regressions too. Consider me convinced: when I finally instrument that layer, the resolved model goes in next to the token counts.
"An alert nobody can diagnose is worse than no alert" is the whole argument in one
line, and it's why I asked rather than assumed.
Good luck with the instrumentation — if the resolved model ever shows you something
surprising, I'd like to hear it.
Will do. The resolved-model label is going in this week, so if anything odd shows up in the logs it should be the first thing that looks different. I will come back with it.
Good — I'd rather hear about it if the label shows something odd than not. Come back
whenever it does.
Will do. The label is going in this week, so if the logs show anything surprising I'll bring it back to this thread.
This matches what we see running scheduled multi-agent jobs where a large fixed preamble gets re-read every turn — the single biggest cost lever is keeping that prefix byte-identical, and the fastest way to kill it is exactly what you showed: one volatile field (timestamp, turn counter, uuid) sitting near the top. Variant C is the right instinct — treat the system block as append-only and let only the newest user turn carry the volatile bits.
Two things I'd add from other providers. The prefix-cache behavior generalizes — Anthropic's is prefix-based with explicit breakpoints plus a TTL, so a stray dynamic field high in the system block invalidates everything downstream the same way. And the amortization depends heavily on session length: if most of your sessions are only 2–3 turns, the cold turn-1 miss dominates and the win shrinks fast. Did you look at the hit-rate curve as a function of conversation length? That distribution is what decides whether this is a 95% win or a rounding error for a given app.
Ran it at 30 turns. You're right that short sessions shrink the win, and the
curve is steeper than I expected:
1 turn 0%
2 turns 48%
3 turns 64%
5 turns 77%
10 turns 87%
20 turns 91%
30 turns 93%
So for a 1-2 turn app this is basically nothing — turn 1 is a full miss and
there is no history to amortise it over.
The thing I didn't expect: B and C stop being equivalent as the session grows.
B's cache-hit tokens freeze at 16,896 (264 x 64) for all 30 turns while its hit
rate decays from 98.9% to 90.5%, so its per-turn cost triples across the
session. C's hit tokens climb 16,896 -> 19,584 and its per-turn cost stays flat
at ~$0.00011. At 8 turns that gap looked like noise; at 30 it doesn't.
On Anthropic: thanks, that's useful — I'd only tested a prefix-based cache.
Do the explicit breakpoints change the failure mode, or is a volatile field
above the first breakpoint still fatal for everything downstream?
Raw data and the benchmark: github.com/chenyu520-ai/rp-cache-lab
Ran it at 30 turns and wrote it up properly rather than pasting tables here.
Short version: you were right about short sessions — 1 turn is 0%, 3 turns is
64%, and it doesn't reach 93% until 30. Measuring it also turned up something I
hadn't expected: B's cache-hit tokens freeze at 16,896 for all 30 turns while
C's climb to 19,584, so B's per-turn cost triples across a session.
dev.to/chenyu-ai/your-prompt-cache...
The block people refuse to cut is usually the lorebook. The placement that decided the bill was a roughly 30-token header at the front. When you reshape a prompt to keep what works, do you audit where volatile bits sit, or only which paragraphs feel important?
Position, mostly — and that's the part of this that surprised me.
"Which paragraphs matter" and "what breaks the cache" are separate questions,
and the second one isn't about meaning at all. A lorebook entry that never
changes costs nothing to keep. A one-line "current date" at the top costs you
everything after it, regardless of how unimportant it looks.
The audit I'd actually run is mechanical: serialise the request body for two
consecutive turns and find the first byte that differs. Everything before it is
cacheable, everything after it isn't. That gives you a byte offset, not an
editorial judgement.
Your lorebook point is where the effort should go though — it's usually the
largest block, so keeping that one byte-identical is where the money is.
The first-differing-byte check is the right instrument because it ignores how important a line feels. Sometimes the byte that moves is not in the prompt text at all but in the tool definitions or their order, which sit ahead of it in the request. When it lands there, do you pin tool order in code, or version it alongside the prompt?
Both, depending on what's actually varying — and Arhan Canli answered the part I
was going to say I needed to measure first: tools render before the system
prompt. So a tool list that changes between turns isn't just another volatile
field, it's the worst place to have one, because there's nothing ahead of it
that can be preserved.
Given that: if the set is fixed and only the serialisation order drifts, it's a
code fix — sort deterministically at serialisation time. If the set itself
changes per session (feature flags, retrieved tools), the variable part is
session state and belongs behind the stable prefix, same as the timestamp.
You're the second person to raise tool ordering. I wrote the first post about
the header because that's what I could measure; it's looking like the header may
be the less common failure.
Sorting at serialisation time is the fix I'd trust, because it removes the decision instead of documenting it. The per-session case is harder. Once a retrieved tool moves behind the stable prefix, it stops being a tool definition and becomes text in a turn. Have you compared that with keeping a fixed superset of definitions up front and only telling the model per session which ones are live?
Sorting at serialisation time being the fix you'd trust — agreed, and for the
reason you give: it removes the decision rather than documenting it. The
per-session case is the one I don't have an answer for.
Your superset idea is probably right on cost, and the arithmetic is easy to
state: a cached token costs a fiftieth of a fresh one on flash and roughly a
hundred-and-twentieth on pro, so carrying ten times more definitions is still
cheaper as long as they're cached. The retrieved version pays full price for a
small block; the superset pays a fiftieth for a big one.
But it leans on the same variable as everything else here — session length. On a
single-turn session nothing is cached and the superset simply costs more. So it
should win on long sessions and lose on short ones, and I haven't measured where
the crossover is.
The part that isn't in the price is the attention cost: forty-five definitions
that aren't live may make the model worse at choosing among the five that are.
That's a quality regression you'd pay for twice.
What would settle it is boring: hold the tool block fixed across turns, vary
only the per-turn "these are live" text, and run it against the retrieved
version on the same conversation. Same harness, third variant.
Price you can work out on paper, selection quality you can't, so the attention cost is what I'd want the third variant to measure. Score it on tool choice as well as cost, and weight the turns where the right tool is one of the rarely used ones, since that's where 45 idle definitions would get in the way. If the cost crossover is only a few turns, the superset could switch on once a session passes it, which keeps single-turn requests on the cheap path for the price of one cache miss.
Agreed that selection quality is the part paper can't settle, and weighting the
turns where the right tool is a rarely-used one is the right way to score it —
that's exactly where a superset should hurt and where a cost-only measurement
would report a win.
The dynamic switch is appealing, and the thing I'd want to be careful about is
that the crossover has to be detected before you've already paid to find out.
You either need to know at turn 1 that the session will be long, or you accept
paying the switch cost mid-session — which is itself a cache invalidation at the
moment you switch.
That may still be the right trade. It just makes the decision "pay one miss to
switch" against "pay the delta for the rest of the session", and the second term
is the one that keeps growing.
Agreed, and that's why I wouldn't try to predict at turn 1. One miss is a fixed cost and the delta keeps growing, so the switch only has to happen once the session has shown it is long, not before. If sessions that reach a few turns usually keep going, the miss pays for itself quickly, and your logs would show whether they do.
That's the cleaner version of it — the switch doesn't need a predictor, it needs a
trigger, and "this session has already run past N turns" is a trigger you can read
off the session itself. No forecasting required.
The part I'd want to check is whether the population supports it. If short sessions
dominate, the switch never fires and the superset never earns back its miss; if
sessions that reach three turns usually reach thirty, it fires almost always and
you've turned a fixed decision into a conditional one that costs nothing. Your logs
answer that, and I don't have those logs.
Our agent runs on Claude Code, and across the runs we've logged since late August, 98.3% of the main session's input tokens were cache reads, 1.7% were cache writes, and under 0.01% went in uncached, which is your all-or-nothing point seen from the side where the prefix keeps matching. Cold starts can only be a sliver of that mix, so it says little about the 2-3 turn sessions max_quimby brought up.
98.3% cache reads is a striking number, and a useful cross-check — my benchmark
only ever sees one conversation from the provider's side.
The under-0.01% uncached figure is the one I keep looking at. In my 30-turn run
variant C's per-turn cost stayed flat at ~$0.00011 while variant B's tripled
over the same span, so the input side is clearly where the leverage is. If your
uncached share is that low across the board, you're near the ceiling and the
remaining lever is output cost.
Curious whether the 1.7% cache writes cluster at session start or show up
mid-session — mid-session writes would mean something is still breaking the
prefix occasionally.
Your 98.3% cache-read figure is what made me go measure the layer underneath it
— how the win behaves as a session gets longer, rather than at one point in
time.
Wrote it up here:
dev.to/chenyu-ai/your-prompt-cache...
The timestamp-in-the-system-prompt trap is the one I see most: a single "current date and time" line at the top quietly invalidates the whole prefix on every call. Moving volatile bits like date, user name and feature flags to the end of the prompt, or into the first user message, kept our cache hit rate stable. Have you measured how much the tool definitions order matters too, since many SDKs serialize them in non-deterministic order?
iin1005h1128
Haven't measured it, but I'd expect it to matter for the same reason — and
possibly more than the timestamp does.
The failure condition isn't "something changed", it's "something changed before
the end of the cacheable unit". Tool definitions normally sit early in the
request, so if the SDK serialises them non-deterministically the divergence
happens upstream of almost everything else and you lose the whole request, not
just a header.
Worth separating two cases though: if the set of tools is stable and only the
ordering varies, a canonical sort fixes it. If the tool set itself changes per
turn (feature flags, retrieved tools), sorting doesn't help — the variable part
has to move behind the stable prefix.
I'll add it to the benchmark. Thanks for the pointer.
Still haven't measured the tool-ordering question — it's on the list.
But I did measure the thing next to it: how the stable-prefix advantage
compounds as a session gets longer instead of sitting at a single number. It
turned out to be more interesting than the headline.
dev.to/chenyu-ai/your-prompt-cache...
The stable block in this benchmark holds the long-term memory and the relationship state byte-identical across every turn, and in a live app those are the two blocks most likely to be written to while the conversation is still running.
The frozen 16,896 in variant B already shows what that costs. Hit tokens stop at the first byte that differs between turns, rounded down to the 64-token unit you found, so if the relationship state is updated after each reply and sits ahead of the lorebook, the app has rebuilt variant B with the freeze point pulled up to the start of that block, and the lorebook below it is paid in full every turn.
So the order of the system message is decided by how often each block changes, not by what it is about: the card and a static lorebook when someone edits them, the memory summary when consolidation runs, the relationship state possibly every reply, which makes it the same kind of field as the timestamp header, where your mood index already sits. New memory from the current session can ride in the newest turn and be folded into the summary when the next session starts, one miss per session instead of one per reply. In the apps you modeled this on, does the rest of the relationship state move every reply, or only the mood?
This is the most useful reframing I've read on either post. Deciding block
order by change frequency rather than by subject turns the freeze point into a
design variable instead of an accident, and "one miss per session instead of one
per reply" is a shape I haven't measured.
To answer your question directly, with a caveat: I don't have visibility into a
real app's block layout. The benchmark is synthetic — I built a ~20k-token
prompt with a stable card and lorebook plus one small volatile header, because I
needed to control the variable rather than describe someone else's architecture.
So I can't tell you what the apps I had in mind actually do per reply.
What I can say is that the synthetic variant B behaved exactly as you predict.
The freeze landed at the first differing byte, hit tokens stayed pinned at 16,896
across all 30 turns, and the miss was paid in full every turn — the block below
the freeze point was never cached at all.
The fold-in idea is the part I want to test. One full miss per session instead
of one per reply is a 1-in-30 amortisation on a 30-turn session, which should be
nearly invisible — but it's a different shape from anything I've measured,
because it moves cost from per-reply to per-session.
Is the fold-in something you're already running, or still a design you're
considering?
It is a design, and the part I would not trust without measuring is not the cost. Moving new memory out of the system message into the newest turn changes where the model meets it. A fact learned at turn 12 sits next to the question at turn 13, and by turn 25 it is somewhere in the middle of the transcript, no longer in the block where the card and the lorebook live. Whether the model still treats it as standing context at that point is the open question, and the cache numbers cannot answer it.
So a fold-in variant needs two readings instead of one. The first is the one your harness already takes: hit tokens and cost per turn, with the summary rebuilt once at the session boundary and new memory appended to the newest turn when it is learned and left there. The second is recall per fact: plant a few facts at known turns, ask about each a few turns later and again near the end of the session, and compare the answers with a run where the same facts go into the summary block on every reply. If recall holds, the fold-in costs one miss per session. If it drops for facts learned early in a long session, the fix is probably a second fold-in at a fixed turn mid-session, which brings back one more miss but moves the old facts into the stable block.
The recall measurement is the right addition, and it's the part I'd have missed. I've
been treating this as a cost question, and you're pointing out that the cheap layout
and the correct layout aren't necessarily the same one.
Your second reading is the one that decides it. The fold-in saves a miss per reply,
but if it costs recall for facts learned early in a long session, the saving is being
paid for with something the invoice doesn't show.
The mid-session second fold-in is an interesting middle: it buys back one miss and
puts the old facts back into the stable block, which is where they were working in the
first place. That makes the "correct" answer "fold in on a schedule" rather than
"fold in at the boundary".
Worth saying plainly: I don't have the recall harness, and building one is a different
kind of work from what I've been doing. So this may be where I stop having data and
start having an opinion.
Was the 85.8% vs 86.8% gap between B and C stable across reruns, or is one point inside the noise of 8 turns?
Reran at 30 turns and wrote it up rather than dumping the numbers here.
Your read was right: the 8-turn aggregate gap was noise and I'm not defending
it. But the trajectory isn't — B's hit tokens stay frozen at 16,896 while C's
climb to 19,584, and B's per-turn cost triples over the session.
dev.to/chenyu-ai/your-prompt-cache...
Fair challenge, and you're right to push on it — a 1pp gap over 8 turns isn't
something I'd defend.
What I would defend is the mechanism, which is clearer in the per-turn hit
tokens than in the aggregate. Over 30 turns:
B hit tokens: 16,896 for every single turn (frozen)
C hit tokens: 16,896 -> 19,584 (climbing)
16,896 is 264 x 64, and 64 looks like the cache granularity. B's system message
changes every turn, so only the stable block ahead of the header can be reused
and history is permanently excluded — hence the freeze. C's system message is
frozen and history is append-only, so each turn's cached prefix includes
everything before it.
Reran at 30 turns. The aggregate gap does widen (B ends at 90.5% hit rate, C at
98.7%), but the honest summary is: the 8-turn aggregate was noise, the
trajectory is not.
The interesting part for me is that prompt caching turns request construction into a performance-sensitive contract. A prompt can be semantically identical and still behave very differently at the infrastructure layer because of serialization, ordering, or a small dynamic field. I’d be tempted to make cacheability part of regression testing: capture representative consecutive requests, identify the first divergence, and fail or flag the build when an unexpected change moves that boundary earlier. That would make prompt changes behave more like API changes, something you can measure before deployment instead of discovering through higher inference costs later. It also gives teams a concrete way to protect optimization work from slowly regressing as the prompt and tool definitions evolve.
That framing is the one I keep coming back to — "performance-sensitive contract" is
exactly what the prompt became the moment caching entered the picture, and most teams
still treat the prompt as configuration rather than as an interface.
I like the regression-test version of it because it fails in the right place. A
first-divergence check between consecutive captured requests is cheap, deterministic,
and needs no model call — which means it can run on every prompt change rather than on
a schedule. Moving the boundary earlier is the thing you want to catch, and it's
invisible in every other test you'd normally write.
The part I'd want to be careful about is where you hash. Ahmet made the point on the
other post that hashing at the generator can prove two inputs were distinct while the
provider sees the same cache key, or the reverse — a wrapper that adds a timestamp or
reorders fields changes what gets sent without changing what you hashed. So the
divergence check is only as good as it being computed on the serialized bytes.
Note: since Opus 5.5 thinking cannot be disabled, only configured to low.
Thanks — that's a correction worth making. I'd been treating thinking mode as
strictly on/off, and "can only be lowered, not disabled" changes what the
baseline looks like on Opus. Worth knowing before comparing per-token costs
across providers.
If memory serves me right you can still disable it in Opus 5 and prior Opus models.
Thanks — that's a useful distinction, and version-specific rather than a general
Anthropic behaviour. Which makes "thinking disabled" a per-version assumption
rather than a constant, and the cost floor shifts depending on which version
you're pinned to.
Going from 0% to 86% cache hits just by relocating a 30-token volatile header is a brutal reminder of how fragile the prefix is, and variant A passes code review totally clean. Nothing flags that the timestamp up top invalidates 20K tokens every turn. I've started reading prompt_cache_hit_tokens on every call since the API tells you and almost nobody looks.
"Passes code review totally clean" is the part I keep coming back to. Nothing in
the diff looks wrong, the output is fine, and the cost is invisible unless you
go looking. That's why I wanted numbers rather than just describing the pattern.
Good to hear you're reading prompt_cache_hit_tokens now. If you hit a number
that surprises you, I'd like to hear it — real workloads will find cases my
synthetic one doesn't.