The Silent 10 Tax: How a Nondeterministic System Prompt Voids Your LLM Prompt Cache

Originally published on hexisteme notes.

Prompt caching is the largest single token lever for any long-lived LLM session — a cached token is served at roughly a tenth of its normal price. It is also one of the easiest things to break by accident, because when it breaks there is no error. One timestamp in the system prompt, one unsorted JSON key, and your cache-read rate quietly goes to zero while the bill goes up tenfold. This is how the failure works, and how to make it loud.

A prompt cache is a prefix match: the cache key is the exact bytes of the rendered prompt up to a breakpoint, and a single byte change anywhere in the prefix invalidates everything after it. So anything volatile near the front of the prompt — a datetime.now() in the system header, a per-request UUID, a dict serialized without sorted keys, a tool list that reorders — silently defeats caching for the entire request. The cost is real (a miss is ~10× a hit on the cached span) and invisible (no exception, no warning). The only signal is one field in the usage object, and the only durable fix is to assert on it.

How prefix caching actually works

The mental model that prevents every bug below is one sentence: the cache key is the bytes of the rendered prompt, matched as a prefix. The provider renders your request in a fixed order — tools, then the system prompt, then the messages — hashes the bytes up to each cache breakpoint, and looks for a prior entry with the same prefix. If it finds one, the shared span is a cache read; the rest is processed fresh. If the very first byte differs, nothing matches and the whole thing is processed at full price.

Three properties follow, and all three are load-bearing:

• Prefix, not fuzzy match. There is no "close enough." Byte N changing invalidates the cache for every breakpoint at position ≥ N. Stable content has to physically come before volatile content in the rendered prompt, or the volatile content poisons everything after it.
• Model-scoped. Caches are keyed per model. Switching the model mid-session — even to a cheaper one for a sub-task — starts from a cold cache.
• Short-lived. On Anthropic's API the default cache entry lives ~5 minutes (a 1-hour tier exists at a higher write premium). A gap longer than the TTL means the next request pays to write the cache again.

The economics are what make this worth caring about. A cache read costs about 0.1× the base input price; a cache write costs about 1.25× for the 5-minute TTL. So the break-even is fast — two requests sharing a prefix already come out ahead — and the downside of a silent miss is steep: every token you could have read at one-tenth price is instead reprocessed at full price. That is the 10× in the title, and on a long agent loop that resends a large fixed preamble every turn, it is most of the bill.

The silent failure: cache_read stays zero and nothing tells you

Here is what makes this class of bug nasty. A broken cache is not an error. Your requests succeed, your outputs are correct, your latency is a little worse and your cost is a lot worse, and nothing in the response says "you just paid full price for 40,000 tokens you had cached thirty seconds ago." The system is behaving exactly as designed; it simply never found a matching prefix.

The only signal is the usage object. Every response reports three numbers you have to actually look at:

Field	Meaning
cache_read_input_tokens	Served from cache — you paid ~0.1×
cache_creation_input_tokens	Written to cache — you paid the ~1.25× write premium
input_tokens	Reprocessed at full price — not cached

If cache_read_input_tokens is zero across repeated requests that should share a prefix, a silent invalidator is at work. That single field is the whole diagnosis. When it's zero and you expected a hit, the move is mechanical: capture the fully rendered prompt bytes from two consecutive requests and diff them. The byte that changed is your invalidator, and it is almost always one of a small, known set.

The invalidators, in order of how often they bite

Every one of these changes the prefix bytes without changing the logical prompt — which is exactly why they're easy to write and hard to notice.

• A timestamp or UUID in the system prompt. The classic. f"Current time: {datetime.now()}" or a per-request trace ID interpolated into the system header changes the prefix on every single request, so the cache is never read even once. Anything that must be dynamic belongs after the last breakpoint, in the message list — a fact injected at turn 5 invalidates nothing before turn 5.
• Non-deterministic serialization. Building any part of the prefix with json.dumps(d) without sort_keys=True, or by iterating a set, can emit different bytes run to run for identical content. Same logical prompt, different key order, cache miss. Sort your keys and order your collections.
• Tool-set churn. Tools render at position zero, ahead of everything. Adding, removing, or reordering a tool changes the very front of the prefix and invalidates the entire cache below it — across the whole session, and across users if the tool set is built per-user. Freeze the tool list and sort it deterministically; for genuinely dynamic tools, use a discovery mechanism that appends schemas rather than swapping the base set.
• Per-user or per-session interpolation into the system prompt. Splicing a user ID, name, or session key into the system header gives every user a distinct prefix and kills all cross-request and cross-user sharing. Move it into the messages.
• Conditional system sections. if flag: system += "..." means every combination of flags is a different prefix. Each variant caches separately and shares nothing.
• Mid-session edits and model switches. Editing the top-level system prompt partway through a conversation re-processes the whole history uncached; switching models throws the cache away entirely (it's model-scoped). If you need to inject an operator instruction mid-run, append it as a message rather than editing the frozen system prompt.

## Why it hurts most exactly where you'd want caching

The cruel part is that the failure concentrates in precisely the workloads caching is for. A one-shot classification call barely cares. A long-running agent — a fleet of workers, a multi-step DAG, an orchestration loop that resends a large fixed system prompt and tool set on every one of hundreds of turns — is entirely built on the assumption that the big fixed prefix is nearly free after the first write. When a worker reassembles its system prompt on each call and one field in that assembly is nondeterministic, that assumption silently inverts: the largest, most-reused span of the prompt becomes the most-repaid. The bigger and more disciplined your prompt architecture, the more a single stray byte costs you.

The audit: make the cache assert itself

Because the failure is silent, the only real defense is to stop trusting and start measuring. The audit is short:

• Classify every input by how often it changes. Never → put it early, before any breakpoint. Per-session → after the global prefix. Per-turn → at the very end. Per-request (timestamps, IDs) → eliminate it from the prefix or move it dead last.
• Check that rendered order matches stability order. Stable bytes must physically precede volatile bytes. A timestamp in the system header defeats every breakpoint after it no matter how many cache_control markers you sprinkle downstream.
• Freeze the load-bearing pieces. System prompt frozen, tools deterministically ordered, model fixed for the session, JSON serialized with sorted keys.
• Then assert. After the first request warms the cache, the code should check usage.cache_read_input_tokens > 0 on the second — and fail loudly if it isn't. A silent cache miss caught by an assertion in development is a non-event; the same miss shipped to a long-running fleet is a standing tax you won't see until you read the bill.

None of this is exotic. It's the same discipline as any other cache: know your key, keep it stable, and verify your hit rate instead of assuming it. The only twist with prompt caching is that the key is the entire rendered prefix, so "keep it stable" reaches all the way back into how you build the system prompt — and the penalty for a stray byte is measured in tens of percent of your token bill, paid quietly, forever, until someone looks at the one field that would have told them.

---

More notes at hexisteme.github.io/notes.