Prompt Caching: Why cache_control Writes But Never Reads

I turned on prompt caching for an agent loop that resends a 12K-token system prompt on every turn. Obvious win, right? Input tokens are the whole bill in a tool loop.

The bill went up.

Not a little. Roughly a quarter. And nothing in the logs looked wrong. Every request returned 200. Latency was the same. The only place the truth was written down was the usage object:

cache_creation_input_tokens: 12184
cache_read_input_tokens: 0
input_tokens: 291

Twelve thousand tokens written to cache. Zero read back. Every single request. I paid the write premium 40 times in a row and never once collected. Prompt caching has exactly one invariant, and I had broken it in the most boring way possible.

TL;DR

• Prompt caching is a prefix match. The cache key is the exact bytes of the rendered prompt up to each cache_control breakpoint. One changed byte at position N invalidates every breakpoint at or after N.
• Render order is tools → system → messages. Anything volatile (timestamps, session IDs, the user's actual question) must sit after your last breakpoint, not before it.
• Cache writes cost 1.25× base input (2× for the 1-hour TTL); reads cost about 0.1×. A cache that only ever writes is not a cache, it is a 25% surcharge.
• Ground truth is usage.cache_read_input_tokens. If it is 0 across repeated requests with "identical" prompts, something upstream is rewriting your prefix.
• Two silent killers that produce byte-identical payloads and still miss: the 20-position lookback window in long tool turns, and parallel fan-out, where an entry is only readable after the first response starts streaming.

Why does prompt caching never read from the cache?

Because your cache_control breakpoint is placed after content that changes every request, so each request writes a brand-new entry that nothing will ever match again.

The mental model that fixes this: the cache is not a key-value store keyed on "my system prompt." It is a prefix match over the rendered request. The API renders tools, then system, then messages, hashes the byte stream up to each breakpoint, and looks for an existing entry. First divergence wins. Everything downstream of it is cold.

So all three of these are the same bug:

# 1. The dynamic header
system = f"You are a support agent. Current time: {datetime.now()}.\n\n{PLAYBOOK}"
# PLAYBOOK is 11K stable tokens sitting behind a string that changes every request.

# 2. The nondeterministic serializer
system = "Schema:\n" + json.dumps(schema)   # no sort_keys=True
# Same dict, different key order, different bytes.

# 3. The per-user tool list
tools = build_tools_for(user)   # tools render at position 0
# Nothing caches across users, ever.

My case was #1, wearing a disguise: the timestamp was injected by a helper three call frames away that I had written months earlier for logging.

The pricing turns this from "suboptimal" into "actively worse than doing nothing." Writes are 1.25× base input price, reads are roughly 0.1×. Two requests that share a prefix break even (1.25 + 0.1 = 1.35 versus 2.0 uncached). One request that writes and is never read is 1.25× for nothing. Forty of them is a line item.

How do I tell whether prompt caching is working?

Read three fields off usage and remember that they partition your prompt: input_tokens is only the uncached remainder.

total prompt tokens = input_tokens
                    + cache_creation_input_tokens
                    + cache_read_input_tokens

That last part trips people up in both directions. An agent that ran for an hour showing input_tokens: 4200 is not magically cheap, and it is not broken either. Check the sum.

In a healthy multi-turn loop, the shape looks like this:

• cache_read_input_tokens covers the whole prior prefix and grows turn over turn.
• cache_creation_input_tokens is small, roughly last turn's output plus the newly appended input, because writes only bill the delta past the highest hit.
• input_tokens is just the tail after your last breakpoint.

If cache_creation_input_tokens is instead near the full conversation size on every turn, the prefix is being rewritten upstream.

Make this a standing check, not a one-time look. The expensive failure mode in production is not a bad first implementation. It is a working implementation that regresses six months later, when someone adds a feature flag to the system prompt, and nothing errors, and the bill just drifts. One integration test earns its keep:

r1 = client.messages.create(**payload)
r2 = client.messages.create(**payload)   # byte-identical
assert r2.usage.cache_read_input_tokens > 0, "cache prefix broke"

To localize a break, log several consecutive request bodies and diff adjacent pairs. In a growing conversation the tail legitimately differs, so what you are checking is the overlap: the previous request's prompt should reappear unchanged as a prefix of the next. Strip cache_control markers before diffing, since the moving marker always differs and is not the culprit. The first divergence inside the overlap is your invalidator.

Where should the cache_control breakpoint go?

At the end of the shared portion of the prompt, never at the end of the whole prompt.

This is the mistake that produced my zero-read logs, and it is the same mistake automatic caching makes on your behalf. A top-level cache_control on the request places one breakpoint on the last cacheable block and slides it forward as the conversation grows. Perfect for a chat thread. Wrong for the pattern where a big fixed preamble is followed by a unique per-request question, because the breakpoint lands after the unique tail, so every request writes a distinct entry over bytes nobody will read.

messages = [{"role": "user", "content": [
    {"type": "text", "text": RETRIEVED_DOCS,        # 9K shared tokens
     "cache_control": {"type": "ephemeral"}},       # breakpoint HERE
    {"type": "text", "text": question},             # unique, unmarked, after
]}]

Four more placement rules worth internalizing:

1. Freeze the system prompt. Date, mode, user name and feature flags do not belong at the front of the prefix. On Opus 5 and Opus 4.8 you can append {"role": "system", "content": "..."} inside messages[] instead, which sits after the cached history and leaves it intact. Otherwise put it in a user turn.
2. Serialize tools deterministically and do not add, remove or reorder them mid-conversation. Tools render at position 0, so a reorder is a full rebuild.
3. Caches are model-scoped. Switching models mid-loop for a "cheap" side task forfeits the whole prefix. Give the subagent its own thread instead.
4. Minimum cacheable prefix is model-dependent and not monotonic (512 tokens on the newest models, 1024 on Opus 4.8 and Sonnet 5, 4096 on Opus 4.6 and Haiku 4.5). Below the minimum you get no error, just cache_creation_input_tokens: 0. A 3K-token prompt caches on some models and silently will not on others. Max 4 breakpoints per request.

What makes prompt caching miss on byte-identical requests?

Two mechanisms, and both of them make you doubt your own diff.

The 20-position lookback. Each breakpoint walks backward at most 20 positions looking for a prior entry. A run of consecutive tool_use blocks counts as one position, and so does a run of consecutive tool_result blocks, so heavy parallel tool calling is fine. A long sequential loop that appends more than 20 positions of other content in one turn is not: the next request's breakpoint never finds the previous entry, and you rewrite the whole conversation with a payload that diffs clean. Fix by placing an intermediate breakpoint every ~15 positions.

Parallel fan-out. An entry becomes readable only once the first response begins streaming. Fire 10 identical-prefix requests at once and all 10 pay full price, because none of them can read what the others are still writing. Send one, await the first streamed token, then fire the remaining nine. Same arithmetic applies to multi-agent designs: N workers each assembling a slightly different prompt over the same context write N entries and read none of each other's.

And on TTL: the default 5-minute entry has its timer refreshed for free by every read, measured from the start of the request. A four-minute generation leaves you about a minute for the next request to begin. The 1-hour TTL doubles the write cost, so it only pays off in the 5-to-60-minute gap, and needs at least three requests to break even.

So why does cache_control write but never read?

Because prompt caching is a prefix match, and a write-only cache means your breakpoint has volatile bytes in front of it. Something before the marker changes on every request (a datetime.now() in the system prompt, an unsorted json.dumps, a per-user tool list) or the marker itself sits after per-request content instead of at the end of the shared prefix. The fix is ordering, not more markers: put everything stable first, put the breakpoint at the last stable byte, put everything volatile after it, then verify with usage.cache_read_input_tokens > 0 on a second identical request. If that field stays at zero, no amount of cache_control will save you, and you are paying 1.25× for the privilege.

---

Written by the developer behind Preterview, an interview prep platform.

Comments

[Raknaos]

"A cache that only writes is not a cache, it is a 25% surcharge" belongs printed above every agent dashboard. We run loops that resend a large system prompt every turn, and the only reason we caught a similar silent miss was that cache_read_input_tokens is tracked per request as a metric — everything else looked healthy: 200s, flat latency, a bill that merely crept up.

The parallel fan-out detail is the one I had never internalized: if an entry is only readable once the first response starts streaming, a fan-out that fires N requests at once is structurally a cold miss. Did you find a cheap way to serialize just the first call, or is it an accept-the-write-premium-once-per-burst tradeoff in practice?