Prompt Caching: How One Dynamic Token Kills the 90% Discount

A team I know shipped a Claude agent with a beautifully cached 12K-token system prompt. Their bill barely moved. The cache hit rate, when they finally logged it, was 0%. The culprit was one line near the top of the system prompt: Current time: 2026-07-18T04:55:12Z, regenerated on every request. That single dynamic token turned a 90% discount into full price on 12,000 tokens, every call, forever.

Prompt caching is the highest-leverage cost lever in LLM serving, and it is also the easiest to silently disable. The trap is that nothing errors. You get correct responses at full price and no warning.

TL;DR

Prompt caching is a prefix match, not a fuzzy match. The cache key is the exact token sequence from the start of the prompt up to a breakpoint. One differing token anywhere in that prefix is a full miss for everything after it.
Order matters more than content. Put static content (tools, system prompt, few-shot examples) first and volatile content (timestamps, user turn, retrieved chunks) last, or you invalidate the cache on every request.
Cache reads are ~10% of base input price on Claude, ~50% off on GPT. Writes cost a premium (~25% for Claude's 5-minute TTL), so caching only pays off when the prefix is reused before it expires.
The usual killers: timestamps in the system prompt, non-deterministic JSON serialization of tools, reordering retrieved documents, and per-user IDs placed above shared instructions.
Log cache_read_input_tokens. If it is zero, your cache is broken and your responses will not tell you.

How does prompt caching actually work?

Prompt caching stores the transformer's internal state (the KV cache) for a prefix of tokens so a later request with the same prefix skips prefill for those tokens. During prefill, the model computes key and value vectors for every input token — that is the compute-heavy, quadratic-ish part of serving. If a request begins with a token sequence the server already processed, it reloads those K/V vectors instead of recomputing them.

The critical property: caching is keyed on an exact token prefix. The server matches the longest run of tokens, starting at position 0, that is byte-for-byte identical to something it has already seen. Match length determines savings. There is no semantic similarity, no partial credit for "mostly the same." Token 4,000 can only be a cache hit if tokens 0 through 3,999 all matched first.

This is why the mental model of "caching my big system prompt" is subtly wrong. You are not caching a block of text. You are caching a prefix that happens to start with your system prompt — and that prefix includes everything the tokenizer emitted before it, in order: tool definitions, system prompt, then message history.

Why does one dynamic token break the whole cache?

Because the prefix match stops at the first differing token, and everything after it becomes a miss. If your dynamic timestamp sits at token 40 of a 12,000-token prefix, the match length is 39. The remaining 11,961 tokens are recomputed and re-billed at full input price on every single request.

Position is everything. The same timestamp placed at the end of the prompt — after all static content — costs you a match on only the last few tokens, which you were never going to cache anyway. Same data, same request count, wildly different bill. The dynamic token is not expensive because it changes; it is expensive because of how much static content sits behind it in the sequence.

This is the single most important intuition: a cache-busting token poisons everything downstream of it, and nothing upstream. Move the poison to the bottom.

Where do developers accidentally break the prefix?

The same handful of mistakes account for nearly every dead cache I have seen:

Timestamps and request IDs in the system prompt. Current time, trace_id, session started at — anything regenerated per request. If the model genuinely needs the time, inject it in the last user turn, not the system block.
Non-deterministic tool serialization. Tool/function definitions are usually the very first tokens in the prompt, so any instability here nukes the entire cache. If you build the tools array from a Python dict without sort_keys, or a set, key order can shuffle between processes. The JSON must serialize to identical bytes every time.
Reordered retrieved documents. RAG pipelines that fetch chunks and concatenate them in nondeterministic order (dict iteration, parallel fetch that resolves out of order) change the prefix even when the set of chunks is identical. Sort chunks by a stable key before assembly.
Per-user data above shared instructions. Putting User: Jane Doe, plan: pro at the top of a system prompt that is otherwise shared across all users means every user gets their own cache lineage and no cross-user reuse. Push per-user context down.
Whitespace and formatting drift. Re-templating that changes indentation, trailing newlines, or joins list items differently produces different tokens. Templates must be byte-stable.

The through-line: treat the prompt prefix as an append-only, deterministic byte stream. Anything that mutates its head is a tax.

How do you structure a prompt to maximize cache hits?

Order from most static to most volatile, and place cache breakpoints at the boundaries. On Claude you mark breakpoints explicitly with cache_control; the cache covers everything from the start of the prompt up to and including each marked block. You get up to four breakpoints, so you can cache tools and a stable system prompt at one level and a slower-changing conversation history at another.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    # Tools come first in the token stream — keep them byte-stable.
    tools=[
        {
            "name": "search_docs",
            "description": "Search internal documentation.",
            "input_schema": {
                "type": "object",
                "properties": {"query": {"type": "string"}},
                "required": ["query"],
            },
            # Breakpoint 1: cache the tool definitions.
            "cache_control": {"type": "ephemeral"},
        }
    ],
    system=[
        {
            "type": "text",
            "text": LARGE_STATIC_SYSTEM_PROMPT,  # instructions, few-shot, policies
            # Breakpoint 2: cache tools + system prompt as one prefix.
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[
        # Volatile content lives here, AFTER the cached prefix.
        {"role": "user", "content": f"Current time: {now}. {user_question}"},
    ],
)

print(response.usage)  # watch cache_read_input_tokens and cache_creation_input_tokens

Two rules make or break this:

Never let a per-request value appear before a cache_control marker. The timestamp in the example is in the messages block, below both breakpoints, so it costs nothing.
Serialize deterministically. If you build tools or system content programmatically, pin key order (json.dumps(obj, sort_keys=True)) and avoid sets and unordered dict merges in the hot path.

For OpenAI's GPT models, caching is automatic — there is no cache_control. The server hashes prompt prefixes and reuses them when the leading tokens match, typically for prompts over ~1,024 tokens, matched in coarse increments. The structural discipline is identical even without explicit markers: static-first, volatile-last, deterministic bytes. You just do not get to choose the breakpoints.

Claude vs GPT prompt caching: what's different?

Both cache exact prefixes, but the control surface and economics differ.

Claude requires explicit cache_control breakpoints and gives you a choice of TTL: a default short-lived ephemeral cache (about 5 minutes, refreshed on each hit) and a longer 1-hour option. You pay a write premium the first time a prefix is cached — roughly 25% over base input tokens for the 5-minute tier, and more for the 1-hour tier — and reads come back at about 10% of the base input price. Because you control breakpoints, you can cache tools, system prompt, and conversation history at separate layers.

GPT caches automatically with no markers and no explicit write premium, discounting cached input tokens by roughly 50%. It is simpler and lower-effort, but you cannot pin what gets cached or extend the TTL, and eviction is opaque. You optimize purely through prompt structure.

Practical consequence: on Claude, an agent that reuses a big tool-plus-system prefix across a burst of turns is dramatically cheaper if you set breakpoints and keep the prefix stable. On GPT you get a meaningful discount for free, but you cannot rescue a badly ordered prompt with configuration — you must fix the ordering.

What does prompt caching actually cost, and when does it pay off?

Caching is not free, and a naive breakpoint on a rarely-reused prefix can lose money. On Claude, the first request writes the cache at a premium; subsequent requests read it at a deep discount. The break-even is simple: the write premium must be recovered by enough reads before the TTL expires.

If your prefix is 10K tokens and you reuse it 20 times within five minutes, you pay one ~1.25x write and 20 reads at ~0.1x instead of 21 reads at 1.0x — a large win. If you cache a prefix used once and never again inside the window, you paid the premium for nothing. High-throughput agents and chat sessions with a fat shared system prompt are the ideal case; one-shot batch jobs with unique prompts are not.

Two operational habits matter more than any config:

Measure the hit rate. Log cache_read_input_tokens and cache_creation_input_tokens from the usage object on every call. A healthy setup shows a big creation number on the first request of a session and reads dominating afterward. All-creation, zero-read means your prefix is changing every request — go hunt the dynamic token.
Keep the prefix warm. The 5-minute TTL refreshes on each hit. If traffic is bursty with gaps longer than the TTL, either use the 1-hour tier or accept periodic re-writes.

The bottom line

Prompt caching kills your 90% discount the moment a single dynamic token lands ahead of your static content, because the cache is an exact-prefix match: it reuses the KV state only for the unbroken run of identical tokens starting at position 0, and it stops — permanently, for that request — at the first byte that differs. A timestamp, a shuffled tool definition, or a per-user ID sitting above your big system prompt invalidates everything behind it and quietly re-bills you at full price with no error. Fix it structurally: order the prompt static-first and volatile-last, serialize tools and templates to deterministic bytes, place cache_control breakpoints (on Claude) at the static boundaries, push timestamps and user turns to the bottom, and log cache_read_input_tokens so a broken cache cannot hide. Do that, and a reused prefix costs about a tenth of what it should; skip it, and you pay full freight while your dashboard shows nothing wrong.