Prompt Caching: Put the Stable Stuff First (and Stop Paying Twice for the Same Tokens)

#ai #llm #python #programming

Most LLM apps send the same few thousand tokens on every request: the tool definitions, the system prompt, a block of reference docs, a handful of examples. Only the last few hundred tokens, the user's actual message, change. Without caching, you pay full input price to reprocess that identical prefix every single time.

Prompt caching fixes that, but only if you build your prompt in the right order. This is the one lesson worth taking away today: stable stuff first, changing stuff last.

What actually gets cached

When a model reads your prompt, it computes intermediate key-value (KV) states for every token. Prompt caching saves those states for a reusable prefix, the unchanged tokens at the start of the prompt. If your next request starts with exactly the same prefix, the provider reuses the saved work and only processes what's new.

The key word is exactly. Both OpenAI and Anthropic match on the full rendered prefix. Change one token near the top and everything after it is a miss.

Think of it like a bookmark in a long book: the model can jump straight back to where it left off, but only if nobody rewrote the earlier pages.

Why it's worth caring about

The numbers from the providers' own docs:

Anthropic: cache reads are billed at 0.1x the base input price on most Claude models. Writing to the default 5-minute cache costs 1.25x, and the 1-hour cache costs 2x. The 5-minute cache refreshes for free each time it's used.
OpenAI: caching is on by default for supported models. On GPT-5.6 and later, cache writes cost 1.25x and reads 0.1x the uncached input rate, and the minimum cacheable prefix is 1,024 tokens. OpenAI's own worked example: one write plus nine full reads costs 2.15x the input price of the prefix, versus 10x without caching.

Caching also cuts time-to-first-token, because the model skips reprocessing the long prefix.

The ordering that makes it work

Build every request in this order:

Tools and the system prompt. These rarely change. On Anthropic, the cache hierarchy is literally tools, then system, then messages.
Shared reference material. Docs, a policy, few-shot examples, a long contract you're asking questions about.
Conversation history. Append new turns; don't rewrite old ones.
The new user message and anything dynamic. Timestamps, user names, per-request context.

The mistake almost everyone makes

Putting something dynamic at the top. The classic one is a system prompt that starts with Current date and time: 2026-10-12 09:04:31. That one line changes on every request, so the prefix never matches and you never get a cache hit.

Anthropic's docs describe a nastier version of the same trap: if you put cache_control on the block that contains the timestamp and the user message, you pay for a fresh cache write on every request and never get a read. Move the breakpoint to the last block that stays identical, and every later request reads from cache.

Other quiet cache-breakers from the docs:

Changing tool names, descriptions, schemas, or their order.
Languages that randomize JSON key order when serializing tool calls (Anthropic calls out Swift and Go).
Toggling settings that get rendered into the prompt, such as tool_choice, images, or reasoning/thinking settings.
Summarizing or compacting history, which rewrites the prefix.

What it looks like in code

Anthropic, with an explicit breakpoint at the end of the stable part:

response = client.messages.create(
    model="<your-claude-model>",
    max_tokens=1024,
    system=[
        {"type": "text", "text": "You are a support assistant for Acme..."},
        {
            "type": "text",
            "text": "<full product docs and policy here>",
            "cache_control": {"type": "ephemeral"},  # cache everything up to here
        },
    ],
    messages=[
        {"role": "user", "content": f"(sent at {now}) {user_question}"},  # dynamic, after the breakpoint
    ],
)
print(response.usage)  # look at cache_read_input_tokens / cache_creation_input_tokens

OpenAI needs no code change for basic caching on supported models. Keep the prefix stable and check usage.input_tokens_details.cached_tokens to confirm it's working.

Verify it, don't assume it

Caching fails silently. A prompt below the minimum length just isn't cached, with no error. So measure:

Anthropic: cache_read_input_tokens and cache_creation_input_tokens. If both are 0, nothing was cached.
OpenAI: usage.input_tokens_details.cached_tokens, plus the Prompt Caching dashboard.

Track a simple ratio, cached tokens divided by total input tokens, per day. If it drops after a deploy, someone put something dynamic in the prefix.