If you run an LLM app in production, there's a good chance most of your input bill is paying for the same tokens over and over. A support bot ships a 2,000-token system prompt and a product manual on every single call. An agent replays its instructions and tool definitions every turn. A RAG pipeline stuffs the same retrieved documents into the prompt for each follow-up question. The model re-reads all of it from scratch, every time, and you pay full price for the privilege.
Prompt caching fixes exactly this. It's Day 31 of my AIFromZero series, and it might be the single highest-leverage cost change you can make.
The one idea
Split your prompt into two parts:
- A big stable prefix: system prompt, tool definitions, long documents, few-shot examples. This is the same on every call.
- A small volatile suffix: the user's actual question. This changes every call.
The first time the model sees the prefix, it runs the expensive prefill step — reading every token and building its internal attention state. That's a cache miss. Prompt caching stores that computed state. On the next call with a byte-identical prefix, it's a cache hit: the state is loaded back instead of recomputed, and only the new question gets processed.
Two things happen on a hit. The cached input tokens are billed at roughly 10% of the normal price — a 90% discount on the part of your prompt that never changes. And the prefill latency for the prefix disappears, so your time-to-first-token drops sharply.
Prefill, decode, and the KV-cache
To see why this works, it helps to know the two phases of generation. Prefill reads the whole prompt in one pass; its cost scales with prompt length. Decode then emits the answer one token at a time. For a long prompt and a short answer, prefill dominates both the time and the input cost.
Within a single generation, models already reuse work: as they decode token 500, they don't re-read tokens 1 through 499 — they keep those tokens' key/value attention tensors in a KV-cache and reuse them. Prompt caching is that same trick, stretched across separate API requests. Instead of throwing away the prefix's prefill state when the call ends, the provider keeps it around, keyed by the exact bytes of the prefix.
Byte-identical or bust
Here's the rule that trips people up: caching is a prefix match. The cache key is the exact bytes from the start of the prompt up to a breakpoint. Change one character anywhere in the prefix and the key changes, so you get a miss and pay to re-prefill everything after that point.
This has two practical consequences:
- Put stable content first, volatile content last. Render order matters as much as content.
-
Keep the prefix deterministic. The classic silent cache-killers are a
datetime.now()in the system prompt, a request ID or username interpolated near the top, an unsortedjson.dumps(), or swapping the tool list mid-conversation. Any of these makes the prefix unique on every call, so nothing ever caches — and you won't get an error, just a bill that never drops.
Always verify. Read the usage fields: cache_creation_input_tokens is what you wrote (the miss), cache_read_input_tokens is what you reused (the hits). If reads stay at zero across identical-looking calls, diff the actual rendered prompt bytes between two requests and hunt down the invalidator.
The cost math
Caching isn't free. A cache write costs a premium — about 1.25x the base input price for a 5-minute TTL, or 2x for a 1-hour TTL. A cache read costs about 0.1x. So the first call is slightly more expensive, and every reuse is roughly 90% cheaper on the cached tokens.
That means you break even fast. With the 5-minute TTL, the second call already pays for the write premium (1.25x + 0.1x is less than paying 2x uncached). The 1-hour TTL needs about three calls. Only cache prefixes that get reused enough to clear that bar, and remember there's a minimum length (roughly 1,024 to 4,096 tokens depending on the model) below which caching silently does nothing.
TTL, providers, and where it shines
Cache entries live on a time-to-live and evict under pressure. Anthropic defaults to 5 minutes with a 1-hour option, and each hit refreshes the timer. For bursty traffic, use the longer TTL or pre-warm the cache with a cheap startup call so your first real user hits a warm cache.
Providers expose this differently. Anthropic is explicit — you place a cache_control breakpoint on the last stable block. OpenAI and others cache long prefixes automatically once they clear a length threshold. The design decision is the same either way: freeze the prefix, keep the volatile stuff out of it.
The payoff is biggest wherever a large prefix repeats: agents, RAG, and any long-system-prompt app. The more you reuse that prefix, the closer your effective input cost slides toward the 10% floor.
I built an interactive demo where you send requests, edit the prefix to watch the cache break, and see the cost and latency meters move in real time:
Top comments (0)