In my last post I broke down how LLMs count tokens and why your bill is decided at the tokenizer. A lot of the follow-up questions were the same: "Okay — so how do I actually pay less?"
This is the answer, and it's the single highest-leverage cost lever available on Claude, GPT, and Gemini today: prompt caching. It requires no model change, no quality tradeoff, and often just a few lines of code. Done well, it cuts input token costs by 70-90% on typical production workloads. Done poorly, it does nothing — or quietly makes things worse.
Let's break down how it works, the real cost math, and the mistakes that leave most of the savings on the table.
What Prompt Caching Actually Does
When a model processes your prompt, it computes attention key-value (KV) tensors for every token. That's real compute, and you pay for it on every request.
Here's the thing: most production prompts are mostly repetition. The system prompt, the tool definitions, the few-shot examples, the big RAG document — that prefix is identical on call after call. Only the last bit (the user's actual message) changes.
Prompt caching stores the already-computed KV tensors for that repeated prefix server-side. When your next request starts with the same prefix, the model skips recomputing it and loads the cached result instead — and bills those cached tokens at a steep discount.
The mental model: you're paying full price to process your system prompt once, then paying a fraction of that to reuse it for the next N requests.
The Golden Rule: Static First, Dynamic Last
Caching works on prefixes. The provider can only reuse everything up to the first point where your prompt differs from last time.
That single fact dictates everything:
Put stable content at the front. Put anything that changes at the very end.
Structure your prompt like this:
[ system prompt ] ← static, cache this
[ tool definitions ] ← static, cache this
[ few-shot examples ] ← static, cache this
[ retrieved documents ] ← semi-static
--------------------------------------------------
[ conversation history ] ← changes
[ user's new message ] ← changes every time
The instant something dynamic leaks into the front — a timestamp, a request ID, a randomly ordered tool list, even an inconsistent trailing newline — the cache invalidates for every token after it. This is the number-one reason teams see "90% off" pricing advertised and a 20% hit rate in reality.
How Each Provider Handles It
The three big providers all discount cached input by roughly 90%, but the mechanics and fine print differ enough to change your architecture.
Anthropic (Claude) — explicit, precise
You opt in by marking stable blocks with cache_control (up to 4 breakpoints). Claude splits your bill into:
- Cache write — the first time a block is processed. Costs more than normal input: ~1.25× base for the 5-minute cache, ~2× for the 1-hour cache.
- Cache read — every subsequent hit. Charged at ~0.10× (a 90% discount).
Default TTL is 5 minutes, and every read refreshes the timer. There's also an automatic mode now, but explicit breakpoints give you the most control. The write surcharge typically pays for itself on the first hit.
OpenAI (GPT) — automatic, zero-setup
Caching happens automatically on supported models — no markers, no cache objects. The catch is a hard 1,024-token minimum prefix: a 900-token system prompt will never cache, no matter how consistent it is. Retention runs from a few minutes of inactivity up to 24 hours on newer models. Simplest to adopt, least to tune.
Google (Gemini) — powerful, but mind the storage fee
Gemini offers implicit caching (automatic, ~10% read rate) and explicit caching (you create a named cache object and reference it by ID). The gotchas:
- A large minimum cacheable block (tens of thousands of tokens — much bigger than Claude or OpenAI).
- Per-hour storage cost: you pay for cached tokens for every hour they sit there, even if nothing reads them. Great for large-context architectures, punishing for low-traffic ones.
Gemini caching rewards big-document workloads, not typical short prompt engineering.
| Claude | GPT | Gemini | |
|---|---|---|---|
| Mode | Explicit (cache_control) + auto |
Automatic | Implicit + explicit |
| Read discount | ~90% off | ~50-90% off | ~90% off |
| Write cost | 1.25×-2× base | none extra | storage/hour |
| Min prefix | small | 1,024 tokens | very large |
| Default TTL | 5 min (refreshes) | up to 24h | configurable |
(Mechanics and rates shift often — confirm on each provider's current docs before budgeting.)
The Cost Math (This Is the Part That Matters)
Let's make it concrete. Say you have a 5,000-token system prompt reused across 10,000 requests/day, on a model at $3 / 1M input with a cache read at $0.30 / 1M (90% off) and a 5-minute write at $3.75 / 1M.
Without caching
Every request pays full price for those 5,000 tokens:
10,000 req × 5,000 tokens = 50,000,000 input tokens/day
50,000,000 / 1,000,000 × $3 = $150/day → ~$4,500/month
(That's just the cached-portion input — actual bills add the dynamic tail and output.)
With caching
Assume the cache is written fresh a handful of times a day as it expires — say ~50 writes — and everything else is a read:
Writes: 50 × 5,000 / 1,000,000 × $3.75 ≈ $0.94/day
Reads: 9,950 × 5,000 / 1,000,000 × $0.30 ≈ $14.93/day
------------------------------------------------------------
Total ≈ $15.87/day → ~$476/month
That's the repeated-prefix cost dropping from ~$4,500 to ~$476 a month — about 89% off — by making one part of your prompt cacheable. The often-cited real-world version of this is an agent that went from $720/month to $72/month by adding three cache breakpoints.
The savings scale with two things: how big your static prefix is and how often you reuse it within the TTL. Big system prompt + high request rate = enormous savings. Tiny prompt + sporadic traffic = little to none.
The Mistakes That Kill Your Savings
Caching fails quietly. You still get correct responses — you just don't get the discount, and nothing errors out to tell you. Watch for these:
- Dynamic content in the prefix. A timestamp, session ID, or version string near the top invalidates everything after it. This is the most common cause of "90% pricing, 20% hit rate."
- Unstable serialization. Tool lists or JSON keys in a non-deterministic order break the exact-match requirement. Sort them.
- Below the minimum. Under OpenAI's 1,024-token or Gemini's large minimum, you'll never see a hit. Check the threshold.
- TTL expiry from low traffic. Claude's default 5-minute window is short. If requests are sparse, the cache expires between them and you pay to rewrite every time. Consider the 1-hour cache — but weigh its higher write cost.
- Cross-model assumptions. Caches aren't shared between models. A Sonnet cache can't be hit by a Haiku request.
- Gemini storage creep. For occasional batch jobs, delete explicit caches after use so you're not paying hourly storage on idle tokens.
How to Actually Verify It's Working
Don't trust the marketing — trust the usage metadata. Every provider exposes cache stats (field names differ):
-
Anthropic:
cache_read_input_tokens,cache_creation_input_tokens -
OpenAI:
cached_tokens - Gemini: cached content token counts
A quick health check:
hit_rate ≈ cached_input_tokens / total_cache_eligible_input_tokens
If that number is consistently near zero, dynamic content has leaked into your prefix. A sudden drop usually means something changed in the reusable part of your prompt.
Practical Takeaways
- Prompt caching is the highest-ROI cost lever on Claude, GPT, and Gemini — no quality tradeoff, minimal code.
- Structure prompts static-first, dynamic-last. This one rule determines your entire hit rate.
- Cached reads run ~90% off, but writes can cost more than normal input — so caching pays off with reuse, not one-offs.
- Mind the minimums and TTLs: OpenAI's 1,024-token floor, Gemini's large block + storage fee, Claude's short-but-refreshing 5-minute window.
- Verify with usage metadata, not assumptions. Watch your real hit rate.
- Re-check the numbers — caching rules and prices change often.
Caching won't fix a badly chosen model or a bloated context. But for the very common case of "big stable prompt, called a lot," it's the closest thing to free money in the LLM stack. Most teams are leaving 70-90% of it on the table.
Have you shipped prompt caching in production? What was your real hit rate — and what broke it? I'd love to hear the war stories in the comments.
Top comments (1)
This is a fantastic write-up. 👏
What really caught my attention isn't even the Cilium/Istio interaction itself, but the distinction between configuration state and actual runtime state.
The ConfigMap was correct, the verification was technically correct, and yet the datapath was still running with the old configuration. That's a particularly nasty class of failure because every individual check can look perfectly reasonable while the overall conclusion is wrong. 🔥
I also really liked the independent subagent approach. The second agent wasn't necessarily “smarter” — it simply wasn't anchored to the previous investigation. It was able to challenge the assumptions and reproduce the verification from a different perspective.
I think this is a really interesting pattern for agentic systems: independent verification shouldn't just repeat the previous agent's reasoning. It should be able to challenge the assumptions that gradually became “facts” during the investigation.
And that final lesson is brutal in the best possible way: sometimes the hypothesis isn't wrong. We simply stopped verifying whether our fix actually reached the system we thought we had changed.
Excellent work. This is exactly the kind of real-world debugging experience that makes an AI engineering article genuinely valuable. 🚀