I spent a while assuming prompt caching was something you turn on. It isn't. It's a decision about where you cut your prompt, and if you cut it in the wrong place the feature quietly does nothing while your code still looks correct.
This is what I learned building TechnoRAG, a hybrid RAG service over a daily-refreshed news corpus. I ended up sustaining a 57 to 60% input cache hit rate, which made cached calls roughly 10x cheaper. Most of that came from one structural change, not from tuning anything.
The thing nobody tells you upfront
Prompt caching is prefix caching. The provider hashes your input from the very first token and matches the longest identical run against what it already has stored.
That word "prefix" is doing all the work. It means the cache breaks at the first byte that differs, and everything after that point is a miss, even if 90% of it is identical to last time.
So this prompt:
You are a research assistant.
Current time: 2026-08-05T14:22:31Z
[2000 tokens of instructions and output schema]
[retrieved context]
[user question]
caches nothing. Not the instructions, not the schema. The timestamp on line two poisons every token after it. I had something close to this for longer than I'd like to admit, looked at my hit rate sitting near zero, and assumed caching just wasn't working on my provider.
Sorting by how often things change
The fix is boring. Order your prompt by rate of change, most stable first:
[1] role and task definition never changes
[2] output schema and format rules never changes
[3] query-type instructions 4 variants, changes per request type
[4] retrieved chunks changes every request
[5] user question changes every request
Layers 1 and 2 are the same on every single call, so they sit at the front and cache permanently. Layers 4 and 5 are different every time and can never cache, so they go at the back where they can't hurt anything.
Layer 3 is the interesting one. TechnoRAG routes queries into four intent types (conceptual, factual, comparative, temporal) and each gets slightly different instructions. My first instinct was that this breaks caching, since the text varies. It doesn't. It just means you maintain four cache prefixes instead of one, and each of them still gets reused across every request of that type. Four warm caches is fine. What isn't fine is putting that block above the schema, because then the schema has to be re-processed four different ways for no reason.
That's the whole idea. Stable stuff first, volatile stuff last, and volatile stuff should never sit above stable stuff.
Things that broke my cache without looking like they would
A few of these cost me real time.
Timestamps anywhere in the system prompt. Obvious in hindsight. If your pipeline is time-aware (mine has a temporal freshness layer) the instinct is to tell the model what time it is up front. Move it down next to the retrieved context instead.
Retrieved chunks in non-deterministic order. My reranker was returning ties in whatever order they came out of the fusion step. Same chunks, different sequence, different hash. This only matters if any of that content sits above something you want cached, but it's worth knowing your ordering is stable.
JSON key ordering. If you're serialising part of your prompt from a dict, Python preserves insertion order but the thing constructing that dict might not be deterministic across runs. sort_keys=True costs nothing.
Trailing whitespace. Two prompt templates that differ by a single newline are two different prefixes. I found one of these by diffing two raw request bodies byte for byte, which is not a fun evening.
Minimum cacheable size
Worth checking your provider's floor before you spend an afternoon restructuring. Caches work in blocks and there's usually a minimum number of tokens below which nothing gets stored at all.
If your static prefix is short, you may get nothing back no matter how well you order it. In that case you either accept it, or you find that padding the stable section with genuinely useful content (fuller output examples, more explicit format rules) pushes you over the line and improves output quality at the same time. That second option felt like cheating until I realised the examples were making the responses better anyway.
How I actually verified it
I didn't trust the hit rate until I could see it per call. LangFuse gives you cached versus uncached input tokens on each generation, so I logged both and watched the ratio over a few hundred requests.
Two things showed up immediately. The rate was near zero on cold start and climbed as prefixes warmed, which is expected but looks alarming if you only check once. And it dropped sharply whenever I edited the system prompt, since every edit invalidates everything downstream of the change. If you're iterating on prompt wording, your hit rate during that session tells you nothing useful.
What I'd tell myself at the start
Draw the line before you write the prompt. Decide which parts are allowed to change per request, put everything else above them, and don't let a single dynamic value sneak into the stable section for convenience.
It took me one restructuring pass to go from a hit rate near zero to the high fifties. The prompt content barely changed. Only the order did.
Building TechnoRAG (hybrid RAG service) and HopLens (step level evaluation for multi hop agentic RAG). Code on GitHub.
Top comments (0)