Prompt Caching: The Patterns That Actually Save Real Money

Prompt caching can meaningfully cut API costs, but only if the prompt is actually structured to take advantage of it — a surprising number of implementations enable caching and see almost no savings because of how the prompt is built.

🏗️ Static Content Has to Come First and Stay Identical

Caching works on a prefix match — the system prompt, tool definitions, and any large static context need to be identical, byte-for-byte, across calls and placed before the variable, per-request content. A single changed character anywhere in the cached prefix breaks the match for that call.

📦 What's Actually Worth Caching

Large static context (reference documents, tool schemas, long system prompts) is where caching pays off — caching a short system prompt that's already cheap barely moves the needle. The bigger and more repeated the static block, the more caching actually saves.

📊 Actually Measure the Hit Rate

Most API responses report whether a call hit the cache. Checking that number directly — not assuming caching is working because it's enabled — catches the common case where a well-intentioned cache setup has a near-zero hit rate due to a subtle prompt-ordering mistake.

---

Originally published on the KLYX blog.