Prompt caching sounds like free money. Send the same long instructions once, reuse them, and pay less on later requests. In practice, the saving depends on one question: how much of your prompt really repeats, word for word, from one request to the next?
A system prompt that never changes is a good candidate. A conversation history that grows with every turn is not the same thing. Neither is a document that changes for every user. If only a small slice repeats, the discount applies only to that slice, and the rest is still billed at the full rate.
To estimate it honestly, split your typical prompt into a stable part and a changing part. Apply the cached rate only to the stable part, the normal rate to the rest, and then look at how many of your requests actually hit the cache. A high cache hit rate with a small stable part still saves little. A large stable part with a low hit rate also saves little. You need both.
That is why I estimate caching with two inputs side by side: the share of tokens that can be cached, and the share of requests that hit the cache. Run it with your real numbers, not with the best case from a pricing page. If the saving is small, shortening the prompt or restructuring the request may save more than caching ever will. You can test the trade-off with this calculator before changing any code.
Top comments (0)