Originally published at Trida AI.
Step nine of a document review looks almost identical to step eight. Same instructions. Same tool definitions, same contract, same conversation so far. One new tool result sits at the end. The model still processes the prompt from the first token, and you pay full input price for work it did a minute ago. Repeat that across a long run, all day, and the repeated prefix becomes the largest line in the bill.
Prompt caching is meant to stop that waste. When a later request starts with the same tokens as an earlier one, the provider can reuse the processing state it already built for that prefix instead of computing it again. The cached part gets billed at a read rate, and the model reaches your new content sooner. The catch is in the first three words of that description: the same tokens. Caching is an exact-match system. It rewards prompts assembled in a stable order and punishes prompts that shuffle, timestamp or personalise their opening lines. In an agent, where the prompt is rebuilt on every step, caching is a design decision about how you assemble context. It is not a switch you flip at the end.
Put the boundary where stability ends
Think of an agent prompt as layers with different rates of change. Tool definitions change when you ship code. System instructions change when you revise the deployment. Shared reference material, a policy document or a contract under review, changes per case. Conversation history grows every turn but never rewrites itself, if you let it append rather than summarise. The current question changes every time. A cache boundary belongs at the end of a layer that stays identical across the requests you want to share it.
Anthropic's prompt caching makes the layering explicit: prefixes are built in the order tools, then system, then messages, you can place up to four breakpoints, and a change at one level invalidates that level and everything after it. Edit a single tool description and the whole cache goes. Teams also put the breakpoint at the end of the prompt, after the per-request block, and then wonder why they pay a cache write on every call and never get a read. If the block at your breakpoint contains a timestamp, the prefix hash changes every request. The fix is to move the breakpoint to the last block that stays the same.
The quiet cache killers are mundane. A date string in the system prompt. A user ID at the top instead of the tail. Tool definitions serialised in a different order by a JSON library that does not preserve key order. A summarisation step that rewrites earlier turns, so history stops being a prefix. None of these throw an error. The answer looks fine, and the usage fields quietly report zero cached tokens.
Prompt assembly as layers, ordered by how often each layer changes: tool definitions change per deploy and sit at the front of the cacheable prefix, system instructions change per revision, shared reference material changes per case, conversation history appends each turn, and the current turn sits after the cache boundary and is never cached.
A TTL is a bet on the next request
A cached prefix does not live forever, and keeping it alive is not free on the explicit systems. On Anthropic's defaults, writing a prefix into the 5-minute cache costs 1.25 times the base input price, a read costs 0.1 times on most models, and each read refreshes the 5 minutes at no extra charge. The 1-hour option writes at 2 times. Do the arithmetic and the 5-minute cache pays for itself after a single read: one write plus one read costs 1.35 against 2.0 for two uncached passes. The 1-hour cache needs two reads before it wins. So the longer TTL is not the safer choice. It is a bet that the next request will arrive after five minutes but within the hour, placed often enough to repay a dearer write.
OpenAI's prompt caching draws the same trade with different numbers. On GPT-5.6 and later, the default minimum lifetime is 30 minutes at the same 1.25 write rate, with a 12-hour option in the Responses API at 2.5 times and reads at 0.1 times on most models. Earlier models charge no write premium at all. Session shape should decide. A tight tool loop lives happily on the shortest TTL. An agent that waits on a person or a slow external system has gaps in tens of minutes, and that is where the longer lifetime earns its premium. An overnight batch reuses nothing either way: its first request tomorrow is a write, whatever you chose.
One shared prefix of 12,000 tokens reused across eight requests in a session: without caching the prefix costs 96,000 token-equivalents; with a 5-minute cache at a 1.25 write rate and 0.1 read rate it costs 23,400, made of one write at 15,000 and seven reads at 1,200 each; the saving applies to the prefix only, not to output or new content.
Take a concrete session. A review agent carries a shared prefix of 12,000 tokens: instructions, tools and the document under review. It makes eight requests before the case closes. Uncached, that prefix alone accounts for 96,000 tokens at full input price. Cached on 5-minute terms, it costs one write at 15,000 token-equivalents and seven reads at 1,200 each, 23,400 in total. That is a real saving, and it is also a narrower one than the headline suggests. The discount applies to the prefix. Output tokens, new tool results and the current question are billed as always. A team that reports a 90 percent saving because reads cost a tenth of input has confused the rate on one component with the shape of the whole bill.
Read the hit rate, not the headline
Every provider exposes the evidence in the usage block of the response. Anthropic reports separate counts for tokens written to the cache and tokens read from it. OpenAI reports a cached token count. Gemini implicit caching, on by default for Gemini 2.5 and newer models, reports hits in a total cached token field. Read those fields as a ratio over time: cached reads against total input for the same workload. Two patterns deserve a look. A sudden rise in cache writes mid-session usually means something in the prefix changed: a deploy, a tool edit, a template gaining a timestamp. A hit rate that never rises usually means the prefix was never stable enough to match, or never long enough to qualify.
There are honest limits. Short prompts may not be cacheable at all: minimums run from 512 to 4,096 tokens depending on the Anthropic model, 1,024 tokens on OpenAI's newer models, and 2,048 to 4,096 across the Gemini range. On Anthropic's system, parallel requests race each other, because an entry becomes available only after the first response starts, so a burst of identical first calls can all pay full price. Caching changes cost and latency, not the answer. And a prefix shared across customers needs care: what you place in a shared prefix is visible to every session that reuses it. Customer-specific material belongs after the boundary, not before it.
Our rule is simple. Design the prompt in layers, put the boundary at the last stable layer, choose the TTL from the gaps you actually see between requests, and watch the read ratio the way you watch error rates. If you cannot show those four things, you do not have a caching strategy. You have a hope, billed monthly.
Work with us
If your agent bill is dominated by the same instructions, tools and documents being re-read on every step, the fix is in how the prompt is assembled, and that is deployment work. Bring us the run you cannot explain the cost of through Trida AI's intake.
Top comments (0)