My "prompt caching" change made the bill go up — here's what I got wrong

I turned on prompt caching for an agent I run all day, watched the next invoice, and the number went up. Not by a rounding error — meaningfully up. This is the write-up of why, because the thing I got wrong is the thing almost every "just add caching" post skips.

The one-sentence version of caching

You send the same long system prompt (plus tool schemas, plus a pasted document) on every call. Without caching you pay full input price to make the model re-read that identical block every single time. Caching stores a prefix of your prompt and, on the next call, charges a fraction to read it back instead of full price.

Sounds free. It is not free. That's the trap.

The part that bit me: writes cost more

A cache entry has two prices, and I only knew about one of them:

• Cache read (a hit): billed at as low as 0.1x the input rate. This is the number everyone quotes. ~90% off the cached tokens. Great.
• Cache write: billed at a premium — around 1.25x the input rate on the short (5-minute) window. This is the number nobody mentions.

So the first time you send a prefix — or any time it's expired — you don't save 90%, you pay a 25% surcharge to store it. Caching only comes out ahead once that same prefix is read back, not just written. The good news: the write costs ~0.25x more than a normal read (1.25x vs 1x), and a single cache hit saves 0.9x (0.1x vs 1x) — so the very first reuse already earns the premium back, and every hit after that is ~90% off. The catch is you have to actually get the hits.

Which means the failure mode is specific and nasty: write a cache entry, then never read it. You paid extra for storage you never used. Do that on every single call and you've built a machine that pays a surcharge to cache things it immediately throws away.

That was exactly my bug.

What I actually did wrong

My "stable" system prefix wasn't stable. Buried near the top of it was this, which I'd added months earlier and forgotten:

Current session started: 2026-07-14T09:41:07Z

A timestamp. Regenerated every call. Which meant the prefix was different every call, so the provider dutifully wrote a fresh cache entry every single time — full write premium — and never once got to read one back, because no two prefixes matched. I had turned on the expensive half of caching and none of the cheap half.

The tell was right there in the response the whole time, I just wasn't looking:

print(resp.usage)
# cache_creation_input_tokens: 3948   <- writing every call
# cache_read_input_tokens: 0          <- never reading. there's your bug.

cache_creation climbing while cache_read sits at zero is the signature of a busted prefix. If you take one thing from this post: log those two fields and watch them.

The fix is discipline, not more caching

Caching rewards a prefix that is byte-for-byte identical across calls and sits before anything that changes. Two rules:

1. Static content first, dynamic content last. System prompt, tool definitions, the big pasted document — all up front and unchanging. The user's actual turn, the timestamp, the per-request ID — all at the end, after the cached region. Anything that varies must live downstream of the cache breakpoint, never inside it.

2. Keep the prefix stone-cold identical. No timestamps, no reordered JSON keys, no "helpful" per-request injection into the system block. One moving character and the whole prefix misses.

For Claude, you mark where the stable prefix ends with a breakpoint:

resp = client.messages.create(
    model="claude-opus-4-8",
    system=[{
        "type": "text",
        "text": LONG_STABLE_INSTRUCTIONS,        # no timestamps in here!
        "cache_control": {"type": "ephemeral"},  # cache everything up to this point
    }],
    messages=[{"role": "user", "content": todays_question}],  # the only thing that varies
    max_tokens=1024,
)

For OpenAI-family models (GPT-5.6 / 5.5) it's automatic above ~1024 tokens — same discipline, no breakpoint: keep the static stuff at the front and let it match the longest common prefix. Two things differ from Claude, worth knowing: there's no write premium (OpenAI doesn't charge to create the cache), and the read discount is smaller than Claude's 0.1x — you still save, just not 90%. You'll see it land in prompt_tokens_details.cached_tokens.

I moved the timestamp to the user message, left the system block untouched between calls, and re-ran. cache_read_input_tokens lit up on the second call and the bill for that agent dropped hard — because it loops twenty-plus times over the same head, and now that head is written once and read twenty times instead of written twenty times and read never.

Where caching actually pays (and where it doesn't)

It's amortization, not magic. It wins when a big, identical prefix is read many times:

• Agent / tool loops — every step resends the same system prompt and tool schemas. Ideal.
• Long-doc Q&A — paste the document once, ask ten questions against it.
• Bulk extraction — fat instruction + few-shot block, only the input row changes.

It does nothing for one-shot calls where the whole prompt is different every time — there's no repeated prefix to amortize against. On Claude, explicitly caching there just burns the 1.25x write for no return; OpenAI won't charge you for the attempt, but it won't help either. Match the tool to the workload.

One gateway gotcha, since I run through one

I don't call the providers directly — I route through a gateway (I use byesu, pay-as-you-go across Claude/GPT/Grok on one key). That adds a failure mode worth naming: some aggregators quietly break caching. If a proxy reorders your fields or strips cache_control or injects a per-request header into the prompt body, the prefix stops matching — and you get an answer with zero cache hits and no error telling you. You just silently pay full price.

So test it on your gateway specifically: send the same request twice and confirm cache_read_input_tokens is non-zero on the second. On the one I use, cache_control and the usage fields pass through to the official upstream unchanged, so the cache key I build is the one that actually gets used and the 0.1x reads show up — but "does my proxy preserve caching" is a question you should answer with the usage object, not take on faith from anyone, me included.

The whole lesson in three lines

• Caching has a write premium (~1.25x); reads are cheap (0.1x). It's amortization — you need repeat reads.
• The prefix must be byte-stable and up front. A timestamp in your system prompt will quietly cost you money.
• Watch cache_read_input_tokens. Zero on a repeat call means it's not working, whatever the marketing said.

If you want the reference version — the breakpoint rules, the OpenAI side, the exact usage fields per provider — I wrote it up over here. But the timestamp story above is really the whole lesson.

Comments

[Luis]

The distinction between cache read and cache write prices is a crucial one, and I appreciate you breaking down the numbers - a 25% surcharge on cache writes can indeed add up quickly. I've had similar experiences with caching in other contexts, where the initial write cost outweighs the benefits of subsequent reads if the cache isn't properly utilized. Your two rules for effective caching - static content first, dynamic content last, and keeping the prefix identical - are excellent guidelines for developers to keep in mind. Have you considered implementing any automated checks or monitors to ensure the cache prefix remains stable and effective over time?