My context optimizer was what broke my prompt cache

Suleiman Ribeiro — Tue, 15 Sep 2026 17:31:13 +0000

I spent weeks chasing the wrong theory about why my agent was expensive.

The theory was the obvious one: expensive context means large context. So I built what seemed like the right fix — a routine that re-scored the tool set on every turn and sent only the tools relevant to that particular message. Fewer tokens out, smaller bill.

Here is what it actually did, measured in production on three trivial turns of the same conversation:

	before	after
tools, turns 1→3	52 → 54 → 61	59 · 59 · 59
cache hit rate	9.6%	80–88%
cost per turn	$0.0168	$0.0036–0.0046

Same model. Same conversation. A 75% reduction on the cached turn, and I got it by removing the optimization.

The mechanism

Provider prompt caching matches the prefix byte for byte, up to the first difference. Your tool schema is part of that prefix.

My routine re-scored tools by embedding the user's message. Different message, different relevance ranking, different tool set: 52, then 54, then 61 tools in a single conversation. Three turns, three prefixes, three full-price cache writes.

I was paying full input price on every turn in order to save the handful of tokens I had trimmed.

The arithmetic that settles it

Pruning 1,000 tokens saves roughly $0.0006.
A cache hit saves roughly $0.014 on the same turn.

Pruning is worth about 4% of what the cache is worth. And in my case, pruning was the thing destroying the cache. The optimization was negative-value by more than an order of magnitude, and it looked like good engineering the whole time.

The root cause was worse than the symptom

The tool-count ceiling was sized by a learned factor — a running average calibrated from real usage. That same factor also fed a budget guard.

Those two consumers want opposite things. The budget guard wants accuracy: it should track reality as closely as possible. The tool ceiling wants stability: it should not move, because every move rewrites the prefix. One number cannot serve both. Worse, the learned factor was not persisted, so every deploy restarted the drift from scratch.

The fix was to split it: a fixed factor sizes the ceiling, the learned factor stays in the budget guard only, and trivial turns use a stable tool set with no per-message embedding.

Why this is not on the anti-pattern lists

Every guide to prompt caching lists the same culprits: a timestamp in the system prompt, user-specific content in the cached prefix, inconsistent whitespace, dumping raw files into context. All of those are carelessness. You read the list, you check your prompt, you move on.

Mine passed every item on that list. The thing invalidating my prefix was a deliberate, well-engineered optimization whose entire purpose was to reduce cost. That is a different failure class, and it is harder to find, because you are not looking for a bug inside the code you wrote to fix the problem.

There is prior work on the general shape of this. Don't Break the Cache (arXiv 2601.06007, Lumer et al.) evaluated caching strategies across 500 agent sessions with 10,000-token system prompts and found 41–80% cost reduction from controlling the cache boundary, plus the counterintuitive result that naive full-context caching can increase latency. Their framing is about where to put the boundary. Mine is one layer up: what in your own system is moving the boundary without telling you.

The rule I kept

Before asking how much context you send, ask what changes between turns. The system prompt, the tool schema and the tool order are a stable contract, not space to squeeze. Cost work starts by measuring cache hit rate, not size.

And a corollary that generalizes past caching: when one value serves two consumers with opposite needs, it is two values. Ask each consumer whether it wants accuracy or stability. If they disagree, split it.

This regression produces no error, never turns a test red, and is invisible in the UI. It shows up on the invoice, weeks later. So the check that catches it is not a behavioral test — it is a test that asserts the ceiling is a constant, and fails if anyone makes it learned again.