Your cache bill is a layout decision, and the layout rots silently

Gal Zahavi went looking for where an agent's API bill was actually going, and found 84% of it sitting on cache reads and writes, split almost evenly. Not the input and output tokens everyone watches. The cache.

Compaction, the periodic squashing of old conversation turns to reclaim room, helped only briefly, he wrote. One request later the numbers would jump again.

That jump is not the cache failing. A prompt cache matches on an exact prefix of bytes, and compaction rewrites the earliest part of the conversation, which is the exact span a cached prefix covers. So the jump is the cache doing precisely what it says it does, and it is the clearest signal you will get that the block you were reusing just changed.

Everything about what a repeated request costs follows from that one property. Cache spend is set by where things sit in your prompt, not by how much you send.

Sort by what changes

Everything that varies per request goes below everything that does not, because a single changed byte ends the match there and you pay to write a new entry for the whole remainder. Writing a cache entry costs more than reading one. A prefix that keeps getting invalidated is a prefix you keep buying.

Sorting does not make compaction harmless, and the two are worth keeping apart. Compaction rewrites conversation history. Sorting is about the block sitting above that history, the long instructions you resend unchanged every turn. Compaction will go on invalidating what sits below. Sorting protects the part that never had to change in the first place.

The shape that punishes you hardest is a request you send over and over with a little new information each time. A polling loop does this, and so does any supervisor that re-reads its own instructions every turn.

We hit it in Favur, our multi-agent system that takes a written spec and ships a tested repository. One of its agents supervises the others. It wakes on a cycle, checks whether anyone has drifted, and goes back to sleep. Same long instructions every time, a small fresh snapshot attached.

So that prompt is sorted by one question. Does this change between cycles? The identity, the scoring rules, the definition of done and the anti-patterns do not, so they sit up top. The current snapshot, this cycle's checks and last cycle's feedback do, so they go below. In a live run the second cycle came back with a cache hit on the system prompt, which is how you confirm the boundary is where you think it is.

Sorting it once is the easy half

A boundary like that decays. Six months and four contributors later, somebody needs a cycle number inside a directive and puts it in the top half, because that is where the surrounding text lives.

Nothing fails. No test goes red. The prompt still reads correctly and still produces the right answer. The bill goes up and stays up, and the only symptom is a cache write count that climbed and never came back down, on a line item most dashboards fold into a total.

So we made the boundary a property of the code rather than a rule to remember. The builder for the cacheable half accepts no per-cycle arguments at all. It cannot vary between cycles, because there is nothing to vary it with. A test then builds that half twice and asserts the two strings match.

If you would rather see what the agents themselves get up to, our runs are public and replay in the browser. Different subject, same system.

Doing this in your own harness

None of this needs our system. It needs a prompt you send more than once.

1. Sort the prompt so everything varying per request sits below everything that does not, and treat that boundary as a real interface rather than a formatting choice.
2. Give the cacheable half exactly one source. Two places that can produce it is two places that can drift.
3. Test it by building that half twice and comparing bytes, and make sure the builder takes no per-request parameters, or the test proves determinism instead of stability.
4. Watch cache writes rather than totals. A write count that climbs and stays up is the drift signal. Reads climbing against flat writes is the cache working.
5. Write down what the test does not cover, next to the test, in the specific.

The useful thing in Zahavi's thread is not that cache can be most of a bill, though it can. It is that his numbers jumped at a particular moment, and that moment had a cause sitting above the boundary. That cause is findable, and once found, it is the kind of thing you can hold in place with a test that costs almost nothing to run.