In Part 2 I made a claim and then walked past it: a single cache-busting mutation in a 30-turn conversation can 10x the cost of that conversation. This post is what that means.
The prompt cache is the difference between an autonomous agent you can afford to run constantly and one you can't. At a $20/month budget, cache discipline isn't an optimization you get to after the features land. It's an invariant you design around from the first line, the same way you'd treat a memory-safety rule or a SQL-injection guard.
Every way you'd break it looks reasonable: injecting a timestamp, reformatting the system prompt, adding a tool mid-conversation, trimming old messages. Each one silently torches your cache-hit rate for the rest of the session.
This post covers the rules: why prefix caching keys the way it does, the four mutations that break it, the operations that are safe, and how to enforce it statically so violations fail loud.
How Prefix Caching Actually Keys
Every compatible provider caches on the prefix of the request. Not the whole request: the prefix, evaluated in order:
┌─────────────────────────────┐
│ 1. System prompt │ ← cached prefix
│ 2. Tool schema (definitions)│ ← cached prefix
│ 3. Message history (turns) │ ← cached prefix
├─────────────────────────────┤
│ 4. The new user turn │ ← uncached (this is fine)
└─────────────────────────────┘
The provider walks that prefix from the top and matches it, token by token, against what it cached on the previous request. As long as the bytes are identical up to some point, you get a cache hit for everything up to that point. You pay roughly 10% of the input cost for the cached span, and the response comes back about 3x faster because the model doesn't re-process tokens it already has warm.
The catch is the phrase "up to some point." The match is a prefix match, not a set match. The instant the bytes diverge (one changed character in the system prompt, one reordered tool, one edited word in an old message) the cache match stops at that point , and everything from there to the end of the request is a cache miss. Re-priced at full rate. Re-processed at full latency.
Think of the prefix as append-only and frozen. Everything before the newest user turn is set in concrete the moment you send the first request. You may add to the end; you may not reach back and touch anything earlier. Every rule in this post is a consequence of that one property. Once you internalize "the prefix is frozen," you stop seeing four separate rules and start seeing one.
This is also why the cost blows up the way it does. A cache-busting mutation doesn't make one turn expensive. It moves the divergence point earlier in the request, so the miss applies to the rest of that turn, and because the mutated prefix is now what gets cached, the same expensive prefix can keep missing on subsequent turns too if the non-determinism repeats. In a long conversation, you're not paying a one-time penalty. You're paying the penalty over and over, on a request that keeps growing.
The Cost Math, Concretely
A conversation's input cost grows with each turn, because every turn resends the entire history as context. Turn 1 sends a small prefix. Turn 30 sends 30 turns' worth. With caching working, almost all of that resent history is a cache hit. You pay ~10% on the warm span and full rate only on the new turn. That's what makes long agent conversations affordable at all.
Now break the cache. Suppose something non-deterministic, say a timestamp in the system prompt, changes on every single turn. The system prompt sits at the very top of the prefix, so a change there moves the divergence point to byte zero. Every turn is now a full cache miss on the entire history. You've gone from paying ~10% on the growing context to paying 100% of it, every turn, for the life of the conversation.
That's the shape of the 10x: the compounding cost of resending an ever-larger history at full price across a 30-turn conversation, when you could have been resending it at a tenth of the price.
This is what makes cache discipline a security-grade concern rather than a tuning knob. The agent still works. The output is identical. Nothing errors, nothing logs a warning, no test goes red. The only signal is the bill, and by the time the bill arrives, the unattended agent has been running the broken pattern thousands of times. You don't get to discover this in code review unless you're specifically looking for it, which is the entire argument for enforcing it statically, later in this post.
On a $20/month self-evolution budget, the math isn't academic. A cache-busting bug doesn't make the agent "a bit pricier." It makes the difference between an agent that runs every 15 minutes forever and one that trips its own budget kill switch by lunchtime. The cost ceiling is what lets the agent be autonomous in the first place, so anything that quietly multiplies cost is an attack on the autonomy itself.
The Four Forbidden Mutations
Here's the spine of the whole discipline. Four mutations, each one a way of reaching back into the frozen prefix. Each looks innocent. Each isn't.
1. Mutating the system prompt after the first dispatch
The system prompt is the very top of the prefix. Touch it and you've moved the divergence point to byte zero: the most expensive possible miss, because nothing after it can hit the cache. Once call_llm has been invoked once with a system prompt, the exact same string has to be reused on every subsequent turn of that conversation. Byte for byte.
The violations don't look like "mutating the system prompt." They look like helpfulness:
// ❌ LOOKS INNOCENT — concatenating "just a little" context
const system = BASE_SYSTEM_PROMPT + `\n\nCurrent focus: ${task}`;
// ^^^ different string every turn = byte-zero miss
// ❌ LOOKS INNOCENT — "harmless" reformatting between turns
const system = prettyPrintMarkdown(BASE_SYSTEM_PROMPT);
// ^^^ even whitespace changes are byte changes
// ❌ LOOKS INNOCENT — telling the model what time it is
const system = `${BASE_SYSTEM_PROMPT}\nIt is ${new Date().toISOString()}.`;
// ^^^ guarantees a miss EVERY turn
// ✅ CORRECT — one frozen string, captured at conversation start, reused verbatim
const system = BASE_SYSTEM_PROMPT; // identical bytes, every turn, no exceptions
The fix is a discipline, not a clever trick: capture the system prompt once, freeze it, and never let anything append to it, reformat it, or inject into it mid-conversation. If the model needs to know the current task or the current time, that information goes in the last user turn , the one part of the request that's supposed to change.
2. Mutating the tool schema mid-conversation
The tool schema sits right below the system prompt in the prefix. Adding a tool, removing a tool, or even reordering the tools between turns busts the cache from the tool-schema boundary onward. And it has a second cost the other mutations don't: it confuses the model's tool planning. A model that learned the tool set on turn 1 and finds a different one on turn 5 plans worse, not slower.
The trap here is the instinct to be "smart" about tools: hide the ones that aren't relevant, surface new ones on demand:
// ❌ LOOKS INNOCENT — dynamically narrowing tools per turn to "help" the model
const tools = relevantToolsFor(currentTask); // different list each turn = cache miss + worse planning
// ❌ LOOKS INNOCENT — appending a tool because the model seemed to want it
tools.push(newlyDiscoveredTool); // mutates the schema mid-conversation
// ✅ CORRECT — decide the full tool set ONCE, at conversation start, then freeze it
const tools = resolveToolSetForSession(); // computed once per session, never per turn
The rule: decide the tool set once per session, not per turn. If you need different tools, that's a different conversation. This is exactly the cliffhanger I left at the end of Part 2 and the whole subject of the next post on MCP in production (November 15): your tool schema is part of the cache prefix, which means the common pattern of wiring tools in and out of an agent mid-session is quietly a cost bug. There's a lot more to say there; it gets its own deep dive.
3. Rewriting message history
Every prior user and assistant turn is part of the frozen prefix, in order. Editing any earlier message, to compress it or redact it or "improve" it, moves the divergence point to that message and invalidates everything after it. The earlier the message you touch, the bigger the miss.
This one is seductive because it wears the mask of a good idea: context is getting long, so trim the old turns to save tokens.
// ❌ LOOKS INNOCENT — compressing old turns to save context window
history[2].content = summarize(history[2].content);
// ^^^ rewriting turn 2 invalidates turns 3..N on the very next request
// ❌ LOOKS INNOCENT — redacting something after the fact
history = history.map(redactSecrets); // every edited turn shifts the divergence point earlier
// ✅ CORRECT — history is append-only; never reach back and edit
history.push(newUserTurn);
history.push(newAssistantTurn); // the prefix grows at the end, never in the middle
You don't "improve" history. You append to it. If the conversation genuinely needs to get shorter, that is the compress operation I describe in the permitted section, not an edit, which ends the conversation rather than mutating it.
4. Non-deterministic content in the prefix
This is the umbrella that the timestamp example lives under, and it's worth naming on its own because it shows up in places you don't expect. Anything non-deterministic baked into the system prompt or an early message ($(date +%s), a random request ID, a UUID, an environment dump, a hostname, a PID) guarantees the prefix differs every run, which guarantees a miss every run.
// ❌ LOOKS INNOCENT — a request ID "for tracing," placed in the system prompt
const system = `${BASE_SYSTEM_PROMPT}\nrequest_id: ${crypto.randomUUID()}`;
// ❌ LOOKS INNOCENT — dumping the environment so the agent "has context"
const system = `${BASE_SYSTEM_PROMPT}\nenv: ${JSON.stringify(process.env)}`;
// ^^^ also a credential-leak waiting to happen
// ✅ CORRECT — non-deterministic content belongs in the LAST (uncached) user turn
const userTurn = `request_id: ${crypto.randomUUID()}\n\n${actualMessage}`;
// the prefix stays byte-stable; the only thing that changes is the part that's supposed to
The principle generalizes: if it changes between runs, it goes in the last user turn, never in the prefix. The final turn is uncached anyway, so that's the correct home for everything dynamic. As a bonus, keeping environment dumps out of the system prompt closes a credential-leak vector at the same time, which is why I treat this rule as security hygiene as much as cost hygiene.
Notice that all four are the same violation: reaching into the frozen prefix and changing it. System prompt, tool schema, message history, dynamic content: they're four locations where the prefix lives. If you only remember one sentence from this post, make it this: the prefix is append-only; everything dynamic goes in the last turn. Every forbidden mutation is a corollary of that, and every permitted operation respects it.
The Operations That Are Actually Permitted
Rules that only say "no" leave people guessing about the legitimate cases. There are exactly three things you are allowed to do, and between them they cover everything a real agent needs.
Append new turns (the normal mode)
This is the whole point of the cache. You add a new user turn, the model responds, you append its assistant turn, and the prefix (everything before the new turn) is byte-identical to last time. So it all hits the cache. You pay full rate only on the new turn and ~10% on the warm history.
// The 95% case. The prefix is untouched; only the tail grows.
history.push({ role: 'user', content: nextMessage });
const reply = await call_llm({ system, tools, messages: history });
history.push({ role: 'assistant', content: reply });
If your agent does nothing but this, you have already won the cache game. The forbidden mutations are all deviations from this happy path; the discipline is mostly about not deviating.
Replace the entire conversation
Sometimes you genuinely need a different system prompt, a different tool set, or a clean slate. That's allowed, as long as you replace the whole conversation rather than mutating the existing one. A fresh session starts a fresh cache prefix. You expect a cache miss on the first turn, you keep nothing stale around, and you're back on the append-only happy path from turn 2 onward.
The distinction that matters: replacing the whole conversation is fine; editing the existing one is the violation. New session, new prefix, clean. Same session, edited prefix, broken.
A single, marked compress step
This is the sanctioned answer to "my context is getting too long." You do not solve it by editing history (forbidden mutation #3). You solve it with a deliberate compress step that ends one conversation and seeds a new one with a summary.
Conversation A (long, getting expensive)
│
▼
[compress] ──► produce a summary of A
│
▼
Conversation B (fresh prefix; seed message = the summary)
The compress step is the only sanctioned write path to the system prompt between conversations. It's a clean boundary: conversation A is finished and discarded, conversation B starts fresh with a summarized seed, and B's prefix is frozen from its own turn 1. Nothing in A's prefix is ever mutated. You took the cache miss intentionally, at a moment you chose, instead of suffering it accidentally on every turn.
In my own daemon, this is what the handoff between sessions looks like. When a conversation has earned a compression, it ends, summarizes itself, and seeds the next one. It's the difference between retiring a conversation and corrupting it.
There are three sanctioned operations and no others: append, replace-whole, compress-and-reseed. If a change to the request you're about to send isn't one of those three, it's a forbidden mutation in disguise, so go find which of the four it is. A closed permitted set is what makes the rule enforceable: you're not asking "is this change okay?" every time, you're asking "is this one of the three allowed moves?"
Enforcing It So Violations Fail Loud
Because this failure is silent, humans will eventually forget the rule. So I don't rely on memory. I make the cheap-correct thing the default and make violations fail loud: before the code merges, and while it runs.
Static enforcement in pre-commit and CI
The first line of defense is a validator that runs in pre-commit and again in CI. It's not clever. It greps the source for the forbidden patterns and fails the commit if it finds them:
-
System-prompt concatenation —
system_prompt + …, template-literal interpolation into the base prompt, any reassignment of the captured prompt after first use. -
Non-deterministic content in the prefix —
date,Date.now(),randomUUID,process.envdumps, PIDs, hostnames anywhere in the system-prompt construction path. -
Mid-conversation tool mutation —
tools.push(...),tools.filter(...), per-turn tool resolution inside the conversation loop instead of once at session start. -
History rewriting — assignment into
history[i],.map()over the message array that returns mutated content, anything that writes back into an existing turn.
A grep-based static check has obvious blind spots (it can't catch a violation laundered through three layers of indirection) but it catches the overwhelming majority, which are exactly the "looks innocent" one-liners from the forbidden section. The goal isn't a proof; it's a tripwire on the common mistakes, sitting at the cheapest possible place to catch them: before they ever merge.
Runtime audit on the actual prefixes
Static analysis can't see what your code actually sends, so the second line of defense is a runtime audit that inspects the real request prefixes. It records the prefix each conversation sends and checks the invariant that matters most: within a single conversation, the system prompt and tool schema must be byte-identical across turns, and the message history must be a strict append of the previous turn's history. If turn 5's prefix isn't a clean extension of turn 4's, something mutated it, and the audit flags exactly where the divergence point landed.
This catches the cases grep can't: the laundered mutation, the third-party library that reformats your prompt behind your back, the subtle reordering that only shows up at runtime. Static catches it cheap and early; runtime catches it true and complete. You want both.
The entire enforcement philosophy is one move: convert a silent, expensive, delayed failure into a noisy, cheap, immediate one. A cache-busting mutation that would have surfaced as a mysterious 10x bill three weeks later instead fails your commit in three seconds, or trips a runtime audit on the first cycle it runs. That's the whole game. You can't remember your way out of a silent failure mode at scale. You have to instrument your way out of it.
Where Cache Discipline Bites
Same honesty as the rest of this series: here are the places this discipline is genuinely hard, and the traps I've actually hit.
Trap 1: "Just a little context" in the system prompt
The single most common violation, and the most tempting, because it feels like good prompt engineering. You want the model to know the current task, the active file, today's date, so you append it to the system prompt where the model "will definitely see it." That append is a byte-zero cache miss every turn. The fix is muscle memory: dynamic context goes in the last user turn , which the model perceives with equal clarity and which doesn't touch the cache.
Trap 2: Compression by editing instead of reseeding
Context gets long, tokens cost money, the obvious move is to trim old turns in place. It's the wrong move: editing history is forbidden mutation #3, and it busts the cache for everything after the edit, so you spend more to "save" tokens. The right move is the compress step: end the conversation, summarize it, seed a fresh one. Editing corrupts; reseeding retires.
Trap 3: Dynamic tools that seem helpful
Narrowing the tool list to "what's relevant this turn" feels like you're helping the model focus. It isn't: it busts the schema cache and degrades tool planning, a double cost. Decide the tool set once per session. If you truly need a different set, start a different conversation. (The full version of this trap is the entire MCP-in-production post, coming November 15; it's the load-bearing constraint there.)
Trap 4: Trusting the framework not to mutate your prefix
You can follow every rule and still get burned by a library that reformats your system prompt, injects a timestamp, or reorders tools "for you" between calls. This is exactly why the runtime audit exists: static checks only see your code. If your cache-hit rate is mysteriously bad and your source looks clean, audit the actual bytes on the wire. The mutation is usually something you didn't write.
Every trap is the same shape as the four forbidden mutations: a reasonable-looking reach into the frozen prefix. The discipline isn't about being clever. It's about resisting a specific, recurring temptation to touch the prefix when appending to the tail would do. Resist it, instrument against it, and the cache takes care of itself.
Conclusion & Key Takeaways
Cache discipline is invisible when it works and brutally expensive when it doesn't. The model stays fine while your bill silently climbs. Treat it as an invariant, not a tip.
How The Cache Keys
- Providers cache on the prefix : system prompt → tool schema → message history, matched in order.
- A hit is ~90% cheaper and ~3x faster than a miss.
- The match is a prefix match: diverge at any point and everything after it is a full-rate, full-latency miss.
The One Rule
The prefix is append-only and frozen. Everything dynamic goes in the last user turn. The four forbidden mutations are all the same violation, reaching back into the prefix, wearing four masks: system-prompt edits, tool-schema changes, history rewriting, and non-deterministic content.
The Three Permitted Moves
- Append new user + assistant turns (the normal mode, which hits the cache).
- Replace the entire conversation (fresh session, intentional miss, nothing stale kept).
- Compress and reseed — end one conversation, summarize it, seed a new one. The only sanctioned write path to the system prompt between conversations.
How To Keep Yourself Honest
- A static validator in pre-commit + CI that greps for the forbidden patterns: cheap and early.
- A runtime audit that checks the real request prefixes are byte-stable and append-only: true and complete.
- Together they convert a silent 10x bill into a loud three-second failure.
Why It's Load-Bearing
At a $20/month budget, the cost ceiling is what lets the agent be autonomous. Cache discipline is what holds that ceiling. Break it and you don't get "a pricier agent." You get an agent that trips its own budget kill switch and stops being autonomous at all. Cheap is the feature.
In this series:
- Part 1 — AI Assistants as Development Partners: the daily workflows where a human stays in the loop.
- Part 2 — Autonomous AI Agents: From Assistants to Automation: the leap to an agent that runs unattended, where this cost constraint first showed up.
- Part 3 — MCP in Production: Wiring Tools Into Agents Without Busting the Cache (November 15): the tool-schema-as-prefix problem in full, and adding capabilities without quietly wrecking cost.
Running an agent on a tight budget, or fighting a mystery bill? Tell me where your cache-hit rate went and what you found when you audited the bytes on the wire. The silent failures are the ones worth comparing notes on.
Top comments (0)