How Long-Horizon Agents Kept Busting the Prompt Cache

OpenAI launched GPT-6 Sol and Luna yesterday, cutting API prices by 50% compared to GPT-5.6. Sol now sits at $2 per million input tokens and $10 for output, while Luna drops to $0.10 and $0.50. Most feeds spent yesterday debating the benchmark cards against Claude Opus 5 on AutomationBench and DeepSWE.

The more telling metric was buried in OpenAI's research acceleration report. At internal API rates, daily token burn exceeded $600 for their median researcher and $7,000 for the top ten percent.

When you run autonomous coding loops, token costs compound because agent harnesses carry heavy context across dozens of iterations. System contracts, tool schemas, repository maps, and execution logs accumulate. Prompt caching offers a 90% discount on reused input tokens, but production harnesses routinely break that cache on turn three.

Three harness habits cause most of those cache misses:

1. Changing tool schemas. When an agent enters a specialized phase, many frameworks prune the tool array to prevent tool hallucinations. Modifying the tool definitions array changes the prompt prefix, which forces a full re-computation of every token that follows.
2. Dialing reasoning effort. A planner turn might need maximum reasoning effort, while a command runner only needs minimal effort. In older API revisions, changing request-level reasoning parameters modified the inference state and invalidated cached prefix tokens.
3. Inserting dynamic instructions mid-context. Appending progress checkpoints or changing operational constraints near the top of the context window shifts the token stream, discarding any existing cache blocks downstream.

The GPT-6 caching update introduces four API controls aimed directly at persistent harnesses:

The first change is configuration_update. Instead of altering the top-level request parameters to switch reasoning effort, an agent harness can append a configuration update block to the context. This adjusts reasoning depth for the next turn while preserving the cached prefix behind it.

The second change is allowed_tools. Rather than rewriting tool lists between turns, harnesses can declare their entire tool registry once in the system prompt. Passing allowed_tools or setting tool_choice to none restricts callable tools on that specific step without altering the static schema definitions in the prompt prefix.

The third change is explicit breakpoints. Harness authors can mark the exact boundary where static context ends and volatile scratchpad data begins. Everything above the breakpoint stays eligible for 30-minute caching discounts even when lower context blocks shift.

The fourth change is cache prewarming. Harnesses can push shared environment definitions, codebase indexes, and system instructions to the cache during startup before receiving user input. That moves token processing out of interactive user wait time.

According to GitHub, applying these prefix preservation controls reduced fresh prompt token processing by more than half across Copilot requests.

If you maintain an agent harness, keeping prefixes stable matters as much as model choice. Place your static instructions and full tool registry at the front, keep schemas immutable, toggle permissions with allowed_tools, and append configuration changes rather than modifying top-level headers.

Comments

[Max Quimby]

The tool-schema-mutation one is the trap that gets everyone, because pruning the tool array feels like the responsible thing to do — you're reducing hallucination surface. But it moves the cache breakpoint to the very top of the prefix, so you pay full price on everything downstream for the rest of the session. Declaring the full registry once and gating with allowed_tools / tool_choice is the fix, and it's a genuine mindset shift: static schema, dynamic permissions.

The dynamic-instruction-injection point is the subtle one I'd underline. Appending a progress checkpoint near the top of context feels harmless, but it shifts every token after it. We moved all of that volatile scratchpad below an explicit breakpoint and treated everything above it as immutable — cache hit rate jumped once the "top" of the prompt genuinely stopped changing.

One question: with configuration_update letting you dial reasoning effort mid-stream without invalidating the prefix, are you seeing any planner/executor turns where the effort switch alone justifies a separate cache segment anyway? Curious where the crossover lands for you.

[Reid Marlow]

The crossover mostly hinges on how much reasoning trace the planner generated. In a unified chain, you pay the cached input fee on every planner token for each subsequent executor step. If a high-effort planner emits 4k tokens of decomposition, carrying that weight across twenty fast shell commands adds up. For loops longer than five or six tool turns, splitting out a lean executor session with just the distilled task spec ends up cheaper than dragging the planner scratchpad along at the ten percent cached rate. For short three-turn fixes, keeping the unified prefix and dropping down via configuration_update wins every time.

[AI Frontier Post]

The interesting tension here is that every one of those cache-busting habits started life as a genuine best practice — pruning tool schemas and injecting checkpoint summaries are exactly what agent-framework tutorials recommended two years ago. The allowed_tools + static schema approach is the cleaner mental model: static schema, dynamic permissions. One caveat I'd add: explicit breakpoints and prewarming push complexity into the harness, and the harness is where the bugs compound fastest. The real win may be measurement — most teams have no idea what their actual cache hit rate is, so prefix stability stays a vague aspiration rather than a budgeted number.

[Reid Marlow]

The harness complexity point is real. Prewarming sounds clean until the background ping races a user turn, or your proxy splits prompt blocks differently than the provider expects. Tracking cache_creation_input_tokens against cache_read_input_tokens per turn is what turned this into an engineering problem for me. Once you put hit rate on a dashboard, you immediately catch when a helper function starts appending an unsorted dict or a localized timestamp.

[jkming]

The tool-pruning trap got us too — gating tools per phase felt like the responsible move until we noticed the cache hit rate collapse in the logs. One thing that bit us on top of this: TTL. If a long-horizon loop parks between phases (human approval, a slow CI gate), the prefix can expire even when it never changed, and the next turn pays full price. We started logging hit rate per turn instead of assuming stable prefix equals savings. Curious how much of the Copilot gain came from configuration_update vs allowed_tools — the reasoning-effort flip was the harder one for us to work around.

[Reid Marlow]

The TTL cliff during CI or human approval is brutal. If an eval suite runs forty minutes, you return to a completely cold cache despite having a byte-identical prefix. We ended up keeping the long-wait background tasks in detached worker sessions so the interactive prompt prefix didn't sit around waiting to expire.

On the Copilot breakdown, the reasoning flip was definitely the heavier hit. Their workload toggles constantly between high reasoning for architecture planning and low reasoning for terminal execution. When that was a top-level parameter change, every single handoff threw away the cached codebase index. allowed_tools is cleaner hygiene, but configuration_update is what stopped the five-turn cache reset cycle.

[Edward Izgorodin]

The 90 percent in the article is the read side of the bill. On the models that support explicit breakpoints, a cache write is billed at 1.25 times the uncached input rate, so a bust is no longer only a lost discount. A token that is written and never read costs more than the same token sent uncached, while writing it once and reading it once already comes to 1.35 times the plain price instead of 2.

That changes the price of the third habit. In implicit mode the breakpoint sits at the end of the latest eligible message, so a volatile block in the middle of the context means every turn writes the whole tail below it at the premium, and the next turn reads none of it. Putting an explicit breakpoint above the scratchpad does not stop the scratchpad itself from being written, because implicit mode keeps its own breakpoint at the end. Explicit-only mode does: content after the last selected breakpoint is billed at the plain input rate and never written.

You already track creation against read per turn, and the check I would add is the same pair offset by one turn: compare the cache_write_tokens of each turn with how much cached_tokens grows on the next one. In a single append-only loop the two should match, and I would read any gap as tokens paid at the write rate that nobody read. Unlike a hit rate, that gap prices the miss in tokens and sits on the turn that paid for it.

[Cor E]

This is a nice article Reid thanks. For some of my stuff, I was working around this before by declaring 4 harness states, and caching them seperately. Caching saves a lot of money, but having 4 states busts the cache anytime you need to change your state's posture. These kinds of changes are much needed.

Cheers!

[Reid Marlow]

The multi-state split is exactly where cache hit rates fall off a cliff. Once you maintain separate prefix chains for planner, coder, and reviewer postures, every mode switch is effectively a cold prompt unless the agent loops in the same state for several turns. Sticking the entire tool catalog behind an immutable prefix and masking capabilities per step with allowed_tools finally lets you keep a single hot cache line across the whole run.