Last month I changed one config value from 32k to 200k and told myself the agent got smarter. It didn't. I moved the cost curve to the right and gave the model more room to get confused. The "you don't need long context" result making the rounds says the same thing more politely: per-token cost climbs steeply as you go deeper into the window — several times the shallow rate by the deep end — while compacting aggressively back to short contexts costs almost nothing in accuracy on long-horizon software engineering tasks.
That's not a benchmark finding. That's a deployment finding that got dressed up as one.
What we actually believed
The myth is simple and I held it for a while: bigger window = more capable agent. Feed it the whole repo, the whole conversation, every tool result, and let attention sort it out. Context length became a spec-sheet number we compared the way people compare GHz — a proxy for capability because it's the number that's printed on the box.
The reason this belief survived contact with reality is that it's locally true. Give an agent more room and short tasks get easier. You stop writing the plumbing. You stop deciding what matters. Everything fits, so you never build the machinery that decides what doesn't.
Then the session runs long, and you find out what you actually bought. Not intelligence. A longer prompt.
The cost curve is the real feature
Here's the part that made me sit up: per-token cost isn't flat across the window. Deeper positions cost more — the reported curve climbs toward 7x by the far end. That's prefill attention and KV cache doing what they do mathematically. It's not a vendor being greedy; it's the shape of the computation.
Do the back-of-envelope on my own loop. A twenty-turn agentic task where each turn appends tool output, and by turn twenty the prompt is 150k tokens. If the tail of that window costs several times the head, the last few turns dominate the bill — and they're the ones where the model is least reliable anyway, because the thing it needed is buried somewhere in the middle. You pay a premium for tokens the model is worst at using.
Compaction inverts it. Keep the live context at 6–10k tokens, rewrite state at boundaries, and cost becomes roughly linear in the number of turns instead of climbing with the square of your session length. Same task, same model, different curve. The window size was never the lever. Where you cut was.
The decision is where in the loop, not how big
This is the part I wish more posts would say plainly, because "just compact" is useless advice without a boundary. In my agents, compaction fires at one of these:
At task boundaries. The agent finishes a subgoal, I have it write a short state note — what changed, what's still open, what it tried and abandoned — then restart clean with that note. Most reliable. Costs me a model call I have to write the prompt for.
At tool-call boundaries. Cheaper, noisier. Keep the last few tool results verbatim (the model genuinely needs the raw error string), collapse everything older into a line. Works well for build-and-fix loops where the newest output is the only output that matters.
On a token budget. Trigger at 60% of your declared limit, not 95%. If you compact at the wall, one oversized tool result puts you over and the run fails mid-step. I learned this the annoying way, twice.
Never in the middle of a multi-step edit. If the agent is six calls into rewriting a module, compacting halfway loses the plan and it restarts the plan badly. Boundary discipline matters more than summarization quality.
And the compaction target shouldn't live in the context window at all. Write it to disk. A scratchpad file the agent re-reads is a real artifact: inspectable, diffable, and it doesn't get evicted by the next tool result that dumps 40k tokens of test output.
Where this myth-busting overcorrects
Long context is still the right tool for some jobs, and I'd rather say so than join a bandwagon. Single-shot long-document QA, one big diff you need reviewed as a unit, a legal contract you can't summarize without losing the clause that matters — compaction there is lossy in a way that costs you correctness. The result is specifically about long-horizon agentic software work, where the state is mostly reconstructible from the repo. That's the load-bearing assumption. If your task's state isn't reconstructible from artifacts, compaction will quietly eat it.
Which is really the honest lesson: compaction isn't a summarization trick, it's a state-management problem. The teams that do it well aren't writing clever summaries. They're making sure nothing important only exists in the context window.
What I'd actually configure
Short default context. Pinned instructions at the head so they stay in attention range. A durable state file outside the window. Compaction at subgoal boundaries with a budget trigger as a safety net. Raw tool output kept verbatim for exactly as long as it's actionable, then collapsed to a fact.
And then measure the thing you actually care about: cost per completed task, not tokens per call. Mine dropped noticeably when I stopped buying window and started writing boundaries. The agent didn't get smarter. It got cheaper and less confused, which in production is the same thing.
If you take one thing from the long-context result, take the reframe. Stop asking how big a window to buy. Start asking what survives the cut, and where you're going to make it.
Top comments (0)