A model advertising a 200,000-token context window can start falling apart at 50,000 tokens. It won't throw an error. It won't flag a warning. It will just get worse, fluently and confidently, on a problem you can only catch by checking its work later.
This isn't a hypothetical. Chroma's 2025 research tested eighteen frontier models, including OpenAI GPT-4.1, Claude Code Opus 4, and Google Gemini 2.5, and found that every one of them degraded as input length grew, often well before the window was anywhere near full. Researchers gave this a name: context rot, a continuous decline in output quality that has nothing to do with hitting a limit.
Most teams don't know this exists. They treat the context window like disk space: bigger is better, and if something feels off, the fix is to feed the model more. A full ticket history instead of a summary. An entire file instead of the relevant function. Three months of chat logs instead of last week's. Each addition feels like it should help. Often it makes things worse, and nothing in the product tells you that it did.
The Capacity Myth
The assumption underneath most AI products is simple: more context means a smarter, more grounded system. Stack the tokens, the model wins.
The research says otherwise. In 2023, Stanford University researchers led by Nelson Liu tested how models handle multi-document question answering and found a U-shaped accuracy curve. Models retrieve information well when it sits at the start or end of the input. When the relevant fact lands in the middle, accuracy drops by more than 30 percent. They called it the lost-in-the-middle effect, and it has since replicated across six model families.
Chroma's broader study identified two more mechanisms compounding the problem. Attention dilution comes from how transformer attention scales: a 100,000-token input creates roughly 10 billion pairwise relationships competing for the model's focus. Distractor interference is sneakier. Content that's topically related but irrelevant doesn't just sit there harmlessly. It actively misleads the model toward the wrong answer.
Context window overflow is a hard stop: tokens get truncated, and you find out immediately. Context rot is the opposite. Performance erodes while everything still appears to be working, which is exactly why most teams don't think they have the problem.
None of this requires hitting the window's stated limit. A model with 200,000 tokens of capacity can show measurable degradation by 50,000. The decline is continuous, not a cliff, and that's exactly why it goes unnoticed until someone checks.
The real metric was never capacity. It's signal-to-noise: how much of what's sitting in the window actually deserves the model's attention right now.
The Budget Nobody Is Managing
Once signal-to-noise is the metric, the context window stops looking like storage and starts looking like a budget. Every product built on an LLM allocates a fixed, shared space across competing claims on that attention:
The system prompt and instructions
Retrieved context: documents, files, search results
Conversation history
Tool outputs: API responses, file reads, search results
Working memory the model uses mid-task
A team that decides to retrieve the top 20 documents instead of the top 5, or to keep full chat history instead of summarizing it, is deciding what the model pays attention to instead of something else. That's the same category of decision as choosing which features make a release. Most teams don't experience it that way. They experience it as a retrieval setting somebody configured once and never revisited.
This is also where the five-dimension framework from earlier in this series gets concrete. Context budget lives inside the interaction model dimension: not whether a product feels conversational, but whether what actually reaches the model at each turn is the right input for the job. A system can score well on every other dimension and still fail here, because nobody assigned ownership of what goes into the window.
The size of the bill is the easiest part of this to track, and the part most teams actually do track. API pricing runs largely per token, so a retrieval step that grabs 30 documents instead of 8 isn't more thorough, it's a standing tax on every call the product makes. At a million queries a month, the gap between a tight context and a bloated one shows up on the margin line, not the engineering backlog.
The Silent Failure, Again
We've made a version of this argument before in this series: eval failures are usually sequencing problems, not metric-selection problems. Skip the detection layer, and a system degrades invisibly until a user notices. Context bloat fails the same way, one layer over, and it needs the same kind of production monitoring this series has already argued for, just watching a different signal.
No error fires when context degrades. The response still looks plausible. It's fluent, it's confident, and it's often wrong in a way only the user catches, because the system has no idea its own attention has been diluted.
Picture a support agent fed a customer's full ticket history instead of a structured summary. Forty messages in, it recommends a fix it already tried, that the customer already rejected. The model didn't get a worse weight update overnight. Its context just got noisier, and nobody was watching for it.
How AI Native Products Actually Manage the Budget
Cursor and Claude Code make a useful contrast, because both face the worst version of this problem: real codebases generate far more text than any context window can comfortably hold.
Claude Code's answer is auto-compaction. As a session approaches its limit, it summarizes the older exchanges and replaces the raw history with a condensed record before forcing a hard reset. Practitioners tracking this closely have found that compacting earlier, around 60 to 75 percent of capacity rather than waiting for the automatic trigger near 95 percent, produces longer and higher-quality sessions. One developer monitoring usage independently found Claude Code reporting only 10 percent of capacity left while his own tracking showed 64 percent had actually been used, a 54-point gap traced back to deliberately conservative compaction thresholds. The unused space wasn't waste. It was headroom protecting signal-to-noise as the session went on.
Cursor takes a different angle on the same problem. Instead of expanding the window to fit a growing codebase, it indexes the codebase into a vector database and retrieves only the chunks semantically relevant to the current query. A natural-language question pulls back a handful of relevant files instead of the entire repository, assembled into context just for that request. The model never sees more than it needs, by design, not by accident.
Neither product asks how much it can fit. Both ask what deserves to be there. That's the actual skill underneath context window economics, and it has nothing to do with how large the advertised window is.
Why This Belongs on the Roadmap
There's a trust cost to context rot that's quieter than the API bill, and harder to track. Users rarely file a bug report that says the model's context got noisy around message forty. They just notice the product got worse over a long session, stop trusting it with anything that matters, and churn without explaining why.
The model didn't get dumber. Its context got noisier.
Ownership matters here too. If the context budget isn't anyone's explicit job, the way conversion rate or onboarding flow is somebody's job, it drifts by default toward "more," because more feels safer than a decision someone has to defend. The instinct to fix a struggling AI feature by feeding it more is the same instinct as fixing it by upgrading to a bigger model. Both substitute scale for design. In most cases, the actual fix is a smaller, better-curated context, and that's a decision that belongs with whoever owns the product, not buried in a retrieval config nobody revisits after launch.
Every additional token placed in front of a model is a decision about what it should attend to instead of something else. Treat it with the discipline of a feature cut or a pricing tier, not a default setting left over from launch week. The teams getting this right don't have the biggest context windows. They decided, on purpose, what doesn't get to be there.
Top comments (0)