The Context-Window Arms Race
Part 2 of a chronological survey of the craft around large language models. Part 1 ended with ReAct turning a prompt into a loop. This installment covers the industry's first answer to the information problem: make the container big enough to hold everything.
TL;DR — In 2024 the bet was that a giant context window would make information management obsolete: just throw it all in. The research said otherwise. Models degrade long before the window fills, position matters as much as presence ("lost in the middle"), and every model tested gets less reliable as input grows ("context rot"). The lesson: capacity is the wrong metric. A clean window beats a big one.
The bet: if the window is big enough, you don't have to be careful
By 2024 a particular optimism had taken hold. If the central difficulty of building on language models was deciding what information to put in front of them, then perhaps that difficulty could be engineered away by sheer capacity. Make the context window — the amount of text a model can take as input at once — large enough, and you could simply put in everything: all your documents, all your tools, the entire conversation history, every instruction. No careful selection, no retrieval, no pruning. Throw it all in and let the model sort it out.
The model builders obliged, and the numbers escalated fast. The trajectory, as documented in Databricks' research and Chroma's later report, looked roughly like this:
From 4,000 tokens to 10 million in roughly four years. Each jump was announced as a qualitative change in what was possible.
The framing in the model documentation reinforced the optimism. Google's Gemini long-context materials, for instance, leaned into use cases like dropping entire long-form texts into the prompt. The implication was clear: the era of fiddly information management was ending. As Drew Breunig later summarized the mood, long contexts "kneecapped RAG enthusiasm (no need to find the best doc when you can fit it all in the prompt!), enabled MCP hype (connect to every tool and models can do any job!), and fueled enthusiasm for agents."
This bet matters to our story because it is the path not taken. The industry tried to make context management unnecessary. The discovery that it couldn't is what made context management into a discipline — the subject of Part 3.
"RAG is dead" — the debate that kept coming back
The first casualty of the optimism was supposed to be RAG.
Retrieval-Augmented Generation was introduced by Patrick Lewis and colleagues at what was then Facebook AI in 2020 (published at NeurIPS 2020). Their framing is worth stating precisely, because it explains RAG's durability.
They combined a model's parametric memory (knowledge baked into the weights of a pre-trained seq2seq model) with a non-parametric memory (a dense vector index — in their case, of Wikipedia) that the model could query at generation time. RAG models, they showed, produced "more specific and factual" output than parametric-only models.
Two of their stated motivations matter for this whole series: retrieval gives you provenance (you can cite which document grounded an answer) and updatable knowledge (you can change the index without retraining the model).
The idea, in short: instead of relying solely on what a model learned during training, you retrieve relevant documents at query time and place them in the prompt. For several years, this was the standard way to give a model knowledge it didn't have — current facts, private documents, domain-specific material.
Long context windows seemed to make RAG obsolete. Why build a retrieval pipeline to find the best few documents when you can stuff all of them into a million-token window? Each time a model shipped a dramatically larger window, a fresh round of "RAG is dead" declarations followed. Breunig notes the pattern explicitly: "Every time a model ups the context window ante, a new 'RAG is Dead' debate is born. The last significant event was when Llama 4 Scout landed with a 10 million token window."
But RAG kept not dying. The reason is the heart of this part: filling the window turned out to have costs that the "throw it all in" picture ignored.
Every bump in context size birthed a fresh "RAG is dead" obituary. RAG kept showing up to its own funeral.
The counter-evidence, part 1: performance falls long before the window is full
The most direct rebuttal came from Databricks' Mosaic Research, which in late 2024 ran a large-scale benchmark — over 2,000 experiments across 13 open and closed models on curated RAG datasets (Databricks DocsQA, FinanceBench, and the academic Natural Questions set), judged by a calibrated LLM-as-judge.
Two findings cut against the optimism. First, retrieving more documents did generally help — more retrieved information raised the odds the right answer was somewhere in the context, and capable long-context models could exploit that. So far, so good for the "more is better" camp.
But second, and decisively: more context was not always better, and most models started getting worse well before their windows were anywhere near full. In Databricks' results, Llama-3.1-405B's answer correctness began to decline after about 32,000 tokens; GPT-4-0125-preview after about 64,000; and only a handful of models maintained consistent performance across all context lengths. The majority of open-source models could handle effective RAG only up to roughly 16k-32k tokens — a small fraction of their advertised capacity.
Breunig drew the obvious conclusion: "If models start to misbehave long before their context windows are filled, what's the point of super large context windows?" His answer was that the huge windows remain genuinely useful for two things — summarization and fact retrieval — but that outside those cases, every model has what he called a distraction ceiling, a length beyond which adding more context degrades rather than improves the response.
A window big enough to hold your data tells you nothing about whether the model will use it well.
Equally revealing were the failure modes Databricks documented, because they showed the degradation wasn't uniform — different models broke in different, idiosyncratic ways. Some models, given very long contexts, would simply summarize the input while ignoring the actual instructions. And in one striking pattern, Claude 3.5's rate of refusing to answer over copyright concerns rose from 3.7% at 16k tokens to 49.5% at 64k; DBRX's instruction-following collapsed from a 5.2% failure rate at 8k to 50.4% at 32k. The same model, fed the same kind of task, behaved like a different system depending on how full its context was.
The counter-evidence, part 2: where information sits matters
If Databricks showed that performance falls as context grows, a study from the year before had already shown that models don't even use the context they have evenly.
In "Lost in the Middle: How Language Models Use Long Contexts" (Nelson Liu and colleagues at Stanford, 2023; published in TACL 2024), researchers placed a single relevant document at different positions within a long context and measured how well models could find and use it. The result was a consistent U-shaped performance curve: models did best when the relevant information sat at the very beginning or the very end of the context, and markedly worse when it was in the middle — even for models explicitly built for long context. GPT-4, the strongest model tested, achieved higher absolute scores than the rest but still showed the same U-shape.
The "lost in the middle" U-curve: relevant information is used reliably at the edges of the context and neglected in the middle.
The authors connected this to a phenomenon long known in psychology — the serial-position effect, the human tendency to best recall the first and last items in a list. Whatever its cause in transformers (attention patterns that over-weight the beginning and end of a sequence), the practical implication was uncomfortable for the "throw it all in" approach: not only does adding context eventually hurt, but a model's ability to use a given fact depends on where that fact happens to land in the pile. Position, not just presence, determines whether information gets used.
The counter-evidence, part 3: "context rot"
The clearest formalization arrived in mid-2025, after the arms race had run for a while, in a report that named the phenomenon. Chroma Research's "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (Kelly Hong, Anton Troynikov, and Jeff Huber, July 2025) evaluated 18 state-of-the-art models — including GPT-4.1, Claude 4 (Opus and Sonnet), Gemini 2.5, and Qwen3.
The headline finding was blunt and universal: models "do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows." Every one of the 18 models degraded as inputs got longer — not some, not most, all of them. This held even on tasks designed to be trivially easy, such as replicating a sequence of repeated words.
The standard industry benchmark for long context, Needle in a Haystack (NIAH), measures only whether a model can find an exact piece of text — so it produces near-perfect scores and hides the problem. When Chroma tested harder things — inferring from semantically related rather than identical information, coping with distractors, varying the surrounding "haystack" — performance fell off non-uniformly and unpredictably.
The vocabulary mattered. Context rot drew a sharp line between two things that had been conflated:
- Context-window overflow — exceeding the model's maximum token limit. A hard cliff.
- Context rot — degradation that happens well before the limit, continuously, as the signal-to-noise ratio of the input falls. A model with a 200k window can degrade significantly at 50k.
The lesson practitioners drew was that capacity is the wrong metric. A window being "big enough" to hold your data says nothing about whether the model will use that data well. What matters is keeping the ratio of relevant to irrelevant tokens high — which is precisely the thing that "throw it all in" abandons.
The winning move wasn't a bigger window. It was a cleaner one.
What the arms race actually settled
By the time the dust cleared, a rough consensus had formed among builders — not that long context windows were useless, but that they were a different tool than the marketing implied:
- Large windows are genuinely valuable for summarization and for retrieval of a specific fact from a large body of text — the cases where the model's job is to compress or to locate, not to reason carefully over everything at once.
- For multi-step reasoning, agentic work, and high-stakes accuracy, more context is a liability past a model-specific ceiling. The window being large is not a license to fill it.
- RAG did not die. The comparative research that followed (ChatQA2 and others) generally found that retrieving a well-chosen set of chunks could match or beat dumping everything in, at far lower cost — and the question shifted from "RAG or long context" to "how to combine them."
The deeper takeaway is the one that sets up everything after this point. The industry had tried to solve the information problem with capacity, and capacity alone didn't solve it. If a model degrades as you fill its context, and if it uses the middle of that context poorly, and if different models rot in different ways, then what you choose to put in the context, and what you leave out, is itself the engineering problem. You cannot opt out of it by buying a bigger window.
That realization needed a name. In mid-2025 it got one. Part 3 is the story of how a scattered set of practices — and a precise taxonomy of the ways contexts fail — crystallized into a named field: context engineering.
Key sources for Part 2
- Databricks Mosaic Research, Long Context RAG Performance of LLMs (2024) and The Long Context RAG Capabilities of OpenAI o1 and Google Gemini (2024) — 2,000+ experiments, 13 models; correctness declines after ~32k (Llama-3.1-405B) / ~64k (GPT-4-0125-preview); model-specific failure modes (summarizing instead of answering; copyright-refusal and instruction-following collapse).
- Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172; TACL 2024) — U-shaped performance curve; serial-position effect; affects even GPT-4 and explicit long-context models.
- Kelly Hong, Anton Troynikov, Jeff Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma Research technical report, July 2025) — 18 models; all degrade with input length, even on trivial tasks; distinguishes context rot from window overflow; NIAH hides the effect.
- Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401; NeurIPS 2020) — the origin of RAG; parametric (seq2seq) + non-parametric (dense Wikipedia index) memory; "more specific and factual" output; provenance and updatable knowledge as motivations.
- Drew Breunig, How Long Contexts Fail and How to Fix Your Context (2025) — the "distraction ceiling" framing and the recurring "RAG is dead" pattern; bridges into Part 3.
Next up · Part 3 — Context Engineering Is Named: Karpathy's endorsement, Breunig's taxonomy of the four ways contexts fail, and the adversarial "cat facts" that make models stumble.
Top comments (0)