Every time a model ships with a bigger context window, someone posts "RAG is dead." Gemini's docs now describe context windows of 1 million tokens or more. So why bother chunking, embedding, and ranking when you can paste in everything?
My take: long context changes how much you retrieve, not whether you retrieve. Retrieval in a real product is a relevance system and a permissions system. A bigger window replaces neither.
The case for "just stuff it" (and where it's right)
Credit where it's due. Anthropic's Contextual Retrieval post says it plainly: if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can include the whole thing in the prompt with no RAG at all. With prompt caching, they report latency cut by more than 2x and costs by up to 90% for repeated prompts.
So if you have one product manual, one contract, or one small codebase that rarely changes, skip the vector database. Fewer moving parts is a real win.
But the same post goes on: "as your knowledge base grows, you'll need a more scalable solution." Most company knowledge (tickets, wikis, Slack exports, PDFs, per-customer data) is not a 500-page static bundle. That's where the "RAG is dead" argument falls apart.
1. Cost and latency are paid per call
A context window is a limit, not a free tier. Google's long-context guide is direct about it: you pay the input token cost every time you send the query, and longer queries generally mean higher latency (time to first token). Its advice: "if you don't need tokens to be passed to the model, it is best to avoid passing them."
Caching helps when the same big prefix is reused. It does not help when every user, every tenant, or every question needs a different slice of a large corpus. Retrieval is how you send 20 relevant chunks instead of 800 pages on every request.
2. Bigger windows don't mean better recall
The "Lost in the Middle" paper (Liu et al., arXiv 2307.03172) found a U-shaped curve: models use information best at the start or end of the context and worst in the middle. For GPT-3.5-Turbo, with the answer buried mid-context, multi-document QA accuracy fell below its closed-book score of 56.1%, worse than giving it no documents at all. The extended-context versions of the models were not better at using the same input.
I covered the prompt-ordering fix for this on 2026-10-12. The point today is different. In that paper's open-domain QA case study, going from 20 to 50 retrieved documents improved accuracy by only about 1.5% (GPT-3.5-Turbo) and about 1% (Claude 1.3), while making the input much longer. More context gave very little back.
Newer models haven't made this go away. Chroma's "Context Rot" report (July 2025) tested 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found performance consistently degrades as input length grows, even on simple tasks. On LongMemEval, models did significantly better on focused prompts (~300 tokens) than on the full ~113k-token history with the same answer inside. Google's own guide says that with multiple "needles" the model "does not perform with the same accuracy" as single-needle tests.
Ranking is a quality feature, not a compression trick.
3. Freshness
A giant cached prompt is a snapshot. When a policy changes at 3pm, your retrieval index can update that one document. A stuffed-context setup has to rebuild and re-cache the whole bundle, or quietly answer from yesterday's version. Retrieval lets you update one document at a time.
4. Permissions are per document, not per prompt
This is the argument I rarely see in "RAG is dead" threads. In an enterprise app, user A can read the HR policy and user B cannot. Customer X's tickets must never reach customer Y.
If you load "everything" into the context, the model can see everything, and a prompt saying "don't reveal restricted docs" is not access control (same lesson as my 2026-10-13 post on agent permissions). Filtering by ACL before documents enter the context is the access-control layer. Retrieval with a permission filter does exactly that. A bigger window just makes the blast radius bigger.
5. Citations need provenance
Users and auditors want "where did this answer come from?" When you retrieve, you know exactly which chunk IDs, document versions, and URLs went into the call, so citations can be checked. With a 1M-token blob, you're asking the model to tell you where it read something, and that is much harder to verify.
So what changes with long context?
Long context is genuinely useful. It just moves the design point:
- Retrieve bigger units. Pull whole sections or documents instead of tiny chunks, so the model gets surrounding context.
- Retrieve more, then rank. Anthropic found passing the top 20 chunks beat top 10 or top 5, and that contextual embeddings + contextual BM25 + reranking cut top-20 retrieval failures by 67%.
- Stuff only when the corpus is small, static, and shared. Under ~200k tokens, same for every user, rarely changing? Put it in context and cache it.
- Filter by permissions first, then rank. Never let the model be the access-control layer.
Practical takeaways
- Measure, don't assume. Run your eval set with top-k retrieval vs full-context stuffing and compare accuracy, cost per call, and p95 latency.
- Track cost per answered question, not just accuracy. A 1M-token prompt that's 2% better can still be the wrong trade-off.
- Keep retrieval as your source of truth for who can see what and which version is current.
- Log retrieved chunk IDs with every answer so citations can be audited.
Long context windows make RAG simpler and more forgiving. They don't make it obsolete. Retrieval decides what's relevant, what's current, and what this user is allowed to see, and none of that goes away with a bigger window.
What's your setup: stuffing everything into a long context, classic top-k RAG, or a hybrid?
References
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (arXiv 2307.03172): https://arxiv.org/abs/2307.03172
- Chroma, "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (July 2025): https://research.trychroma.com/context-rot
- Google, Gemini API docs, "Long context": https://ai.google.dev/gemini-api/docs/long-context
- Anthropic, "Introducing Contextual Retrieval": https://www.anthropic.com/news/contextual-retrieval
Top comments (0)