We stopped measuring retrieval quality by how many tokens we could fit into the prompt.
When long-context models became available, many of us made the same assumption.
If an LLM can read 128K tokens, retrieval suddenly feels less important. Why spend time carefully selecting documents if the model can simply read everything?
It sounds reasonable.
In practice, it wasn't.
More context, worse answers
Imagine asking your internal assistant:
Why did we abandon microservices?
Retrieval returns thirty documents.
- an architecture decision record
- a few Jira tickets
- Slack discussions
- meeting notes
- a glossary page
Everything is related.
Almost nothing answers the question.
The actual decision lives in a single ADR written months earlier. It explains the trade-offs: team size, latency, deployment complexity, operational cost.
But that document isn't especially similar to the query. It doesn't repeat the same vocabulary. It doesn't even mention "microservices" very often.
So it gets buried.
The model now receives thirty relevant documents and does what language models are very good at: it produces a coherent explanation.
The problem is that coherence is not the same thing as faithfulness.
Instead of recovering the original decision, it often synthesizes one from recurring themes across the retrieved documents.
The answer sounds plausible.
It just isn't the answer that was originally made.
Bigger windows don't fix retrieval
Research has already shown that models struggle with information buried inside very long contexts. The Lost in the Middle paper is probably the best-known example.
Our experience suggested something slightly different.
Sometimes the answer isn't lost because the context is long.
It's lost because the retrieval stage couldn't distinguish the document that contains the decision from documents that merely discuss the same topic.
Adding more context doesn't necessarily solve that problem.
Sometimes it simply gives the model more material to average together.
We were optimizing the wrong thing
For a while we treated retrieval as a packing exercise.
How many useful chunks can we fit into the prompt?
Over time the question changed.
Why is this document here?
Should it be here at all?
Does it explain the decision, or does it merely mention the same technology?
Those questions turned out to matter much more than the size of the context window.
Retrieval is a selection problem
The biggest shift wasn't moving from 8K to 128K tokens.
It was realizing that retrieval isn't about fitting more information into a prompt.
It's about selecting the few pieces of information that actually explain the answer.
Large context windows are incredibly useful.
They just don't compensate for weak retrieval.
If anything, they make weak retrieval look convincing.
Next time I'll look at another assumption I no longer believe: that documents should be treated as bags of chunks.
Top comments (42)
ran into this with document retrieval agents - more context made outputs worse. the model averaged across noise instead of locking on signal. tighter retrieval with smaller windows fixed precision. bigger context just diluted the signal-to-noise ratio
Exactly — that's the core insight. More context doesn't fix bad retrieval; it just distributes the error more smoothly. Tight retrieval with a small window forces the system to earn every token it sends to the model.
the "earn every token" framing is one I want to steal for our retrieval specs. it forces the design question away from "how much context is enough" and toward "what is each chunk actually doing for the model" — which catches noise accumulation before it ships
That microservices ADR example nails the trap perfectly: the document that holds the real decision is often the one that mentions it the least, because a good decision doc argues the trade-offs instead of repeating the keyword. Pure vector similarity is almost designed to miss that. What I keep coming back to is your "coherence is not faithfulness" line, since a model handed thirty near-misses will always produce something that reads right. Did reranking move the needle here, or did the fix end up being more about what you let into the candidate set in the first place? I'd bet this is more a recall problem at the retrieval stage than a reordering problem for what you already pulled.
That's a really interesting way to frame it.
I think your ADR example captures an important failure mode: the document containing the actual decision isn't necessarily the one with the strongest lexical or semantic overlap with the query. It may spend most of its space discussing trade-offs instead.
As for your question, I can't confidently separate the effects of candidate generation and reranking because I haven't run controlled experiments comparing them in isolation.
My observation was simply that improving reranking alone didn't fully address the issue if the right document never made it into the candidate set to begin with. That makes me think candidate selection is at least part of the problem.
Whether it's primarily a recall problem at retrieval or a reranking problem is something I'd want to test rather than claim. My suspicion is that both stages matter, but missing the relevant document early in the pipeline is much harder to recover from later.
thanks gfor sharing!
The counterintuitive part is that recall at retrieval and precision at generation pull in opposite directions, so padding the window past what the question needs just hands the model more distractors to average over. Dumping 30 loosely related docs into 128K tends to lose to a tight top-k with a rerank for exactly that reason. When you cut back, did you find the sweet spot by tuning k directly, or by watching how answer quality moved as you added each chunk?
Thanks, that's a good question.
We didn't arrive at a fixed value of k through a parameter sweep.
Instead, we mainly watched how answer quality changed as we varied the amount of retrieved context. My observation was that adding the first few relevant chunks often helped, but beyond a certain point additional context sometimes made answers less focused rather than better.
I haven't run a systematic study that identifies an optimal k, so I wouldn't claim there's a universal sweet spot.
One direction we're exploring is making retrieval depth depend on the query instead of treating k as a fixed parameter. The idea is that highly specific questions may only need a few precise passages, while broader questions may benefit from more context. At this stage, I'd describe that as an architectural direction rather than a validated result.
This matches what I keep seeing: more context is not the same as better retrieval. A larger window can hide retrieval mistakes because the answer still sounds informed. The useful measurement is whether the right evidence was selected and used, not whether enough text was available somewhere in the prompt.
I think that's exactly the shift in perspective I was trying to argue for.
A larger context window can make a system look more capable because the answers remain fluent, even when the retrieval step wasn't particularly good.
That's why I'm becoming more interested in evaluating retrieval directly, rather than treating answer quality as the only signal. If the right evidence never made it into the retrieved set, that's a very different problem from the model failing to use evidence that was already available.
Those two failure modes need different fixes, but they're easy to confuse if we only look at the final answer.
So I agree with your framing: before asking how much context the model can consume, it's worth asking whether we selected the right evidence in the first place.
Exactly. Retrieval quality deserves its own inspection layer. If the evidence set is wrong, answer grading can only tell you that something felt off, not whether the fix belongs in chunking, ranking, query rewriting, freshness, or the final synthesis step.
I like the way you framed it.
Once retrieval has its own inspection layer, a "bad answer" stops being a single failure category. It becomes a starting point for asking why it happened.
Was the evidence never retrieved? Was it retrieved but ranked too low? Was the query interpreted poorly? Or was the evidence there and the model simply didn't use it?
Those are very different failure modes, and they call for different fixes. If we only evaluate the final answer, they all collapse into "the model got it wrong," which isn't very helpful for improving the system.
That's exactly why I'm becoming more interested in retrieval as something to inspect in its own right, rather than treating it as a black box before generation.
That is the useful split. Once retrieval gets its own inspection layer, a bad answer is no longer one vague failure. You can ask whether the source was missing, retrieved poorly, ranked wrong, or interpreted badly.
Yes. Once those failure modes split apart, the fix can be much smaller. Bad retrieval, bad ranking, bad query rewriting, and bad answer grounding need different interventions. A single final-answer score compresses all of that into one vague red light.
The line I would underline is coherence is not the same as faithfulness, and the corollary that makes it dangerous: a bigger window does not just amplify weak retrieval, it deletes the symptom. An incoherent answer used to be the tell that retrieval whiffed. Now the model can average thirty topical neighbors into a fluent decision that was never made, and the miss is invisible at answer time, because the output looks identical whether the ADR was in the context or not.
Which is why retrieval precision is necessary but not sufficient on its own. The missing primitive is a sufficiency check separate from ranking: does the retrieved set actually contain a document that answers this query, or only documents about the topic. Reranking sorts what you pulled, it cannot tell you the decision-bearing doc is absent. When it is absent, the honest move is to abstain and say no decision record was retrieved, not to synthesize one from the noise. Coherent fiction is what you get when the system is never allowed to say it does not know.
I think you've articulated something I only hinted at in the article.
"A bigger window doesn't just amplify weak retrieval, it can hide the retrieval failure altogether."
That's a much sharper way to describe the problem.
I also agree with your distinction between ranking and sufficiency. Reranking can only choose among the candidates that were retrieved. It can't tell you whether the document that actually answers the question is missing entirely.
That distinction has influenced the way I'm thinking about retrieval. Instead of asking only "which document is most relevant?", I'm increasingly interested in asking "do I have enough evidence to answer this question at all?"
That's one of the motivations behind the L2 Essence layer. Rather than treating retrieval as a search for similar text, I'm exploring whether it can also represent what kinds of claims or decisions a document actually contains. If that representation suggests the evidence is incomplete, I'd rather abstain than let the model interpolate a plausible answer.
I haven't proven that this is the right architecture, but I do think the ability to detect missing evidence is a different problem from ranking retrieved evidence. Your comment captures that distinction really well.
This is the right shift, and I think the piece that turns L2 Essence into an actual decision procedure is typing the query the same way you type the document.
Representing what claims a doc contains gives you one half. Sufficiency is a relation, not a property, so you also need what claim the question requires: is it asking for a decision record, a definition, a number, a causal link. Then the check becomes concrete instead of a vibe. The query needs a claim of type decision-record, the retrieved set has docs about that topic but none carrying a decision-type claim, so abstain. Same topic, wrong claim type is exactly the case a similarity score cannot see and a reranker cannot rescue.
The part I would guard hardest: the essence layer has to distinguish asserts X from mentions X, or it just moves the coherent-fiction problem down one level. An LLM-derived representation will happily tag a doc as containing a decision when it only discusses one, and now your sufficiency check is confidently wrong instead of the generator.
The good news is this is one of the few RAG properties you can measure offline. Build a set where you deliberately withhold the decision-bearing doc and check the system abstains instead of averaging the neighbors. Coherence hides the miss at answer time, so a held-out sufficiency set is the only place you catch it before a user does.
great post. this reminds me of something anthropic published.
they call it context rot: as context windows get bigger, the model's ability to actually retain and recall information gets worse, not better.
more tokens ≠ better reasoning.
the point is that context is a finite resource that needs to be curated, not just stuffed.
link here if curious: anthropic.com/engineering/effectiv...
i'd also add: garbage in, garbage out.
the quality of what goes into the context window is the first-order problem. retrieval precision, document structure at ingestion, synthetic questions... all of these are really just about raising signal quality before the model ever sees anything. a 128k window full of loosely related chunks is just a very expensive way to confuse the model
Thanks for sharing that—I hadn't connected it to Anthropic's "context rot" terminology, but it describes a very similar failure mode.
The part that resonates most with me is exactly what you said: context is a finite resource that needs to be curated, not just expanded.
I also like your "garbage in, garbage out" framing. The more I worked on retrieval, the more it felt like most of the problem lives upstream of generation. Retrieval quality, document structure, and things like synthetic questions all have the same goal: increase the signal before the model starts reasoning over the context.
I don't think of those as independent optimizations so much as different ways of improving the quality of the evidence the model receives.
And I agree with your last point. A very large context window full of loosely related information doesn't necessarily make the model more reliable. In fact, it can make retrieval failures harder to notice because the answer remains fluent even when the supporting evidence is weak or incomplete.
The Anthropic article and the discussion here seem to be pointing at the same broader idea from different directions: bigger context windows don't remove the need for careful retrieval—they arguably make good retrieval even more important.
One thing worth trying: at ingestion, prompt an LLM for 3–5 questions each chunk answers, and then instead of indexing the document text itself, index those questions instead.
So for your ADR, you'd generate something like "why did we move away from microservices" as a synthetic question, even though the ADR itself never uses that phrase. Now the user's query matches the question, not the raw text, so the vocabulary mismatch stops mattering as much.
This isn't a new idea, by the way. It's basically what 'doc2query' does, just usually described for BM25 search rather than embeddings. Worth filtering out weak/hallucinated questions before you index them though, or you're back to noise, just a different kind.
Wouldn't replace your hybrid setup, just sits next to it. Curious if you've tried anything like this versus just tuning the reranker.
That's a good suggestion, and the doc2query comparison is fair.
We've experimented with something similar, although we don't replace the document representation with synthetic questions. Instead, we treat them as an additional view of the same document.
At the L2 stage, we generate both an essence (a compact representation of what the document is about) and a small set of synthetic questions describing what the document can answer. Both are embedded and indexed alongside the original document representation.
That gives us multiple retrieval paths. A query may match the document through its essence, through one of the synthetic questions, or through the original content, and we merge those candidates before reranking.
I see synthetic questions as addressing a different failure mode than reranking. If the user's wording differs significantly from the document's wording, synthetic questions can help the document enter the candidate set. Once it's there, reranking has a much better chance of selecting it.
One challenge is making sure the generated questions stay faithful to the source document. As you pointed out, weak or overly broad questions can introduce a different kind of retrieval noise, so we treat them as derived metadata rather than authoritative content.
We also don't apply this path uniformly. Our query analyzer helps decide which retrieval strategies are likely to be useful for a given query, so synthetic questions become one signal among several rather than the default retrieval mechanism.
I haven't isolated the contribution of synthetic questions versus reranking in controlled benchmarks, so I wouldn't claim one is more important than the other. My current view is that they solve different problems: synthetic questions expand the retrieval surface, while reranking improves selection within the candidate set.
My AI isn’t just for generating code. It analyzes existing code, finds issues, explains the root cause, and helps fix them across 16 programming languages.
The trap here is that cosine rewards documents that talk about the topic, but a real decision doc is written in the language of trade-offs, not the keyword itself, so the ADR loses to ten Jira tickets repeating "microservices" even though none of them made the call. Did you fix this at retrieval (reranking, doc-type weighting) or upstream by tagging decisions as their own type at ingestion? Reranking still scores on that same buried signal, so my bet is the upstream move does more work.
That's an interesting hypothesis.
I agree that enriching documents with structural information during ingestion could improve candidate generation. If the retrieval stage never sees the right ADR, no amount of reranking can recover it later.
One architecture we're exploring is to capture document structure—such as distinguishing decisions from background or discussion—before retrieval. The idea is that this could help the candidate set contain the documents that actually answer the question.
Reranking would then solve a different problem: choosing the most relevant document from that candidate set.
I haven't measured how much each stage contributes independently, so I can't say whether ingestion or reranking matters more. My intuition is that they're complementary rather than competing approaches, but that's still something that needs to be validated.
Complementary is right, but "which matters more" has an answer once you fix the axis: reranking can only reorder what retrieval already surfaced, so its ceiling is capped by candidate-set recall. If the ADR isn't in the top-k candidates, no reranker recovers it. That also makes the two measurable separately without much work: hold retrieval flat and track recall@k of decision docs specifically (not aggregate recall) before and after you type them at ingestion. If typing lifts ADR recall@50, that's pure upstream contribution, and whatever reranking adds sits on top of that number. My bet is you'll see ingestion move recall and reranking move precision, which is exactly why they don't compete: they fix different failures.
The selection-over-packing reframing matches what I hit in production, and the fix that moved the needle for us was all at the retrieval stage. Dense-only retrieval is what buries the ADR, so we ran lexical (BM25) and dense in parallel and merged the results, which recovered documents the embedding ranked low. A cross-encoder rerank over a wider candidate set then pulled the low-similarity-but-correct doc back up before it ever reached the prompt. The other lever was query fan-out: expanding the short question into a few reformulations and unioning the hits, since one embedding of a terse query rarely lands near a long trade-off document. None of this needs a bigger window, and as you say, a bigger window mostly hides the miss. I wrote up the hybrid plus parallel fan-out approach here: renezander.com/blog/agentic-knowle...
Thanks for sharing this. I always find production experience more interesting than synthetic benchmarks.
Your point about hybrid retrieval resonates with what I've been observing. Dense retrieval and lexical retrieval tend to fail in different ways, so combining them gives the reranker a better candidate set to work with.
I also like your distinction between improving retrieval and increasing context size. Those are often treated as interchangeable, but I don't think they are. If the right document never enters the candidate set, giving the model a larger context window doesn't solve the underlying problem.
The query fan-out idea is interesting as well. I've been exploring a related direction from the document side by generating additional representations of a document, including synthetic questions, to reduce vocabulary mismatch. They're different approaches, but they're trying to address the same failure mode.
One thing I haven't measured in isolation is how much each technique contributes on its own. So I wouldn't claim that hybrid retrieval, reranking, or query expansion is the key improvement. My impression is that they complement each other by addressing different stages of the retrieval pipeline.
I'll take a look at your write-up—thanks for sharing it.
The retrieval grading point really resonated. We've seen a similar pattern at IT Path Solutions most production RAG issues weren't coming from embeddings or prompts, but from missing feedback loops around retrieval quality and confidence scoring. Once the system starts deciding when not to answer, user trust improves much faster than trying to squeeze a few more points out of semantic search. Nice breakdown of the architecture.
Thank you — I think that's a really important point.
One thing I've found is that retrieval quality isn't just something to optimize internally; it's also something the system should expose when appropriate. If the evidence is weak or incomplete, I'd rather make that visible than present a confident answer that overstates what the retrieved context actually supports.
In the architecture I'm exploring, confidence isn't based on a single similarity score. It combines several signals from retrieval and the retrieved evidence before deciding how much confidence to place in an answer. The exact weighting is still evolving, but the goal is always the same: distinguish "I found strong supporting evidence" from "I found something related, but it may not fully answer the question."
That also changes how I think about "I don't know." Rather than treating it as a failure, I'd rather explain why confidence is limited—for example, if the retrieved evidence is incomplete or internally inconsistent.
I also agree with your observation that many production issues aren't caused by embeddings or prompts in isolation. In my experience, they're often failures of observability: the retrieval step quietly misses the right evidence, while the generated answer still sounds convincing.
That's one reason I've been putting more emphasis on making retrieval observable—through provenance, confidence signals, and explicit uncertainty—than on chasing incremental improvements in any single retrieval metric.