We stopped measuring retrieval quality by how many tokens we could fit into the prompt.
When long-context models became available, many of us made...
For further actions, you may consider blocking this person and/or reporting abuse
ran into this with document retrieval agents - more context made outputs worse. the model averaged across noise instead of locking on signal. tighter retrieval with smaller windows fixed precision. bigger context just diluted the signal-to-noise ratio
Exactly — that's the core insight. More context doesn't fix bad retrieval; it just distributes the error more smoothly. Tight retrieval with a small window forces the system to earn every token it sends to the model.
the "earn every token" framing is one I want to steal for our retrieval specs. it forces the design question away from "how much context is enough" and toward "what is each chunk actually doing for the model" — which catches noise accumulation before it ships
That microservices ADR example nails the trap perfectly: the document that holds the real decision is often the one that mentions it the least, because a good decision doc argues the trade-offs instead of repeating the keyword. Pure vector similarity is almost designed to miss that. What I keep coming back to is your "coherence is not faithfulness" line, since a model handed thirty near-misses will always produce something that reads right. Did reranking move the needle here, or did the fix end up being more about what you let into the candidate set in the first place? I'd bet this is more a recall problem at the retrieval stage than a reordering problem for what you already pulled.
That's a really interesting way to frame it.
I think your ADR example captures an important failure mode: the document containing the actual decision isn't necessarily the one with the strongest lexical or semantic overlap with the query. It may spend most of its space discussing trade-offs instead.
As for your question, I can't confidently separate the effects of candidate generation and reranking because I haven't run controlled experiments comparing them in isolation.
My observation was simply that improving reranking alone didn't fully address the issue if the right document never made it into the candidate set to begin with. That makes me think candidate selection is at least part of the problem.
Whether it's primarily a recall problem at retrieval or a reranking problem is something I'd want to test rather than claim. My suspicion is that both stages matter, but missing the relevant document early in the pipeline is much harder to recover from later.
thanks gfor sharing!
The counterintuitive part is that recall at retrieval and precision at generation pull in opposite directions, so padding the window past what the question needs just hands the model more distractors to average over. Dumping 30 loosely related docs into 128K tends to lose to a tight top-k with a rerank for exactly that reason. When you cut back, did you find the sweet spot by tuning k directly, or by watching how answer quality moved as you added each chunk?
Thanks, that's a good question.
We didn't arrive at a fixed value of k through a parameter sweep.
Instead, we mainly watched how answer quality changed as we varied the amount of retrieved context. My observation was that adding the first few relevant chunks often helped, but beyond a certain point additional context sometimes made answers less focused rather than better.
I haven't run a systematic study that identifies an optimal k, so I wouldn't claim there's a universal sweet spot.
One direction we're exploring is making retrieval depth depend on the query instead of treating k as a fixed parameter. The idea is that highly specific questions may only need a few precise passages, while broader questions may benefit from more context. At this stage, I'd describe that as an architectural direction rather than a validated result.
This matches what I keep seeing: more context is not the same as better retrieval. A larger window can hide retrieval mistakes because the answer still sounds informed. The useful measurement is whether the right evidence was selected and used, not whether enough text was available somewhere in the prompt.
I think that's exactly the shift in perspective I was trying to argue for.
A larger context window can make a system look more capable because the answers remain fluent, even when the retrieval step wasn't particularly good.
That's why I'm becoming more interested in evaluating retrieval directly, rather than treating answer quality as the only signal. If the right evidence never made it into the retrieved set, that's a very different problem from the model failing to use evidence that was already available.
Those two failure modes need different fixes, but they're easy to confuse if we only look at the final answer.
So I agree with your framing: before asking how much context the model can consume, it's worth asking whether we selected the right evidence in the first place.
Exactly. Retrieval quality deserves its own inspection layer. If the evidence set is wrong, answer grading can only tell you that something felt off, not whether the fix belongs in chunking, ranking, query rewriting, freshness, or the final synthesis step.
I like the way you framed it.
Once retrieval has its own inspection layer, a "bad answer" stops being a single failure category. It becomes a starting point for asking why it happened.
Was the evidence never retrieved? Was it retrieved but ranked too low? Was the query interpreted poorly? Or was the evidence there and the model simply didn't use it?
Those are very different failure modes, and they call for different fixes. If we only evaluate the final answer, they all collapse into "the model got it wrong," which isn't very helpful for improving the system.
That's exactly why I'm becoming more interested in retrieval as something to inspect in its own right, rather than treating it as a black box before generation.
That is the useful split. Once retrieval gets its own inspection layer, a bad answer is no longer one vague failure. You can ask whether the source was missing, retrieved poorly, ranked wrong, or interpreted badly.
Yes. Once those failure modes split apart, the fix can be much smaller. Bad retrieval, bad ranking, bad query rewriting, and bad answer grounding need different interventions. A single final-answer score compresses all of that into one vague red light.
The line I would underline is coherence is not the same as faithfulness, and the corollary that makes it dangerous: a bigger window does not just amplify weak retrieval, it deletes the symptom. An incoherent answer used to be the tell that retrieval whiffed. Now the model can average thirty topical neighbors into a fluent decision that was never made, and the miss is invisible at answer time, because the output looks identical whether the ADR was in the context or not.
Which is why retrieval precision is necessary but not sufficient on its own. The missing primitive is a sufficiency check separate from ranking: does the retrieved set actually contain a document that answers this query, or only documents about the topic. Reranking sorts what you pulled, it cannot tell you the decision-bearing doc is absent. When it is absent, the honest move is to abstain and say no decision record was retrieved, not to synthesize one from the noise. Coherent fiction is what you get when the system is never allowed to say it does not know.
I think you've articulated something I only hinted at in the article.
"A bigger window doesn't just amplify weak retrieval, it can hide the retrieval failure altogether."
That's a much sharper way to describe the problem.
I also agree with your distinction between ranking and sufficiency. Reranking can only choose among the candidates that were retrieved. It can't tell you whether the document that actually answers the question is missing entirely.
That distinction has influenced the way I'm thinking about retrieval. Instead of asking only "which document is most relevant?", I'm increasingly interested in asking "do I have enough evidence to answer this question at all?"
That's one of the motivations behind the L2 Essence layer. Rather than treating retrieval as a search for similar text, I'm exploring whether it can also represent what kinds of claims or decisions a document actually contains. If that representation suggests the evidence is incomplete, I'd rather abstain than let the model interpolate a plausible answer.
I haven't proven that this is the right architecture, but I do think the ability to detect missing evidence is a different problem from ranking retrieved evidence. Your comment captures that distinction really well.
This is the right shift, and I think the piece that turns L2 Essence into an actual decision procedure is typing the query the same way you type the document.
Representing what claims a doc contains gives you one half. Sufficiency is a relation, not a property, so you also need what claim the question requires: is it asking for a decision record, a definition, a number, a causal link. Then the check becomes concrete instead of a vibe. The query needs a claim of type decision-record, the retrieved set has docs about that topic but none carrying a decision-type claim, so abstain. Same topic, wrong claim type is exactly the case a similarity score cannot see and a reranker cannot rescue.
The part I would guard hardest: the essence layer has to distinguish asserts X from mentions X, or it just moves the coherent-fiction problem down one level. An LLM-derived representation will happily tag a doc as containing a decision when it only discusses one, and now your sufficiency check is confidently wrong instead of the generator.
The good news is this is one of the few RAG properties you can measure offline. Build a set where you deliberately withhold the decision-bearing doc and check the system abstains instead of averaging the neighbors. Coherence hides the miss at answer time, so a held-out sufficiency set is the only place you catch it before a user does.
great post. this reminds me of something anthropic published.
they call it context rot: as context windows get bigger, the model's ability to actually retain and recall information gets worse, not better.
more tokens ≠ better reasoning.
the point is that context is a finite resource that needs to be curated, not just stuffed.
link here if curious: anthropic.com/engineering/effectiv...
i'd also add: garbage in, garbage out.
the quality of what goes into the context window is the first-order problem. retrieval precision, document structure at ingestion, synthetic questions... all of these are really just about raising signal quality before the model ever sees anything. a 128k window full of loosely related chunks is just a very expensive way to confuse the model
Thanks for sharing that—I hadn't connected it to Anthropic's "context rot" terminology, but it describes a very similar failure mode.
The part that resonates most with me is exactly what you said: context is a finite resource that needs to be curated, not just expanded.
I also like your "garbage in, garbage out" framing. The more I worked on retrieval, the more it felt like most of the problem lives upstream of generation. Retrieval quality, document structure, and things like synthetic questions all have the same goal: increase the signal before the model starts reasoning over the context.
I don't think of those as independent optimizations so much as different ways of improving the quality of the evidence the model receives.
And I agree with your last point. A very large context window full of loosely related information doesn't necessarily make the model more reliable. In fact, it can make retrieval failures harder to notice because the answer remains fluent even when the supporting evidence is weak or incomplete.
The Anthropic article and the discussion here seem to be pointing at the same broader idea from different directions: bigger context windows don't remove the need for careful retrieval—they arguably make good retrieval even more important.
One thing worth trying: at ingestion, prompt an LLM for 3–5 questions each chunk answers, and then instead of indexing the document text itself, index those questions instead.
So for your ADR, you'd generate something like "why did we move away from microservices" as a synthetic question, even though the ADR itself never uses that phrase. Now the user's query matches the question, not the raw text, so the vocabulary mismatch stops mattering as much.
This isn't a new idea, by the way. It's basically what 'doc2query' does, just usually described for BM25 search rather than embeddings. Worth filtering out weak/hallucinated questions before you index them though, or you're back to noise, just a different kind.
Wouldn't replace your hybrid setup, just sits next to it. Curious if you've tried anything like this versus just tuning the reranker.
That's a good suggestion, and the doc2query comparison is fair.
We've experimented with something similar, although we don't replace the document representation with synthetic questions. Instead, we treat them as an additional view of the same document.
At the L2 stage, we generate both an essence (a compact representation of what the document is about) and a small set of synthetic questions describing what the document can answer. Both are embedded and indexed alongside the original document representation.
That gives us multiple retrieval paths. A query may match the document through its essence, through one of the synthetic questions, or through the original content, and we merge those candidates before reranking.
I see synthetic questions as addressing a different failure mode than reranking. If the user's wording differs significantly from the document's wording, synthetic questions can help the document enter the candidate set. Once it's there, reranking has a much better chance of selecting it.
One challenge is making sure the generated questions stay faithful to the source document. As you pointed out, weak or overly broad questions can introduce a different kind of retrieval noise, so we treat them as derived metadata rather than authoritative content.
We also don't apply this path uniformly. Our query analyzer helps decide which retrieval strategies are likely to be useful for a given query, so synthetic questions become one signal among several rather than the default retrieval mechanism.
I haven't isolated the contribution of synthetic questions versus reranking in controlled benchmarks, so I wouldn't claim one is more important than the other. My current view is that they solve different problems: synthetic questions expand the retrieval surface, while reranking improves selection within the candidate set.
My AI isn’t just for generating code. It analyzes existing code, finds issues, explains the root cause, and helps fix them across 16 programming languages.
The trap here is that cosine rewards documents that talk about the topic, but a real decision doc is written in the language of trade-offs, not the keyword itself, so the ADR loses to ten Jira tickets repeating "microservices" even though none of them made the call. Did you fix this at retrieval (reranking, doc-type weighting) or upstream by tagging decisions as their own type at ingestion? Reranking still scores on that same buried signal, so my bet is the upstream move does more work.
That's an interesting hypothesis.
I agree that enriching documents with structural information during ingestion could improve candidate generation. If the retrieval stage never sees the right ADR, no amount of reranking can recover it later.
One architecture we're exploring is to capture document structure—such as distinguishing decisions from background or discussion—before retrieval. The idea is that this could help the candidate set contain the documents that actually answer the question.
Reranking would then solve a different problem: choosing the most relevant document from that candidate set.
I haven't measured how much each stage contributes independently, so I can't say whether ingestion or reranking matters more. My intuition is that they're complementary rather than competing approaches, but that's still something that needs to be validated.
Complementary is right, but "which matters more" has an answer once you fix the axis: reranking can only reorder what retrieval already surfaced, so its ceiling is capped by candidate-set recall. If the ADR isn't in the top-k candidates, no reranker recovers it. That also makes the two measurable separately without much work: hold retrieval flat and track recall@k of decision docs specifically (not aggregate recall) before and after you type them at ingestion. If typing lifts ADR recall@50, that's pure upstream contribution, and whatever reranking adds sits on top of that number. My bet is you'll see ingestion move recall and reranking move precision, which is exactly why they don't compete: they fix different failures.
The selection-over-packing reframing matches what I hit in production, and the fix that moved the needle for us was all at the retrieval stage. Dense-only retrieval is what buries the ADR, so we ran lexical (BM25) and dense in parallel and merged the results, which recovered documents the embedding ranked low. A cross-encoder rerank over a wider candidate set then pulled the low-similarity-but-correct doc back up before it ever reached the prompt. The other lever was query fan-out: expanding the short question into a few reformulations and unioning the hits, since one embedding of a terse query rarely lands near a long trade-off document. None of this needs a bigger window, and as you say, a bigger window mostly hides the miss. I wrote up the hybrid plus parallel fan-out approach here: renezander.com/blog/agentic-knowle...
Thanks for sharing this. I always find production experience more interesting than synthetic benchmarks.
Your point about hybrid retrieval resonates with what I've been observing. Dense retrieval and lexical retrieval tend to fail in different ways, so combining them gives the reranker a better candidate set to work with.
I also like your distinction between improving retrieval and increasing context size. Those are often treated as interchangeable, but I don't think they are. If the right document never enters the candidate set, giving the model a larger context window doesn't solve the underlying problem.
The query fan-out idea is interesting as well. I've been exploring a related direction from the document side by generating additional representations of a document, including synthetic questions, to reduce vocabulary mismatch. They're different approaches, but they're trying to address the same failure mode.
One thing I haven't measured in isolation is how much each technique contributes on its own. So I wouldn't claim that hybrid retrieval, reranking, or query expansion is the key improvement. My impression is that they complement each other by addressing different stages of the retrieval pipeline.
I'll take a look at your write-up—thanks for sharing it.
The retrieval grading point really resonated. We've seen a similar pattern at IT Path Solutions most production RAG issues weren't coming from embeddings or prompts, but from missing feedback loops around retrieval quality and confidence scoring. Once the system starts deciding when not to answer, user trust improves much faster than trying to squeeze a few more points out of semantic search. Nice breakdown of the architecture.
Thank you — I think that's a really important point.
One thing I've found is that retrieval quality isn't just something to optimize internally; it's also something the system should expose when appropriate. If the evidence is weak or incomplete, I'd rather make that visible than present a confident answer that overstates what the retrieved context actually supports.
In the architecture I'm exploring, confidence isn't based on a single similarity score. It combines several signals from retrieval and the retrieved evidence before deciding how much confidence to place in an answer. The exact weighting is still evolving, but the goal is always the same: distinguish "I found strong supporting evidence" from "I found something related, but it may not fully answer the question."
That also changes how I think about "I don't know." Rather than treating it as a failure, I'd rather explain why confidence is limited—for example, if the retrieved evidence is incomplete or internally inconsistent.
I also agree with your observation that many production issues aren't caused by embeddings or prompts in isolation. In my experience, they're often failures of observability: the retrieval step quietly misses the right evidence, while the generated answer still sounds convincing.
That's one reason I've been putting more emphasis on making retrieval observable—through provenance, confidence signals, and explicit uncertainty—than on chasing incremental improvements in any single retrieval metric.
Bigger context often hides retrieval problems instead of fixing them. The question I like is: if the answer was wrong, can we tell whether retrieval missed the right source, ranked it too low, or included it but the model ignored it? Token volume alone will not answer that.
I think that's exactly the right debugging question.
When an answer is wrong, I'm much more interested in why it failed than in whether the model had enough tokens available.
The three failure modes you describe are also the ones I've been thinking about:
Those failures call for different fixes, so treating them as the same problem can be misleading.
One reason I've been exploring retrieval observability is to make those stages easier to inspect instead of treating retrieval as a black box.
And I agree with your broader point: simply increasing the context window doesn't tell you which failure occurred. In some cases, it can actually make the underlying retrieval issue harder to notice because the model still produces a coherent answer.
"coherence is not the same thing as faithfulness" — that line belongs on every RAG architecture diagram.
we hit this from the other direction: building content for AI search (AEO/GEO) rather than consuming it. same structural failure: a page that directly answers gets buried under pages that discuss the same topic in richer vocabulary. authoritative source loses to keyword density. retrieval failure from the authoring side.
the fix was the same framing shift you're describing: stop asking what's similar and start asking whether the document actually contains the answer. for content, that meant
llms.txtplus concise answer blocks over long prose. sounds like L2 Essence is attempting the same move from the retrieval side.is Essence doing claim type classification or just topicality? "asserts X" versus "mentions X" seems like the hard part.
I like the AEO/GEO analogy. It feels like the same problem viewed from opposite sides.
On the authoring side, you're trying to make the answer easy to retrieve. On the retrieval side, I'm trying to recognize that the answer is actually present rather than simply retrieving documents that discuss the same topic.
As for your question, L2 Essence is intended to capture more than topicality. The goal is to preserve the main claims or decisions a document contains, not just produce a shorter semantic representation.
I think the distinction you make between "asserts X" and "mentions X" is exactly the difficult part. A document can be highly relevant to a topic while never actually answering the question being asked.
I'm still exploring how far that distinction can be pushed reliably. If the L2 representation can preserve the document's claims well enough, it gives retrieval and reranking a much stronger signal than topical similarity alone. If it can't, then it risks becoming just another summary with the same limitations as the original retrieval pipeline.
So I'd say the ambition is closer to claim representation than simple topicality, but I wouldn't claim it's a solved problem.
Interesting take. I've noticed that beyond a certain window size, the marginal gains in context don't always translate to better retrieval accuracy—especially if the model isn't fine-tuned for dense information extraction. In our work with VoltageGPU, we had to balance memory constraints with the model's ability to retain semantic focus under longer prompts.
That's an interesting observation, and it matches what I've seen as well.
The point about semantic focus resonates with me. Beyond a certain amount of context, the question isn't just whether the model can fit more tokens—it's whether the additional information actually helps it identify the evidence that matters.
I also like your point about memory constraints. One thing I've been thinking about is that hard constraints sometimes lead to better retrieval architectures because they force you to be selective. If you can't rely on a huge context window, you have to improve candidate selection instead.
That's one of the motivations behind the layered retrieval approach I'm exploring. Rather than asking the model to extract the answer from a large mixed context, the goal is to narrow the evidence before generation begins.
I haven't isolated how much of this comes from retrieval versus the model's ability to reason over long prompts, but my intuition is that retrieval quality becomes even more important as context windows grow.
This matches what we kept running into building moteDB — a Rust embedded multimodal store for agents and robots. Once context got cheap we assumed retrieval was solved too, and the same thing happened: more tokens in the window just meant more near-misses competing for attention. The thirty-documents-that-all-feel-related problem doesn't go away with a bigger window; it gets quieter.
On edge and embodied devices the 'just read everything' option isn't even on the table — there isn't the RAM or the latency budget to stuff history into the prompt. What actually helped was pulling long-term state out of the context entirely and persisting it to a local embedded store, then retrieving only the slice the current step needs. Have you measured what happens to answer quality when the standing memory lives outside the window instead of inside it?
That's a really interesting observation, especially from the edge perspective where the context window is a hard constraint rather than an optimization target.
I haven't run a controlled experiment comparing "memory inside the window" versus "memory outside the window," so I can't claim quantitative improvements.
The direction I'm exploring is to keep most long-term knowledge outside the prompt and retrieve only the evidence needed for the current step. The motivation isn't just token efficiency—it's to reduce the amount of competing context the model has to reason over.
Your experience suggests the same architectural direction, just driven by hardware constraints instead of large-context models.
The "selection vs packing" reframe is the part I keep coming back to. We hit the exact same wall building memory for on-device agents, except the constraint was stricter: at the edge you often can't stuff 30 documents into the prompt even if you wanted to, so weak retrieval isn't hidden by a big window â it's exposed immediately.
What bit us specifically was your point about the decision doc getting buried because it isn't lexically similar to the query. The ADR-style "why we did X" record is semantically distant from the day-to-day chatter, so pure embedding similarity systematically under-ranks it. Our partial fix was to stop treating the decision as just another embedded chunk and instead store the reasoning trace (the rejected paths, the constraints, the actual trade-off) as a structured record with its own metadata, so it can be retrieved by type and recency rather than only by cosine distance.
Still unsolved on our side: how do you keep that selection honest as the corpus grows into the thousands? Do you maintain a separate index of "decision records" or rely on reranking?
Good question. I don't rely on reranking alone.
One thing I'm exploring is preserving document structure during ingestion rather than treating every document as an interchangeable chunk. Decision records carry different semantics from background discussions, so keeping that distinction helps retrieval before reranking even begins.
Reranking is still important, but I see it as solving a different problem. It can only reorder the candidate set it receives. If the retrieval stage never surfaces the relevant decision record, reranking can't recover it.
As the corpus grows, that's the part I'm paying the most attention to: making sure retrieval can identify the right class of evidence before asking the model to generate an answer.
This matches what I’ve seen too: long context changes the failure mode, it does not remove the retrieval problem.
The distinction between “related documents” and “decision-bearing documents” is especially important. A retrieval system can look good on semantic similarity while still burying the one ADR, ticket, or approval note that actually answers the question.
One practical pattern I like is evaluating retrieval with question types, not just chunks: policy lookup, historical decision, current metric, troubleshooting, customer-specific fact. Each type needs different evidence. If the eval only asks “did we retrieve something relevant?”, bigger context can hide weak retrieval for a surprisingly long time.
That's an interesting way to think about evaluation.
I especially like the distinction between retrieving something relevant and retrieving the kind of evidence the question actually requires.
A historical decision, a policy lookup, and a customer-specific fact may all require very different retrieval behavior, even if traditional relevance metrics treat them similarly.
One thing this discussion has made me think about is whether retrieval evaluation should include the expected evidence type, not just semantic relevance. A system can retrieve documents that are clearly related to the topic while still missing the document that actually answers the question.
And I agree with your broader point: larger context windows can make that failure harder to notice. If the evaluation only asks whether some relevant material was retrieved, the system may appear to perform well even when the decision-bearing or evidence-bearing document is missing.
Bigger context often hides retrieval problems instead of fixing them. If the system cannot select the right evidence in a small window, a larger window just gives the model more plausible distractions. I like measuring retrieval quality before celebrating token budget.