The problem
You build a RAG pipeline. You test it with 3 retrieved documents and the correct answer is right there in document 2 - the model nails it every time. Feeling good, you bump top_k from 3 to 15 "to be safe," figuring more context can only help.
Accuracy drops. Not a crash, not a timeout, no error in your logs - the model just starts confidently giving wrong answers, or missing facts that are sitting in plain text inside the prompt you sent it. You re-read the context window by hand and the answer is right there. The model saw it. It just didn't use it.
If you've hit this, you didn't build a bad retriever. You ran into a well-documented, model-agnostic failure mode: LLMs don't recall information uniformly across a context window. They recall the beginning and the end. The middle is where information goes to die.
Why it happens
This isn't a rumor - it's been measured directly. Liu et al.'s "Lost in the Middle" study (2023) ran controlled retrieval tests across context lengths and found a consistent U-shaped accuracy curve: performance is highest when the relevant fact sits at the very start or very end of the context, and drops - sometimes below random-guessing-adjacent territory - when it's buried in the middle. This held across multiple model families, not one vendor's quirk.
The mechanism is architectural, not a bug you can patch. Self-attention doesn't distribute recall evenly across token positions. Models get heavy exposure to short-range dependencies during training (the next word usually depends on nearby words), and comparatively little training signal that rewards precise retrieval from the geometric middle of a long, unstructured span. Positional encodings compound this - most schemes bias attention toward strong local recency and, separately, toward the sequence start (a natural anchor point), leaving the middle structurally under-attended.
Here's the part that actually bites in production: the advertised context window and the effectively usable context window are different numbers, and vendors report the former. A model with a "1M token context" doesn't mean it retrieves reliably across 1M tokens - it means it doesn't error out before 1M tokens. Those are not the same claim, and the gap between them is exactly where "it worked in the demo, it's flaky in prod" bugs come from.
And it's silent by design. There's no exception to catch. Your retriever did its job, your prompt assembly did its job, the tokens are unambiguously present in the context - the failure is purely in what the model chooses to attend to, which is invisible from the outside unless you're specifically testing for it.
What to do about it
Stop treating top_k as a safety dial. More retrieved chunks doesn't monotonically improve recall - past a point it actively pushes your one correct chunk further into the dead zone. Tune top_k down, not up, and measure.
Reorder, don't just retrieve. After ranking, place your highest-confidence chunks at the start and end of the context, not in ranked order top-to-bottom. A chunk ranked #1 by your retriever but positioned dead-center in the prompt will underperform a chunk ranked #4 placed at the edges.
Add a reranker stage. A cheap cross-encoder rerank pass after initial retrieval, feeding only the top few into the prompt in a position-aware order, consistently outperforms "retrieve broadly, stuff it all in, let the model sort it out."
Build your own needle-in-a-haystack eval. Don't trust a vendor's long-context benchmark for your workload. Position sensitivity varies by model, by model version (it can regress on a silent upgrade), and by the structure of your specific documents. Take a real query from your logs, plant the known-correct fact at position 10%, 50%, and 90% of your typical context length, and measure accuracy at each. Rerun it every time you swap models.
Consider narrowing before you widen. A multi-step retrieval loop that progressively filters down to a small, high-confidence context often beats a single giant context stuffed with "everything that might be relevant." Fewer, better-placed tokens beat more tokens almost every time.
Key takeaways
- Context window size and effective retrieval range are different numbers - don't conflate them.
- The "Lost in the Middle" effect is architectural and measured across model families, not a one-off bug.
- The failure mode is silent: no error, just quietly wrong answers on facts that are technically present.
- Fixes: shrink
top_k, rerank and position-aware reorder, build your own positional eval, and prefer narrowing retrieval over widening it.
Top comments (0)