The support bot
had a simple pipeline.
Search the documents.
Take the top twenty passages.
Put them all in the prompt,
then the question at the end.
The answer to the question
was in passage eleven.
The bot said
it could not find that information.
Nothing was missing.
It was all there,
inside the window,
well under the limit.
The model just did not use it.
There is a well known paper on this,
from 2023,
called Lost in the Middle.
The researchers hid the one useful passage
among many useless ones
and moved it around.
Accuracy was best
when the passage sat at the start
or at the end.
Put it in the middle
and accuracy fell,
sometimes below what the model managed
with no documents at all.
The curve looked like a U.
Newer models
are better at this.
None of them are perfect,
and none of them tell you
when they skipped something.
So a large context window
is a promise about what fits,
not about what gets read
with equal care.
The fix is mostly
about sending less.
Twenty passages
because twenty fit
is not a reason.
Rank them properly.
Cut the ones below a score
you have actually tested.
Five good passages
beat twenty with the right one
buried in the middle.
Put the strongest one first.
Keep the question
close to the end,
and say it again there
if the prompt is long.
When the order matters,
do not leave it
to whatever order the search returned.
Then measure it.
Take one question
with a known answer.
Run it with the right passage
at the start,
in the middle,
at the end.
If the answers change,
you have learned
something your benchmark never told you.
A long context
is a long corridor.
The model checks the door it came in
and the door it is leaving by.
Do not hide
the thing that matters
halfway down.
– Serguey Asael Shinder
Top comments (0)