A RAG system retrieves the passage containing the answer.
The passage reaches the model.
The answer is still wrong.
That failure interests me because retrieval quality alone cannot explain it. We also need to investigate how the model uses the evidence it receives.
For this challenge, I’m exploring a research question:
If the evidence stays the same, can changing its position in the context change the answer?
The research behind the challenge
The paper Lost in the Middle: How Language Models Use Long Contexts investigates how models use information placed at different positions in their input.
Across its experiments, performance was often strongest when relevant information appeared near the beginning or end. Performance could drop when the information appeared in the middle of a long context.
The paper studies multi-document question answering and key-value retrieval.
That finding raises a practical concern for RAG: fitting evidence inside a context window does not guarantee that a model will use it reliably.
It also gives me a hypothesis to test—not a result I should assume holds for every current model.
Measure how sensitive answer quality is to the position of supporting evidence.
I would start with questions that each have one clearly identified supporting passage. This makes it easier to isolate position before introducing questions that require multiple pieces of evidence.
For each question, I would construct three versions:
- Supporting passage at the beginning.
- Supporting passage in the middle.
- Supporting passage at the end.
The question, passage text, distractor passages, and instructions would remain unchanged. Only passage order would change.
For example:
Question: “How long does a customer have to request a refund?”
Supporting evidence: “Refund requests must be submitted within 30 days of purchase.”
Can the model consistently answer “30 days” and cite the correct passage across all three positions?
Making the comparison meaningful
One example would not tell me much.
I would use a labelled question set, repeat the comparison with several shuffled distractor orders, and test different context lengths.
Within each comparison, I would keep the model version, generation settings, and output budget fixed. Passage IDs would stay attached to their passages when reordered.
I would also check that no input is truncated. Otherwise, I might accidentally measure missing evidence rather than evidence position.
A supporting-passage-only baseline would help identify questions the model struggles with even without distractors.
What I would measure
Answer accuracy: Does the response match the labelled answer?
Citation correctness: Does the cited passage actually support the answer? Returning an existing passage ID is not enough.
Position sensitivity: How much does accuracy change between beginning, middle, and end placement?
I would report results separately by model and context length, including uncertainty around the estimates.
What would make the findings useful?
If middle placement performs worse, the next experiment could compare context-ordering strategies under the same token budget.
If no clear difference appears, that is useful too. It would suggest the paper’s pattern did not clearly reproduce under my tested conditions.
Either outcome needs a bounded conclusion: these models, these questions, and these context lengths.
My takeaway
A successful retrieval step is only part of a successful RAG answer.
The next question is whether the model can reliably use the evidence we give it—and whether our context construction makes that easier or harder.
This post describes a research-backed experiment plan. I have not measured the results yet.
Research reference
Liu et al. — Lost in the Middle: How Language Models Use Long Contexts
https://arxiv.org/abs/2307.03172
Have you tested whether passage order changes the answers in your RAG application?
Top comments (0)