What happens when a RAG system retrieves the passage containing the answer, but that passage never reaches the language model?
Imagine retrieving 20 passages and passing only the first five into the prompt. The strongest evidence is sitting at position 12.
The retrieval stage found it. The context-selection step left it behind.
After exploring hybrid search, context position, and duplicate records in earlier posts, this feels like a useful next question:
Can reranking help us choose better evidence from the passages we already retrieved?
Where reranking fits
Hybrid search combines signals from keyword and semantic retrieval. That can help us find a useful set of candidate passages, but we still need to decide which ones deserve space in the final context.
A passage might mention all the right terms while missing the condition that actually matters to the question.
Reranking adds another relevance assessment between retrieval and context selection:
Question → Retrieve candidates → Rerank → Select context → Generate an answer
The first stage narrows the search. The reranker examines that shortlist more closely.
How a reranker evaluates a passage
A common approach is a cross-encoder, which processes the question and a candidate passage together to produce a relevance score.
This differs from a typical dense retriever, where the question and passages are encoded separately and compared using their vectors.
Nogueira and Cho’s Passage Re-ranking with BERT describes the joint approach: the model receives a query–passage pair, scores its relevance, and uses those scores to reorder the candidates.
The useful distinction is that the reranker can evaluate a passage against the wording and conditions of the specific question.
That still leaves room for mistakes. A relevance score is a model’s prediction, not a guarantee.
An example: the exception matters
Consider this question:
“Can I get a refund after 30 days if the product is defective?”
Suppose retrieval returns these passages:
| Passage | What it covers |
|---|---|
| A | The standard 30-day refund policy |
| B | Instructions for submitting a return |
| C | An exception allowing refunds for defective products after 30 days|
All three concern returns. Passage C directly addresses the exception in the question.
If C ranks below the context cutoff, the answer model might receive the general policy without the exception. It could then produce an incomplete answer despite useful evidence being present in the retrieved candidates.
A successful reranker would move C into the selected context.
This is an illustrative example, not an experimental result. It shows the kind of ranking error worth looking for.
What the research tells us
Three papers help frame the problem.
Passage Re-ranking with BERT — Nogueira and Cho
This work demonstrates a two-stage approach: retrieve candidates first, then score query–passage pairs with BERT.
The practical lesson I take from it is that candidate retrieval and final evidence ranking can be separate decisions.
BEIR — Thakur and colleagues
BEIR evaluated 10 retrieval systems across 18 datasets spanning different tasks and domains. Among the evaluated approaches, reranking and late-interaction methods achieved the best zero-shot performance on average, with higher computational costs.
That is a benchmark finding, not a promise that adding a reranker will improve every application. The methods, documents, and questions in our own pipeline still need evaluation.
Is ChatGPT Good at Search? — Sun and colleagues
This paper investigates generative language models as reranking agents. The authors found that appropriately instructed LLMs could produce competitive relevance rankings on the benchmarks they evaluated.
It offers another approach to compare with cross-encoders. However, strong ranking results alone do not establish that a RAG system will generate more accurate answers. That needs a separate check.
What reranking cannot fix
The biggest limitation is straightforward:
A reranker cannot recover a passage that never entered its candidate set.
If the required evidence is missing from the initial 20 passages, reordering those 20 will not solve the problem.
This helps separate two failure points:
- Retrieval failure: The required evidence never entered the candidate set.
- Selection failure: The evidence was retrieved but did not survive the final cutoff.
Reranking also does not establish whether a source is correct, current, or authoritative. An outdated passage can be highly relevant to a question.
And because the reranker adds another processing stage, any quality improvement comes with a latency cost that should be measured.
How I would test it
I would start with a small, fixed set of questions and manually label the passages that support each answer.
Then I would compare:
Baseline: Retrieve 20 candidates with hybrid search and keep the first five.
With reranking: Take those same 20 candidates, rerank them, and keep the first five.
Keeping the candidate set identical matters. Otherwise, changes in retrieval could be mistaken for improvements from reranking.
For each configuration, I would measure:
| Metric | What it measures |
|---|---|
| Recall@5 | The fraction of all labeled relevant passages that appear in the final five |
| MRR@5 | How early the first relevant passage appears, averaged across questions; zero if none appears in the top five |
| Added latency | The extra processing time introduced by reranking |
For example, if a question has four labeled relevant passages and two appear in the final five, its Recall@5 is 2 ÷ 4 = 0.5.
If its first relevant passage appears at position two, its reciprocal rank is 1 ÷ 2 = 0.5.
With exactly one relevant passage per question, Recall@5 becomes a hit-or-miss measure. With several relevant passages, finding one does not mean we have recovered all the required evidence.
I would also check whether the initial 20 candidates contain the relevant passages. That helps explain whether a poor result began with retrieval or with reranking.
Finally, I would run answer generation with the same model and prompt, then inspect answer correctness and citation support. Better passage rankings are useful, but I want to know whether they improve the answer the user receives.
The failures are worth inspecting
An average improvement can hide individual regressions.
I would look closely at questions where reranking:
- Pushes an answer-bearing passage below the cutoff.
- Prefers a general explanation over a specific exception.
- Selects several repetitive passages while losing complementary evidence.
Those cases can tell us where the reranker’s relevance judgments differ from what the application actually needs.
Today’s takeaway
Reranking gives a RAG pipeline another opportunity to select useful evidence before generation.
Whether it earns its place depends on the results: does it preserve more relevant evidence, rank it earlier, and improve answer quality at an acceptable latency?
Have you measured whether reranking improves Recall@5 in your RAG pipeline—and how much latency did it add?
I’d be interested in results where it helped, as well as cases where it made things worse.
Top comments (0)