Your RAG app retrieves five chunks. They look relevant, but none actually answers the question.
The LLM responds confidently anyway.
The result? A fluent answer built on the wrong evidence.
Before changing the prompt or switching models, ask:
Did the retriever find the evidence needed to answer?
That’s the question I’ll explore by comparing BM25, vector search, and hybrid retrieval with Reciprocal Rank Fusion (RRF).
Where Vector Search Helps—and Where It Can Miss
Vector search uses embeddings to find passages with similar meaning, even when the wording differs.
Question: “How do I reset my password?”
Document: “Recover account access by updating your credentials.”
Semantic similarity can help connect these passages.
But consider:
How do I fix error
AUTH_403?
Vector search might rank general login troubleshooting advice above the passage containing the specific solution.
It can retrieve exact identifiers, too. However, semantic similarity doesn’t guarantee that the most useful evidence ranks first.
Where BM25 Helps
BM25 ranks documents using keyword matches, accounting for term frequency, term rarity, and document length.
It can be useful for:
- Exact error codes such as
AUTH_403 - Product names and technical terms
- Queries where specific words matter
Its weakness is vocabulary mismatch. A question about “resetting a password” may miss a passage about “credential recovery” when important terms don’t overlap.
| Method | Main strength |
|---|---|
| BM25 | Keyword matching |
| Vector search | Semantic similarity |
Their strengths are complementary, but neither guarantees the right evidence.
How Hybrid Search Uses RRF
Hybrid search combines keyword and vector retrieval.
One common way to merge their results is Reciprocal Rank Fusion:
RRF(d) = Σ 1 / (k + rankᵢ(d))
Here:
-
dis a document or chunk. -
rankᵢ(d)is its position in each result list, starting at 1. -
kis a smoothing constant, commonly set to 60. - A chunk absent from a list receives no contribution from that list.
RRF combines ranking positions instead of directly adding BM25 scores to vector similarity scores, which have different scales.
Chunks appearing near the top of multiple lists can receive strong fused scores.
But agreement between retrievers doesn’t prove that a chunk answers the question.
A Simple Example: RRF Can Rank General Advice First
Suppose the question is:
How do I fix error
AUTH_403?
We have three chunks:
- A: Contains the exact solution.
- B: Contains general login troubleshooting advice.
- C: Explains password resets.
The rankings are:
| Rank | BM25 | Vector Search |
|---|---|---|
| 1 | A | B |
| 2 | B | C |
| 3 | C | A |
With equal-weight RRF and k = 60, Chunk B ranks first, followed by A and C.
B appears near the top of both lists, even though A contains the answer.
This is an illustrative example, not an experimental result. It demonstrates a ranking limitation: A remains retrievable, but it loses the first position.
RRF combines rankings. It doesn’t independently verify answer relevance.
What I’ll Test
I’ll compare three strategies on the same document collection:
- BM25 only
- Vector search only
- Hybrid search with RRF
The initial pilot will contain 15–20 questions covering exact identifiers, paraphrases, and queries combining both.
Before comparing outputs, I’ll manually label the chunks containing supporting evidence. Every method will use the same corpus, chunking, questions, and labels.
Each standalone retriever will return up to 20 candidates. Hybrid retrieval will fuse up to 20 from each branch—a maximum of 40 unique chunks—before selecting the top five.
What I’ll Measure
- Hit@5: Does at least one supporting chunk appear in the top five?
- MRR@5: How highly does the first supporting chunk rank? A miss scores zero.
- Retrieval latency: How long does retrieval take, including query embedding time where applicable?
- Failure patterns: Which questions does each method miss, and why?
I’ll document whether hybrid branches run concurrently. Its larger candidate pool also matters when interpreting quality and latency differences.
This is a proposed pilot, not completed research. It should reveal useful failure patterns, but stronger conclusions would require a larger evaluation.
The Takeaway
Hybrid search combines lexical matching with semantic similarity. Whether it improves retrieval depends on the documents, questions, and settings.
Measure it. Inspect the misses. Check whether the retrieved chunks support an answer.
Before blaming the LLM, inspect the context it receives.
Does your RAG system struggle more with exact identifiers or with questions phrased differently from your documents?
Top comments (1)
For anyone interested in exploring the research behind BM25, dense retrieval, and RRF:
BM25 and Beyond:
doi.org/10.1561/1500000019
Dense Passage Retrieval for Open-Domain Question Answering:
aclanthology.org/2020.emnlp-main.550/
Reciprocal Rank Fusion:
doi.org/10.1145/1571941.1572114
These papers provide useful background on the retrieval methods discussed in this post.