What makes a retrieval architecture work and what makes it fail?
Retrieval is one of the easiest parts of a RAG system to underestimate.
Discussions about RAG performance often focus on the LLM: the model, the prompt, the context window, or the quality of generation. But before any of that matters, the retriever has to find the right evidence.
Vector, hybrid, and graph retrieval make different assumptions about what makes a document relevant. The interesting part isn't simply which one wins a benchmark. It's understanding where those assumptions hold, where they break, and what each architecture depends on.
That requires more than swapping one retriever for another.
If chunking, embeddings, top-k, prompts, or the LLM change at the same time, any difference in performance becomes difficult to attribute. A fair comparison therefore requires controlling those variables and changing one thing at a time.
So that's what this benchmark does.
And that's where things got interesting.
The benchmark didn't just reveal which retriever performed best. It also exposed how easily a perfectly reasonable looking number could lead to the wrong conclusion.
Why this benchmark matters
Retrieval-Augmented Generation has become a common architecture for grounding LLMs. But discussions around retrieval often sound surprisingly absolute.
- Graph RAG is the future.
- Hybrid always wins.
- Just add a reranker.
- Bigger models produce better answers.
The goal here wasn't simply to declare a winner. It was to understand what makes each retrieval architecture work, where it breaks, and what its performance depends on.
That meant building three retrieval pipelines under controlled conditions and changing exactly one component: the retriever.
The Rules
The benchmark compares three retrieval strategies.
- Vector (FAISS): semantic retrieval
- Hybrid (BM25 + Reciprocal Rank Fusion): lexical + semantic retrieval
- Graph: spaCy entity graph
Everything else stayed identical.
- Same corpus
- Same chunking
- Same embeddings
- Same prompt
- Same LLM
- Same top-k
Only the retriever changed.
- Pure retrieval evaluation: nDCG@10 on BEIR ground truth (no LLM in the loop).
- End-to-end QA: identical prompt, context format, and LLM.
Trap #1: The Graph Found the Best Document
Except it didn't.
The graph retriever worked well on a small corpus. Then I increased the dataset from roughly 100 documents to 500. Suddenly, a question that had always worked started failing.
What is the capital of Afghanistan?
The answer became:
Not enough information.
Retrieval is deterministic, so I knew this wasn't random. I inspected the rankings.
| document | score | semantic similarity | entity bonus | # entities |
|---|---|---|---|---|
| June | 6.11 | 0.000 | 20.37 | 53 |
| People's Republic of China | 3.06 | 0.000 | 10.18 | 26 |
| Islamic world | 2.39 | 0.000 | 7.96 | 22 |
| Afghanistan | 2.09 | 0.62 | 4.90 | 11 |
| Afghanistan | 2.04 | 0.76 | 4.26 | 11 |
The winning document wasn't Afghanistan. It was June, a Wikipedia page listing national holidays. Its semantic similarity to the question was literally 0.000.
The culprit wasn't the graph itself. It was my scoring function.
I had combined:
$$\text{Final Score} = \text{Semantic Similarity} + \text{Entity Bonus}$$
The semantic similarity correctly favored Afghanistan. The entity bonus overwhelmed everything else. Documents mentioning many entities accumulated enormous bonuses, even when they weren't relevant.
My score wasn't measuring relevance anymore. It was measuring entity count.
The fix itself was simple. Normalize the entity bonus.
The difficult part was deciding how much to normalize. I was sure a gentle penalty would win. Instead of trusting that intuition, I evaluated six normalization strategies on a held-out validation split.
| normalizer | validation score |
|---|---|
| p75 | 0.4864 |
| linear | 0.4849 |
| √ | 0.4806 |
| log | 0.4793 |
| p25 | 0.4648 |
| none | 0.4533 |
A strong normalization won. I had expected the opposite.
Interestingly, a different normalization produced a slightly better score on one test dataset. Changing to it would have improved the benchmark. It also would have invalidated the experiment.
That's exactly why validation and test sets exist.
📌 Key takeaway
A scoring function can optimize exactly what you asked it to optimize while completely missing what you actually care about.
Always inspect why a document ranks first, not just that it ranks first.
The benchmark, implementation, and evaluation code are open source:
The full repository includes the retrieval implementations, benchmark configuration, evaluation code, and experiments discussed in this article.
Part 1 of the series. More traps to come.
There are some lessons you can only learn the hard way.
As we say in French: “C’était Guy Olivier.” ✌️
Top comments (0)