Breaking RAG Benchmarks: 5 Metric Traps I Fell Into — Part 1

Guy Olivier Millimouno — Fri, 21 Aug 2026 12:22:49 +0000

What makes a retrieval architecture work and what makes it fail?

Retrieval is one of the easiest parts of a RAG system to underestimate.

Discussions about RAG performance often focus on the LLM: the model, the prompt, the context window, or the quality of generation. But before any of that matters, the retriever has to find the right evidence.

Vector, hybrid, and graph retrieval make different assumptions about what makes a document relevant. The interesting part isn't simply which one wins a benchmark. It's understanding where those assumptions hold, where they break, and what each architecture depends on.

That requires more than swapping one retriever for another.

If chunking, embeddings, top-k, prompts, or the LLM change at the same time, any difference in performance becomes difficult to attribute. A fair comparison therefore requires controlling those variables and changing one thing at a time.

So that's what this benchmark does.

And that's where things got interesting.

The benchmark didn't just reveal which retriever performed best. It also exposed how easily a perfectly reasonable looking number could lead to the wrong conclusion.

Why this benchmark matters

Retrieval-Augmented Generation has become a common architecture for grounding LLMs. But discussions around retrieval often sound surprisingly absolute.

Graph RAG is the future.
Hybrid always wins.
Just add a reranker.
Bigger models produce better answers.

The goal here wasn't simply to declare a winner. It was to understand what makes each retrieval architecture work, where it breaks, and what its performance depends on.

That meant building three retrieval pipelines under controlled conditions and changing exactly one component: the retriever.

The Rules

The benchmark compares three retrieval strategies.

Vector (FAISS): semantic retrieval
Hybrid (BM25 + Reciprocal Rank Fusion): lexical + semantic retrieval
Graph: spaCy entity graph

Everything else stayed identical.

Same corpus
Same chunking
Same embeddings
Same prompt
Same LLM
Same top-k

Only the retriever changed.

Pure retrieval evaluation: nDCG@10 on BEIR ground truth (no LLM in the loop).
End-to-end QA: identical prompt, context format, and LLM.

Trap #1: The Graph Found the Best Document

Except it didn't.

The graph retriever worked well on a small corpus. Then I increased the dataset from roughly 100 documents to 500. Suddenly, a question that had always worked started failing.

What is the capital of Afghanistan?

The answer became:

Not enough information.

Retrieval is deterministic, so I knew this wasn't random. I inspected the rankings.

document	score	semantic similarity	entity bonus	# entities
June	6.11	0.000	20.37	53
People's Republic of China	3.06	0.000	10.18	26
Islamic world	2.39	0.000	7.96	22
Afghanistan	2.09	0.62	4.90	11
Afghanistan	2.04	0.76	4.26	11

The winning document wasn't Afghanistan. It was June, a Wikipedia page listing national holidays. Its semantic similarity to the question was literally 0.000.

The culprit wasn't the graph itself. It was my scoring function.

I had combined:

$$\text{Final Score} = \text{Semantic Similarity} + \text{Entity Bonus}$$

The semantic similarity correctly favored Afghanistan. The entity bonus overwhelmed everything else. Documents mentioning many entities accumulated enormous bonuses, even when they weren't relevant.

My score wasn't measuring relevance anymore. It was measuring entity count.

The fix itself was simple. Normalize the entity bonus.

The difficult part was deciding how much to normalize. I was sure a gentle penalty would win. Instead of trusting that intuition, I evaluated six normalization strategies on a held-out validation split.

normalizer	validation score
p75	0.4864
linear	0.4849
√	0.4806
log	0.4793
p25	0.4648
none	0.4533

A strong normalization won. I had expected the opposite.

Interestingly, a different normalization produced a slightly better score on one test dataset. Changing to it would have improved the benchmark. It also would have invalidated the experiment.

That's exactly why validation and test sets exist.

📌 Key takeaway

A scoring function can optimize exactly what you asked it to optimize while completely missing what you actually care about.

Always inspect why a document ranks first, not just that it ranks first.

The benchmark, implementation, and evaluation code are open source:

GitHub repository

The full repository includes the retrieval implementations, benchmark configuration, evaluation code, and experiments discussed in this article.

This is Part 1 of RAG Under the Microscope. More traps to come.

There are some lessons you can only learn the hard way.

As we say in French: “C’était Guy Olivier.” ✌️

DEV Community: Guy Olivier Millimouno

Breaking RAG Benchmarks: 5 Metric Traps I Fell Into — Part 1

Why this benchmark matters

The Rules

Trap #1: The Graph Found the Best Document