When RAG Retrieval Improves Overall but Breaks Critical Queries

When building a Retrieval-Augmented Generation (RAG) application, changing an embedding model, chunking strategy, or retriever can improve overall retrieval metrics.

But does that mean every important query works better?

Not necessarily.

The problem

Imagine a RAG application with three queries: privacy, installation, and warranty information.

After updating the retrieval system, the following happens:

Query	Baseline Hit@1	Candidate Hit@1
Privacy (critical)	1	0
Installation	0	1
Warranty	0	1
Average	0.33	0.67

The average retrieval score improves, but the privacy query loses its relevant document.

A deployment decision based only on the aggregate metric could miss this regression.

Introducing OpenRAG Bench

I am developing OpenRAG Bench, an open-source, MIT-licensed Python toolkit for evaluating and comparing RAG retrieval results.

It provides:

• Retrieval metrics including Precision@k, Recall@k, MRR, MAP, and NDCG
• Baseline and candidate comparisons
• Configurable quality policies with OpenRAG Guard
• Golden Dataset Studio for human relevance labeling
• Capture & Replay for evaluating saved retrieval results
• Offline HTML and JSON reports

The core evaluation does not require an LLM API key or GPU.

Try it locally

You can reproduce the critical-query regression example using the current GitHub source code.

git clone https://github.com/ilker-Akbal/openrag-bench.git
cd openrag-bench

python -m pip install -e .

python examples/quickstart_capture_replay.py --output-dir reports/first-demo

Then open:

reports/first-demo/guard.html

The example performs actual keyword-based retrieval over a small synthetic document collection.

The candidate improves average NDCG@1 while losing a critical privacy document.

Expected outcome: BLOCKED

The example script exits successfully when it verifies that Guard correctly blocked the candidate. A direct blocked Replay command returns exit code 2.

This demonstration is available in the current GitHub source checkout, not in the published v0.6.0 wheel alone.

Real-data evaluation

I also evaluated lexical retrieval approaches using the BEIR SciFact test dataset, containing 5,183 documents and 300 judged queries.

The measured NDCG@10 results were:

Method	NDCG@10
TF-IDF	0.6329
BM25	0.6605

These results demonstrate a difference on this particular dataset. They do not establish universal model superiority or statistical significance.

What comes next?

OpenRAG Bench v0.6.0 is available as a public GitHub alpha release.

The next priority is independent developer feedback, particularly from people using LangChain, LlamaIndex, FAISS, or custom retrieval pipelines.

I would appreciate feedback about installation, stable document IDs, integration challenges, and confusing evaluation results.

GitHub: https://github.com/ilker-Akbal/openrag-bench

Quickstart: https://github.com/ilker-Akbal/openrag-bench/blob/main/docs/DEMO_60_SECONDS.md

Feedback: https://github.com/ilker-Akbal/openrag-bench/issues/4

OpenRAG Bench measures retrieval relevance. It does not directly evaluate generated answer factuality, hallucinations, or overall LLM safety.