If you've read more than three RAG tutorials, you've absorbed the same advice I have. Use hybrid search, not just dense. Agonise over your chunking strategy. Rerankers are a nice-to-have you can add later.
I'd nodded along to all of it without ever measuring any of it. So I built a small search lab, put each claim on a scale, and weighed it against real numbers on a real document.
The short version: most of that advice is "it depends on your data" wearing a lab coat. Some knobs the tutorials tell you to obsess over barely moved the needle for my corpus — and would move it differently for yours. One knob, treated as optional, helped clearly, and that one isn't really up for debate.
The interesting work is knowing which is which. The only way to know is to measure on your own data.
What I built
The corpus is the Digital Personal Data Protection Act, 2023 — India's data protection law. Public PDF, dense legal prose, exactly the kind of document a compliance assistant has to retrieve from.
The eval crosses three knobs, each a swappable lever in the code:
- Chunking: fixed-size vs. recursive (token-based, ~240 tokens, with overlap)
- Search mode: dense, sparse (BM25 via OpenSearch), hybrid (both, fused with Reciprocal Rank Fusion)
-
Reranker: none, a local cross-encoder (
ms-marco-MiniLM-L6-v2), or Cohere'srerank-v3.5
Embeddings are all-MiniLM-L6-v2, local, zero API spend. One uv command runs every config and prints a table.
The knobs are independent and swappable on purpose. I didn't know going in which one would matter, so I refused to hardcode a pipeline around a guess. That refusal is the whole design — and it's what lets anyone re-run the same grid on their own corpus.
Why no eval framework
When you sit down to evaluate RAG, the pull is toward a framework — Ragas, DeepEval, an LLM-as-judge that hands you a tidy score. I skipped all of them, on purpose.
The best thing I read while building this was Hamel Husain's Your AI Product Needs Evals, and his one rule is look at your data. A generic judge would cheerfully call a chunk "relevant" because it's topically about data protection — while missing that it sliced a legal clause in half right where the answer lived.
Generic tools hide exactly the specifics that make your case your case. Same reason there's no default stack: the details are the point, and a tool that averages them away can't help you choose.
So I evaluated retrieval only — no generation, no judge. Just: did the right span of text land near the top?
The metrics are deterministic and dull in the best way: MRR (how high the first correct result lands) and Hit@{1,3,5}. Relevance is labelled by source span — (page, char_start, char_end) — not chunk ID, so the same golden set scores both chunking strategies fairly. Twenty questions, hand-labelled.
What the numbers said
Hybrid barely beat dense. The "always use hybrid" reflex scored 0.648 vs. dense's 0.637 — +0.011, which is noise. On this corpus, the lexical half wasn't earning its keep on its own.
Sparse was weakest, and the reason is the lesson. Pure BM25 came in at 0.443, well under dense. This is a legal document queried in plain English: users ask to "delete my data," the Act says "erasure"; users say "close my account," the Act says the Data Principal may "withdraw consent." BM25 matches words and can't cross that gap — and legal text is full of it.
Flip the setup, though: a corpus of product SKUs or error codes, queried by people typing those tokens verbatim, and BM25 climbs. That weakness is a fact about my corpus and my users, not about sparse retrieval — the clearest "it depends" in the whole run.
Chunking didn't matter here. Fixed and recursive landed on identical MRR (0.746) on the reranked configs. I'd budgeted real worry for this. The data said: relax.
Reranking helped — clearly, and not just for me. Adding a reranker on top of hybrid moved MRR from 0.648 to 0.746 (local) or 0.817 (Cohere). That's a bigger jump than any retrieval-mode change, and unlike the others it isn't a quirk of my corpus — reranking improving retrieval is about as close to settled as RAG advice gets. The free local model captured most of the gain; Cohere took it further.
The real question with reranking isn't whether, it's cost — and that's genuinely two-sided. It adds a model call. But by sending the LLM fewer, better chunks, it can lower token cost and sometimes overall latency too. I didn't measure latency here, so I'll leave it at: the tradeoff isn't one-directional, and it's worth measuring for your own pipeline before assuming reranking is "expensive."
The bug my eval caught (that I'd never have spotted otherwise)
Question 4 — the definition of a Consent Manager — scored 0 across every single config. Not low. Zero. That's not a retrieval failure, it's a "you labelled it wrong" failure: I'd tagged the answer on page 5; it lived on page 2. A small per-query diagnostic (rank + coverage for each question) is what surfaced it. The takeaway is the same as the no-framework decision — you can't trust "measure it on your data" if you don't check your own measurements.
What I'd hold loosely
Two honest edges, stated once and not belaboured:
- Twenty questions is small, and my queries lean one way — 18 of 20 are full questions, only 2 are keyword fragments. Fragments are where BM25 tends to do better, so my sparse numbers are a lower bound for lexical retrieval, not a verdict on it.
-
I excluded the penalty table on page 21 —
pypdfshreds it into scattered fragments on extraction, so there's no honest span to label. Tabular extraction is a real problem, but a different one from retrieval.
Neither dents the headline. They're just more of the same point: change the corpus, the query style, or the model, and the contingent findings move. The reranking one mostly won't.
TL;DR
I measured RAG's common advice on one legal document with a 20-question retrieval eval. Most of it is data-dependent: "always use hybrid" barely beat plain dense, chunking strategy made no difference, and sparse was weakest only because legal prose and plain-English queries don't share vocabulary — on a token-exact corpus it would climb. The exception is reranking, which helped clearly and isn't really in doubt; the open question there is cost, not whether. That mix is exactly why the code keeps chunking, search mode, and reranker as independent levers — so you can run the same measurement on your data and get your answer. And the oldest rule held best: look at your data before you trust your metrics. It caught me labelling an answer on the wrong page.
Written with the assistance of an AI writing tool, working from my own eval results, code, and notes. Every claim and number comes from the experiment I ran; I've reviewed and rewritten throughout.
Full code — chunkers, retrievers, the eval harness, the golden set, and the script that made that chart:
iamchetansharma8
/
search-lab
Retrieval-evaluation harness over the DPDP Act 2023: measures MRR & Hit@k across chunking strategies, dense/sparse/hybrid search, and rerankers on a hand-labeled question set.
search-lab
A retrieval-evaluation harness over the Digital Personal Data Protection (DPDP) Act, 2023. It indexes the Act's text under interchangeable retrieval strategies and measures retrieval quality quantitatively against a fixed, hand-labeled question set, so that chunking, search-mode, and reranking choices can be made on evidence rather than assumption.
The system supports two chunking strategies, three search modes, and two rerankers, evaluated in any combination. Results are reported as Mean Reciprocal Rank (MRR) and Hit@{1,3,5} per configuration.
📝 A narrative writeup of these findings is on dev.to: Your RAG stack depends on your data. I measured how much.
Pipeline
- Ingestion — the DPDP Act PDF is parsed into per-page text with character-level provenance.
- Chunking — two strategies, sized in token space: fixed-size token windows and recursive (separator-aware) splitting.
-
Embedding — chunks are embedded locally with
all-MiniLM-L6-v2and persisted to Chroma. - Retrieval — dense (vector similarity), sparse (BM25 via OpenSearch), and…
If you run the grid on your own corpus, I'd genuinely like to know which findings held and which flipped — especially whether sparse climbs on a more keyword-heavy dataset. And if anyone has measured the latency and cost side of reranking properly, point me to it.
Top comments (1)
Very insightful