My RAG evaluation would tell me hybrid: 0.86 and I'd sit there with no idea what to actually change. Raise it how? Different embedder? Smaller chunks? Add a reranker? The aggregate score names a winner; it says nothing about why the losers lost.
So I built a small tool to answer the "why," and running it turned up something I didn't expect: in my best-performing configurations, the component I'd added to improve results — the reranker — was responsible for almost every remaining failure.
Here's the idea, the finding, and how to check it on your own corpus.
The problem with a single retrieval score
hit@k = 0.86 is an average over a lot of independent failures, and those failures don't all have the same cause. A query can miss because:
- the answer text never made it into any chunk (a chunking/ingestion problem),
- neither dense nor sparse retrieval fetched the right chunk into the shortlist,
- dense + sparse fusion dropped it,
- the reranker reordered it out of the top-k,
- it landed just past the top-k cutoff,
- or the context-token budget dropped it.
Every one of those wants a different fix. Averaging them into
0.86throws away the only information that tells you which lever to pull.
Attribute each miss to the stage that lost it
The fix is to stop treating retrieval as a black box and record what survived at each stage of the pipeline, then attribute every missed query to the earliest stage that could no longer cover the answer:
representation → ann_index → candidate_generation → fusion →
reranker_demotion → final_cutoff → budget_cutoff
One important detail that makes this stable: gold answers are stored as character spans in the source document, not chunk IDs. That way the labels don't break when you change chunking strategy, and the scorer can credit an answer that's covered by several chunks together.
What the sweep actually showed
I ran 50 configurations (chunking × embedding × dense/BM25/hybrid × reranking, with real E5/BGE embedders and a cross-encoder reranker) over a small synthetic API-documentation corpus — 22 documents, 400 labeled queries. Three things jumped out, and only the attribution view makes them visible.
1. Chunking was the biggest lever — not the embedding model
The spread from worst to best config was hit@k 0.79 → 0.99. The bottom of the table was dominated by small fixed 200 chunks, whose failures were mostly "not retrieved" and "final cutoff," and whose hits were often flagged "fragile" — the answer was covered only because several chunks pieced it together, so a small chunking change would break it. The top was parent-child 800x200. Swapping E5 for BGE barely moved anything by comparison. If I'd only stared at aggregate scores I'd have fiddled with embedders; the attribution said chunk strategy was where the wins were.
2. In the strong configs, the remaining misses were almost all the reranker
This is the one that surprised me. Take a strong config — e5 · semantic · dense · rerank ce, hit@k 0.97. Where did its remaining misses go?
Reranker demotion: 13 (all of them)
Every single miss was the cross-encoder pulling the correct chunk out of the top-k. And it wasn't a one-off — across the reranked configs the residual misses were overwhelmingly reranker_demotion. Compare the same config without the reranker (e5 · semantic · dense, hit@k 0.96): now the misses are all final_cutoff — chunks that were ranked fine but landed one slot past k.
So the honest read isn't "rerankers are bad." The reranker raised MRR nicely (0.80 → 0.86) and nudged hit@k up. But once retrieval was already strong, the reranker became the single largest source of the failures that were left — which points at a precise, small fix (increase candidate depth / rerank top-N, or tune the reranker), not "retrieval is broken, start over."
Update: Two comments pushed on this finding, and they were right to. The reranker_demotion count is checked earliest-stage-first — a query only reaches that check once every upstream stage already succeeded for it — so it's conditional on the rest of the pipeline, not a standalone measure of the reranker's damage. Diffing rerank=none vs rerank=ce on the same 400 queries by query id (same candidate set both times, the reranker only reorders it) gives 12 promoted, 9 demoted, net +3 hits. The reranker is net positive.
It's also worse than a conditional count: of the 13 misses labeled reranker_demotion, only 9 are queries the reranker actually flipped. The other 4 were already outside the top-k before reranking (final_cutoff under the no-rerank config) — the reranker changed nothing for them either way. But once a reranker is configured, final_cutoff is unreachable in the attribution, so those 4 pre-existing failures inherit the reranker_demotion label anyway. "13, all of them" overstated the reranker's real damage by about 30%.
The 9 genuine demotions aren't random either — several cluster on passages that pack multiple same-type named entities close together (three competing stadium names in one paragraph; two named opponents in another). It reads as a precision problem on crowded passages, not "rerankers are bad" and not a query-type pattern.
3. Confidence intervals stop you from over-reading the leaderboard
The top config was 0.99 [0.98–1.00]; the next few were 0.98 [0.97–0.99]. Those intervals overlap — on 400 queries, 0.99 vs 0.98 is a tie, not a win. Without the CI you'd "pick the 0.99" and congratulate yourself on noise.
4. Half the context for ~two points of quality
The best config hit 0.99 but at ~577 average retrieved tokens. An e5 · recursive 400 · hybrid · ce config reached 0.963 at ~311 tokens — within about three points for roughly half the context. If you're context- or cost-bound, that's the smarter pick, and the quality-vs-tokens (Pareto) view is what surfaces it.
Try it on your corpus
The offline demo needs no API key or model download:
pip install retrieval-lab
retrieval-lab demo
For a real sweep you give it two JSONL files — your documents and your labeled queries — and it writes a single self-contained HTML report (rankings, confidence intervals, per-stage attribution, latency/cost, and the Pareto view):
pip install "retrieval-lab[real-embed,rerank]"
retrieval-lab run \
--corpus docs.jsonl \
--queries queries.jsonl \
--embed-models e5,bge \
--chunkers fixed:200,fixed:400,recursive:400,parentchild:800x200 \
--retrieval dense,sparse,hybrid \
--rerank none,ce \
--html report.html
Live example report (the 50-config sweep above): https://ashwinugale.github.io/Retrieval-Lab/
Code: https://github.com/AshwinUgale/Retrieval-Lab
The honest limits
- It's only as representative as your labeled query set — a thin or biased set biases the winner. Every score describes your corpus, never "best" in the abstract.
- Missing valid gold alternatives make measured recall a lower bound.
- Latency and index cost are whatever your machine reports.
- Stage attribution needs a decomposable pipeline; a black-box retriever can only be scored at its output. It's beta. If the attribution gets something wrong for you — misattributes a miss, or blames a stage you don't think is at fault — that's exactly the feedback I want. What's the failure stage you wish your RAG eval could point at?
Top comments (7)
Attributing each miss to a stage is the right move and the character-span decision is the one that makes it reusable. Storing gold as chunk IDs is how these harnesses quietly stop working the moment somebody changes the chunker, and almost everyone does it that way first.
One caution about reading the output. Attributing to the earliest stage that could no longer cover the answer is the correct rule, and it makes the counts sequential rather than independent, so they are not effect sizes. A query lost at
candidate_generationnever reaches the reranker, so the reranker's count is conditional on everything upstream having succeeded. Fix the retrieval stage and the reranker's share will move even though the reranker did not change. That is expected under the rule and it does look alarming the first time.The consequence is that "the reranker caused most of my remaining misses" is a statement about your current configuration rather than about rerankers, which I think you know, but the headline will travel further than the caveat.
If you want the counterfactual instead of the attribution, the cheap version is to re-run the same queries with the reranker removed and diff the covered set. Attribution tells you where the answer died. The ablation tells you whether the stage is net negative, and those come apart more often than you would expect, because a reranker that demotes 40 answers can still be promoting 200.
rerank=none vs rerank=ce on the same 400 queries — same candidate_n=15 set, the reranker only reorders it: 12 promoted, 9 demoted, net +3 hits (0.960 → 0.9675), MRR 0.803 → 0.856. Net positive — that's the with/without comparison you're asking for.
There's a second effect stacked on top of the conditional-count issue: of the 13 queries the report labels reranker_demotion, only 9 are queries the reranker actually flipped. The other 4 were already outside the top-5 before reranking (final_cutoff under the no-rerank config) — the reranker made no difference to them either way. But once a reranker is configured, final_cutoff is unreachable in the attribution DAG, so those 4 pre-existing failures inherit the reranker_demotion label anyway. 13-misses-all-reranker overstates the reranker's actual damage by about 30%.
Adding the promoted/demoted diff as a first-class view in the tool, and correcting the post with these numbers.
The miss taxonomy is the useful part here. A leaderboard number tells you the reranker helped on average, but reranker demotion tells you exactly where to spend the next hour. I like that because it turns tuning into a small change to candidate depth instead of a vague retrieval rewrite.
The earliest-stage attribution framing is the part I'd been missing. I'd been looking at aggregate hit@k and arguing about embedders when the actual bottleneck was further up. The character-span approach for gold labels is something I'd want to lift into our pipeline too: we've been using chunk IDs and every chunking experiment means re-annotating labels, which is exactly the thing that makes iteration slow. Curious whether you found the reranker's demotion pattern varied by query type or was it pretty uniform across categories?
Not by query type in any clean way — I checked. The demotions land at about the same low rate across who/what/how-many buckets (who 4, what 3, how-many 1 out of 9), so no category is systematically fragile, and at n=9 the buckets are too small to read much into anyway.
The pattern that does hold is at the passage level, not the query level: the demotions cluster on passages that pack several same-type named entities close together. Four are venue-selection questions — which stadium, who voted, when — and that passage names three competing stadiums across two cities. Two more are opponent questions where the same paragraph names both the divisional-round opponent and the championship opponent. The cross-encoder is good at judging whether a passage is on-topic and weaker at picking the one correct entity out of several plausible same-type candidates crammed together — so it's a property of the source text, not the phrasing.
On the character-span point — that's the piece I'd most encourage lifting. The chunk-ID re-annotation tax is exactly what it removes: label once against source offsets, and every chunking change re-scores by coverage with no re-labeling. It's also what lets the scorer credit an answer split across several chunks, which chunk-ID matching can't represent at all.
Attributing each miss to the stage that lost it is the move almost nobody makes, and it is why a single hit@k number is so misleading, it hides five different bugs under one average. The reranker result does not surprise me: a cross-encoder trained on generic relevance will happily demote a chunk that is exactly right for your domain. Did the reranker misses cluster on one query type, or were they spread evenly?
The 9 demoted queries are too few for wh-word buckets to mean much (who: 5, what: 3, how-many: 1), but there's a real cluster underneath: several sit on passages with multiple named entities of the same type packed close together.
Four of the 13 reranker_demotion-labeled misses concern the venue-selection process — which stadium, who voted, when — and that passage names three competing stadiums across two cities. Two more are opponent-disambiguation questions, where the source passage names both the divisional-round opponent and the championship opponent in the same paragraph.
The pattern isn't query type, it's passage density. A cross-encoder trained on generic relevance is good at judging whether a passage is on-topic and weaker at picking the one correct entity out of several same-type candidates in a single passage. n=9, not proven, but the mechanism is concrete enough to expect it to replicate.