The reranker I added to improve RAG was causing most of my remaining misses

My RAG evaluation would tell me hybrid: 0.86 and I'd sit there with no idea what to actually change. Raise it how? Different embedder? Smaller chunks? Add a reranker? The aggregate score names a winner; it says nothing about why the losers lost.

So I built a small tool to answer the "why," and running it turned up something I didn't expect: in my best-performing configurations, the component I'd added to improve results — the reranker — was responsible for almost every remaining failure.

Here's the idea, the finding, and how to check it on your own corpus.

The problem with a single retrieval score

hit@k = 0.86 is an average over a lot of independent failures, and those failures don't all have the same cause. A query can miss because:

• the answer text never made it into any chunk (a chunking/ingestion problem),
• neither dense nor sparse retrieval fetched the right chunk into the shortlist,
• dense + sparse fusion dropped it,
• the reranker reordered it out of the top-k,
• it landed just past the top-k cutoff,
• or the context-token budget dropped it. Every one of those wants a different fix. Averaging them into 0.86 throws away the only information that tells you which lever to pull.

Attribute each miss to the stage that lost it

The fix is to stop treating retrieval as a black box and record what survived at each stage of the pipeline, then attribute every missed query to the earliest stage that could no longer cover the answer:

representation → ann_index → candidate_generation → fusion →
reranker_demotion → final_cutoff → budget_cutoff

One important detail that makes this stable: gold answers are stored as character spans in the source document, not chunk IDs. That way the labels don't break when you change chunking strategy, and the scorer can credit an answer that's covered by several chunks together.

What the sweep actually showed

I ran 50 configurations (chunking × embedding × dense/BM25/hybrid × reranking, with real E5/BGE embedders and a cross-encoder reranker) over a small synthetic API-documentation corpus — 22 documents, 400 labeled queries. Three things jumped out, and only the attribution view makes them visible.

1. Chunking was the biggest lever — not the embedding model

The spread from worst to best config was hit@k 0.79 → 0.99. The bottom of the table was dominated by small fixed 200 chunks, whose failures were mostly "not retrieved" and "final cutoff," and whose hits were often flagged "fragile" — the answer was covered only because several chunks pieced it together, so a small chunking change would break it. The top was parent-child 800x200. Swapping E5 for BGE barely moved anything by comparison. If I'd only stared at aggregate scores I'd have fiddled with embedders; the attribution said chunk strategy was where the wins were.

2. In the strong configs, the remaining misses were almost all the reranker

This is the one that surprised me. Take a strong config — e5 · semantic · dense · rerank ce, hit@k 0.97. Where did its remaining misses go?

Reranker demotion: 13   (all of them)

Every single miss was the cross-encoder pulling the correct chunk out of the top-k. And it wasn't a one-off — across the reranked configs the residual misses were overwhelmingly reranker_demotion. Compare the same config without the reranker (e5 · semantic · dense, hit@k 0.96): now the misses are all final_cutoff — chunks that were ranked fine but landed one slot past k.

So the honest read isn't "rerankers are bad." The reranker raised MRR nicely (0.80 → 0.86) and nudged hit@k up. But once retrieval was already strong, the reranker became the single largest source of the failures that were left — which points at a precise, small fix (increase candidate depth / rerank top-N, or tune the reranker), not "retrieval is broken, start over."

3. Confidence intervals stop you from over-reading the leaderboard

The top config was 0.99 [0.98–1.00]; the next few were 0.98 [0.97–0.99]. Those intervals overlap — on 400 queries, 0.99 vs 0.98 is a tie, not a win. Without the CI you'd "pick the 0.99" and congratulate yourself on noise.

4. Half the context for ~two points of quality

The best config hit 0.99 but at ~577 average retrieved tokens. An e5 · recursive 400 · hybrid · ce config reached 0.963 at ~311 tokens — within about three points for roughly half the context. If you're context- or cost-bound, that's the smarter pick, and the quality-vs-tokens (Pareto) view is what surfaces it.

Try it on your corpus

The offline demo needs no API key or model download:

pip install retrieval-lab
retrieval-lab demo

For a real sweep you give it two JSONL files — your documents and your labeled queries — and it writes a single self-contained HTML report (rankings, confidence intervals, per-stage attribution, latency/cost, and the Pareto view):

pip install "retrieval-lab[real-embed,rerank]"

retrieval-lab run \
  --corpus docs.jsonl \
  --queries queries.jsonl \
  --embed-models e5,bge \
  --chunkers fixed:200,fixed:400,recursive:400,parentchild:800x200 \
  --retrieval dense,sparse,hybrid \
  --rerank none,ce \
  --html report.html

Live example report (the 50-config sweep above): https://ashwinugale.github.io/Retrieval-Lab/
Code: https://github.com/AshwinUgale/Retrieval-Lab

The honest limits

• It's only as representative as your labeled query set — a thin or biased set biases the winner. Every score describes your corpus, never "best" in the abstract.
• Missing valid gold alternatives make measured recall a lower bound.
• Latency and index cost are whatever your machine reports.
• Stage attribution needs a decomposable pipeline; a black-box retriever can only be scored at its output. It's beta. If the attribution gets something wrong for you — misattributes a miss, or blames a stage you don't think is at fault — that's exactly the feedback I want. What's the failure stage you wish your RAG eval could point at?

Comments

[Reid Marlow]

The miss taxonomy is the useful part here. A leaderboard number tells you the reranker helped on average, but reranker demotion tells you exactly where to spend the next hour. I like that because it turns tuning into a small change to candidate depth instead of a vague retrieval rewrite.