The reranker I added to improve RAG was causing most of my remaining misses

Ashwin Ugale — Mon, 10 Aug 2026 18:45:32 +0000

My RAG evaluation would tell me hybrid: 0.86 and I'd sit there with no idea what to actually change. Raise it how? Different embedder? Smaller chunks? Add a reranker? The aggregate score names a winner; it says nothing about why the losers lost.

So I built a small tool to answer the "why," and running it turned up something I didn't expect: in my best-performing configurations, the component I'd added to improve results — the reranker — was responsible for almost every remaining failure.

Here's the idea, the finding, and how to check it on your own corpus.

The problem with a single retrieval score

hit@k = 0.86 is an average over a lot of independent failures, and those failures don't all have the same cause. A query can miss because:

the answer text never made it into any chunk (a chunking/ingestion problem),
neither dense nor sparse retrieval fetched the right chunk into the shortlist,
dense + sparse fusion dropped it,
the reranker reordered it out of the top-k,
it landed just past the top-k cutoff,
or the context-token budget dropped it. Every one of those wants a different fix. Averaging them into 0.86 throws away the only information that tells you which lever to pull.

Attribute each miss to the stage that lost it

The fix is to stop treating retrieval as a black box and record what survived at each stage of the pipeline, then attribute every missed query to the earliest stage that could no longer cover the answer:

representation → ann_index → candidate_generation → fusion →
reranker_demotion → final_cutoff → budget_cutoff

One important detail that makes this stable: gold answers are stored as character spans in the source document, not chunk IDs. That way the labels don't break when you change chunking strategy, and the scorer can credit an answer that's covered by several chunks together.

What the sweep actually showed

I ran 50 configurations (chunking × embedding × dense/BM25/hybrid × reranking, with real E5/BGE embedders and a cross-encoder reranker) over a small synthetic API-documentation corpus — 22 documents, 400 labeled queries. Three things jumped out, and only the attribution view makes them visible.

1. Chunking was the biggest lever — not the embedding model

The spread from worst to best config was hit@k 0.79 → 0.99. The bottom of the table was dominated by small fixed 200 chunks, whose failures were mostly "not retrieved" and "final cutoff," and whose hits were often flagged "fragile" — the answer was covered only because several chunks pieced it together, so a small chunking change would break it. The top was parent-child 800x200. Swapping E5 for BGE barely moved anything by comparison. If I'd only stared at aggregate scores I'd have fiddled with embedders; the attribution said chunk strategy was where the wins were.

2. In the strong configs, the remaining misses were almost all the reranker

This is the one that surprised me. Take a strong config — e5 · semantic · dense · rerank ce, hit@k 0.97. Where did its remaining misses go?

Reranker demotion: 13   (all of them)

Every single miss was the cross-encoder pulling the correct chunk out of the top-k. And it wasn't a one-off — across the reranked configs the residual misses were overwhelmingly reranker_demotion. Compare the same config without the reranker (e5 · semantic · dense, hit@k 0.96): now the misses are all final_cutoff — chunks that were ranked fine but landed one slot past k.

So the honest read isn't "rerankers are bad." The reranker raised MRR nicely (0.80 → 0.86) and nudged hit@k up. But once retrieval was already strong, the reranker became the single largest source of the failures that were left — which points at a precise, small fix (increase candidate depth / rerank top-N, or tune the reranker), not "retrieval is broken, start over."

3. Confidence intervals stop you from over-reading the leaderboard

The top config was 0.99 [0.98–1.00]; the next few were 0.98 [0.97–0.99]. Those intervals overlap — on 400 queries, 0.99 vs 0.98 is a tie, not a win. Without the CI you'd "pick the 0.99" and congratulate yourself on noise.

4. Half the context for ~two points of quality

The best config hit 0.99 but at ~577 average retrieved tokens. An e5 · recursive 400 · hybrid · ce config reached 0.963 at ~311 tokens — within about three points for roughly half the context. If you're context- or cost-bound, that's the smarter pick, and the quality-vs-tokens (Pareto) view is what surfaces it.

Try it on your corpus

The offline demo needs no API key or model download:

pip install retrieval-lab
retrieval-lab demo

For a real sweep you give it two JSONL files — your documents and your labeled queries — and it writes a single self-contained HTML report (rankings, confidence intervals, per-stage attribution, latency/cost, and the Pareto view):

pip install "retrieval-lab[real-embed,rerank]"

retrieval-lab run \
  --corpus docs.jsonl \
  --queries queries.jsonl \
  --embed-models e5,bge \
  --chunkers fixed:200,fixed:400,recursive:400,parentchild:800x200 \
  --retrieval dense,sparse,hybrid \
  --rerank none,ce \
  --html report.html

Live example report (the 50-config sweep above): https://ashwinugale.github.io/Retrieval-Lab/
Code: https://github.com/AshwinUgale/Retrieval-Lab

The honest limits

It's only as representative as your labeled query set — a thin or biased set biases the winner. Every score describes your corpus, never "best" in the abstract.
Missing valid gold alternatives make measured recall a lower bound.
Latency and index cost are whatever your machine reports.
Stage attribution needs a decomposable pipeline; a black-box retriever can only be scored at its output. It's beta. If the attribution gets something wrong for you — misattributes a miss, or blames a stage you don't think is at fault — that's exactly the feedback I want. What's the failure stage you wish your RAG eval could point at?

Mutation testing, but for LLM evals — early experiment, would love feedback

Ashwin Ugale — Wed, 22 Jul 2026 19:54:49 +0000

I write evals for LLM features, wire them into CI, and move on. But a passing eval suite has a blind spot: it tells you nothing about whether it would actually fail if the model quietly got worse. Green isn't the same as good.

So I built muteval to measure that directly. It borrows mutation testing from software engineering — you deliberately break the thing under test and check whether your tests catch it. muteval degrades the system (weakens a prompt rule, drops a retrieved doc, swaps in a weaker model), reruns your existing eval suite against each degraded version, and reports what fraction of those injected regressions your evals caught. The ones they miss are concrete coverage gaps.

Mutation score: 33% (2/6 caught)
SURVIVED: deleted "if the answer isn't in the context, say you don't know"

A bit on how it works. It ships around 18 of these mutations, each modeling a way a real system quietly degrades: softening a "must" to a "should", flipping a "do not", dropping or corrupting a retrieved document, shuffling context, swapping in a cheaper model, or breaking a tool's output for agent setups. Before mutating anything it confirms your suite passes on the original system — if it doesn't, there's nothing meaningful to measure — then reruns your evals against each mutation, ranks whatever survives by severity, and suggests an eval that would close each gap.

Does it actually find anything real? I ran it on Vectara's open-rag-eval. Its citation check caught mutations that made the model stop citing its sources — but it completely missed one that removed the rule telling the model to say "I don't know" when the answer isn't in the context. The model kept citing sources while being free to make things up, and a citation check simply can't catch that. Once I added a check for that behavior, it caught it.

It's pure Python, no required dependencies, and works with your deepeval/RAGAS/promptfoo metrics or its own checks:

pip install muteval
muteval init      # scaffold a config
muteval check     # validate it, then muteval run

Where I think this goes next is a tool that rates an eval suite across several angles and suggests the evals you're missing — but I'd like to hear how other people approach it first. If you write LLM evals: how do you know they're any good?

Repo (Apache-2.0): https://github.com/AshwinUgale/muteval

DEV Community: Ashwin Ugale