DEV Community: Igor Eduardo

Extra Retrieval Rounds Can Make RAG Worse at Saying "I Don't Know"

Igor Eduardo — Fri, 23 Oct 2026 07:24:51 +0000

A new benchmark paper, DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge (arXiv 2610.28340, posted 2026-10-20), measures something most RAG scoreboards skip: how often a system admits that neither its documents nor its weights can answer the question.

The setup is clever. The authors build a retrieval corpus from Minecraft Wiki pages as they stood at a 31 March 2025 cut-off, then write unanswerable questions about game features documented only after that date. The benchmark has 700 answerable and 300 unanswerable questions, and they evaluate 27 LLMs. Their headline metric is Correct Abstention Rate (CAR): the share of unanswerable questions where the model acknowledges it lacks the answer. They score "premise denial", where the model asserts without support that the item does not exist, separately from an honest "I don't know".

What they report:

Single-shot retrieval helps. Compared with no retrieval, it raises CAR significantly in 38 of 40 model/thinking configurations. Mean CAR goes from about 17.1% to 56.4%.
Iterative retrieval gives part of that back. When the model can request up to two more retrieval rounds, CAR drops significantly in 30 of 40 configurations, to a mean of about 46.4%. Answer Accuracy on the answerable questions barely moves (about 68.5% â†’ 69.3%).
Adding query rewriting plus reranking to hybrid retrieval lowered CAR for all 12 models tested, from a mean of about 54.7% to 49.8%. The authors note that only 3 of the 60 per-model paired comparisons against plain hybrid are individually significant, so the direction is consistent but each per-model gap is small.
Retrieving more chunks also lowered it. On an 11-model subset, going from 5 to 10 chunks dropped mean CAR from about 56.8% to 51.7%.

The authors are careful about what this does and does not show. Their aggregate comparison does not isolate which part of iterative RAG causes the drop: chunk count, context length, follow-up queries and self-reflection all change together. Their corpus is one domain with one chunking scheme. Code and data are public at declinecraft-benchmark.

Those are their numbers, not mine. I have not re-run the benchmark. What I want to formalize is the eval habit the paper forces into the open.

The stake

Teams will ship the wrong upgrade.

The usual path for improving a RAG system is to add things: more chunks, a reranker, query rewriting, a second retrieval round. Each one is justified by an accuracy chart on answerable questions. In this paper, accuracy on answerable questions stays roughly flat while the system gets measurably worse at declining the questions it cannot answer. If your eval set has no unanswerable questions, that regression is invisible. You will ship it as an improvement.

AI failures rarely fail loud. A confident answer to a question your corpus cannot support looks exactly like a correct one on a dashboard that only grades answerable items.

Opinion (one sentence)

Every change to a retrieval pipeline should be graded on unanswerable questions too, because "more context" can buy a little accuracy while quietly spending the system's ability to say "I don't know."

That is a preference with a limiter, not a benchmark result. I am not saying iterative retrieval, reranking or query rewriting are mistakes. I am saying the contract for adding them must include the column that can go down.

What an abstention-aware eval contract looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Answerable accuracy	Did we answer the questions the corpus supports?	Shipping on vibes
Correct abstention rate	Did we decline the questions nothing supports?	Confident answers with no evidence
Premise denial, counted separately	Did we claim something does not exist without being able to verify it?	"No such thing" passing as caution
Attempt rate on answerable items	Did we refuse questions we could have answered?	A system that looks safe because it answers nothing
Per-change deltas	Which single change moved each column?	Crediting the whole pipeline for one component's regression

None of that needs a private harness. It needs someone to refuse a pipeline change that only reports the accuracy column.

Why "I don't know" is not the same as "no such thing"

The distinction I would steal first is premise denial.

When a corpus is older than the question, "I can't find this" and "this doesn't exist" are different claims. The first is honest. The second is a statement about the world that the system cannot back. A metric that lumps both into "declined" will reward the second. The paper's own example is a model that, asked about an item added after the cut-off, says the item is not recognized and then answers about a different item instead.

If you run retrieval over documentation that changes, such as product docs, internal wikis or release notes, this is the failure to watch. The question can be newer than your index.

Where this connects to habits I already keep public

My public work is mostly retrieve-first systems, and the same habit applies there: measure the retrieval half on its own before letting the answer vouch for it. My hybrid retrieval work is built around that habit. This week's paper adds a column I would put next to it: what the system does when the right evidence is not in the index at all.

Steal this check

Pick one RAG pipeline you are about to "improve" and add four things before you merge:

Write a small unanswerable slice. Questions about things newer than, or missing from, your corpus. Even a few dozen is better than none.
Grade abstention next to accuracy. Report both columns for the before and after versions of the pipeline.
Split premise denial from abstention. "I can't find it" and "it doesn't exist" go in different buckets.
Change one thing at a time. Extra chunks, reranking, rewriting and extra rounds each get their own before/after row.

Then ask whether the change is still an upgrade once the abstention column is visible.

What I am not claiming

I am not reporting DeclineCraft results as my own, and I have not reproduced the benchmark.
I am not saying iterative RAG, reranking or query rewriting should be removed. The paper's own caveat is that it cannot isolate which component drives the drop.
I am not publishing thresholds. What counts as an acceptable abstention rate depends on the product and on what a wrong answer costs.
One domain (a game wiki) is not every domain. The authors say so too.

The point

An accuracy chart on answerable questions cannot tell you whether your upgrade made the system more willing to bluff. Put unanswerable questions in the eval, grade "I don't know" as a real outcome, and make every pipeline change earn its place on both columns.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

An 8-Point nDCG Bump That Costs Two Minutes Is Not Free

Igor Eduardo — Mon, 12 Oct 2026 16:12:42 +0000

A new NVIDIA NeMo Retriever paper, Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks (arXiv 2610.05750, posted around 2026-10-05), puts numbers on a trade everyone arguing about agentic RAG already feels. With the same embedding model, their agentic retrieval setup lifts nDCG@10 by about 8.7 points over standard dense retrieval. It also averages roughly 107.4 seconds of latency against about 0.67 seconds for the standard path, and burns on the order of 764.1K input tokens plus 5.8K output tokens per query. They also report competitive results on ViDoRe v3 and BRIGHT without task-specific specialization. Their retrieval-bench notes live in the public NeMo-Retriever tree.

Those are their numbers, not mine. I have not re-run the bench. What I want to formalize is the eval shape the paper forces into the open.

Here is what I would require before I trusted an "agentic retrieval win," and what I would not claim yet.

The stake

Teams will ship the wrong default.

Most retrieval scoreboards still answer one question: "did we find better documents?" That question is real. It is incomplete. An 8-point nDCG bump that costs two minutes and three-quarters of a million input tokens is a different product than a sub-second dense call. If your dashboard only shows the quality column, the expensive path looks like a free upgrade. In production it is a budget, a timeout policy, and a fallback story.

AI failures rarely fail loud. They fail as bills that looked like wins on the eval sheet.

Opinion (one sentence)

An eval contract that reports only retrieval quality without latency, token cost, and failure/fallback rate is grading a different product than the one you will ship.

That is a preference with a limiter, not a benchmark result. The NeMo numbers stay theirs. What I am formalizing is the contract: quality, cost, and reliability sit in the same report, or you are not evaluating the system you will actually run.

What a cost-aware retrieval contract looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Retrieval quality	Did we find the right evidence under the stated metric?	Shipping on vibes
Latency distribution	How long did the path take, including agent loops?	A "win" that times out in UX or SLA
Token / dollar cost	What did one query cost on average and at the tail?	Quality gains that bankrupt the budget
Failure / fallback rate	How often did the agentic path abort, stall, or hand off?	A green mean that hides a red path
Same-model / same-corpus controls	Did we change one variable at a time?	Crediting the agent loop for an embedding upgrade

None of that needs a private harness. It needs someone to refuse a scoreboard that only prints nDCG.

Why the latency column belongs next to nDCG

The paper's latency gap is the part I would steal first as a reporting habit.

A sub-second dense path and a ~two-minute agentic path are not two settings of the same product. They are different products with different failure modes. One fails by missing evidence. The other can also fail by being too slow to use, too expensive to run at volume, or too brittle when a tool call hangs. If you only publish the quality delta, you are arguing about a system your users will never wait for.

I would report latency and token cost next to the quality metric, not bury them in an appendix. The same goes for failure and fallback rate: every run that aborted or handed back to a cheaper path belongs in its own column.

Where this connects to retrieval habits I already keep public

My public work is mostly retrieve-first systems, and the same split shows up there. A fluent answer can look like a retrieval win the same way a slow agent loop can look like a quality win. In both cases the fix is the same move: measure the retrieval half on its own, and refuse to let one number vouch for the whole product.

The hybrid retrieval pipeline I keep public (hybrid-rag-pipeline) is built around that habit: measure retrieval separately before letting generation, or orchestration, speak for it. This week's paper is a cost-shaped version of the same lesson.

Steal this check

Pick one retrieval path you already call "agentic" or "multi-hop" and add four things:

Keep the quality metric. Report nDCG (or your usual ranking metric) with the same embedding model and corpus controls you already trust.
Add latency next to it. Mean is not enough; look at the tail your users will feel.
Add token cost next to it. Input and output, average and heavy queries, in the same table as quality.
Count failure / fallback separately. Every abort, timeout, or handoff to a cheaper path goes in its own column. That column is the one that becomes an incident or a surprise bill.

Then ask whether the quality bump is still the headline once those columns are visible.

What I am not claiming

I am not reporting NeMo Retriever results as my own, and I have not reproduced their bench.
I am not saying agentic retrieval is a mistake. The paper shows a real quality gain on hard tasks. Cost is part of the grade, not a reason to ignore the gain.
I am not publishing thresholds. What counts as acceptable latency or token spend depends on the product and the query class.

The point

An 8-point nDCG bump that costs two minutes is not free. Grade the product you will ship: quality, cost, and the rate at which the expensive path fails closed.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Done Is Testimony. Terminal State Is the Grade.

Igor Eduardo — Mon, 05 Oct 2026 15:54:00 +0000

Two things landed in the same week and they are the same argument.

Microsoft and Hugging Face published ThinkingBox, a benchmark that grades agents on the backend state and side effects they leave behind instead of the sentences they generate, and then asks whether they can do it twenty times in a row. Their opening example is an agent that makes nine clean tool calls, closes a ticket as resolved, and is wrong: the carrier exception was still open, so the required end state was "on hold." A grader reading tool calls sees nine well-formed calls. The database disagrees.

On Dev.to, The Witness Was the Suspect made the same point from the incident side: when the agent writes its own success log, the record of the incident was authored by the cause of the incident.

Here is what I would require before I trusted an agent's "done," and what I would not claim yet.

The stake

Teams will ship the wrong default.

Most agent dashboards still answer "did it say it succeeded?" Status strings, tool-call traces, and a tidy final message are all produced by the same run that took the action. They are useful for debugging. They are not a grade. When the eval and the agent share a witness, a confident wrong action scores the same as a correct one, and the failure shows up later as a customer, a ledger, or an auditor.

AI failures rarely fail loud. They fail plausible. That is exactly why the grade has to come from somewhere the agent cannot write.

Opinion (one sentence)

An agent's "done" is testimony, not evidence. The eval contract should grade the terminal state through a read path the agent does not control, and it should grade it more than once.

That is a preference with a limiter, not a benchmark result. I have not run ThinkingBox, and its numbers (507 stateful workflows, 20 runs each) are theirs. What I am formalizing is the shape: separate the actor from the grader, and treat "said done while the state is red" as a named failure, not as noise.

What separating the witness looks like (menu, not recipe)

Field	Question it answers	Failure if skipped
Required end state	What should the record look like when the task is truly finished?	"Resolved" accepted when "on hold" was correct
Independent read path	Who reads the final state, and can the agent influence that read?	The agent's own log grades the agent
Side-effect ledger	What else changed (tickets, refunds, emails, rows)?	Correct primary record, wrong collateral writes
Done-while-red	Did the run claim success while the state check failed?	Confident wrong actions blend into the pass rate
Repeat consistency	Does the same task reach the same end state across N runs?	One lucky green run treated as a capability

None of that needs a private harness. It needs someone to write down the required end state before the run, and a grader that reads the system of record after the run.

Why repeats belong in the contract

ThinkingBox's "twenty times in a row" framing is the part I would steal first.

A single green run tells you the agent can reach the right state. Production needs to know whether it will. If the same ticket ends "resolved" on some runs and "on hold" on others, you do not have a capability with a small error rate. You have a coin with a nice transcript. I would report consistency next to the pass rate, not bury it in an appendix.

Where this connects to retrieval

My public work is mostly retrieve-first systems, and the same split shows up there. A RAG answer can cite cleanly and still be wrong about the source, the same way an agent can log cleanly and still leave the wrong record. In both cases the fix is the same move: grade against something the generator did not produce. For retrieval that is a gold source set; for agents it is the terminal state.

The hybrid retrieval pipeline I keep public (hybrid-rag-pipeline) is built around that habit: measure the retrieval half on its own before letting a fluent answer vouch for it.

Steal this check

Pick one agent workflow you already run and add three things:

Write the required end state first. Not "the agent responds," but "the ticket is on hold, no refund issued, customer told why."
Grade from the system of record. Read the final state with a query the agent never touches, and ignore the agent's own status message for scoring.
Count done-while-red separately. Every run where the agent claimed success but the state check failed goes in its own column. That column is the one that becomes an incident.

Then run it more than once and report how often it lands in the same place.

What I am not claiming

I am not reporting ThinkingBox results or reproducing their benchmark.
I am not saying traces and logs are useless. They are how you debug. They are just the wrong thing to score.
I am not publishing thresholds. What counts as acceptable consistency depends on what the action costs when it is wrong.

The point

The process that acted should not be the only witness that it worked. Grade the record, not the report.

Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.

Green Scores Without a Pinned Harness Are Screenshots

Igor Eduardo — Mon, 28 Sep 2026 16:09:46 +0000

Everyone is talking about Evaluation Cards and reproducible eval benches — the EvalEval × UK AISI write-up on Hugging Face, plus a week of discourse where green numbers still hide the wrong harness (and DeepEval-style “which half is broken?” chatter keeps landing). That heat is a live market object, not a niche footnote.

Here is what I would require before I trusted a green eval score for high-stakes retrieval or generation — and what I would not claim yet.

The stake

Teams will ship the wrong default.

A dashboard can show a comforting pass rate while the report never pins how the score was earned: which protocol, which compute budget, which dataset slice, and whether the number measured retrieval, generation, or a blended soup. Celebrate that number and you optimize a screenshot. The next engineer cannot reproduce the gate, and the next incident cannot tell which half failed.

If your domain is identifier-heavy, multi-hop, or regulated-adjacent, a fluent answer after an unpinned harness is not a win. It is a silent miss with a pretty badge.

Opinion (one sentence)

A green score without a pinned harness — protocol, compute, dataset slice, and which half of the pipeline it measures — is not an eval contract; it is a screenshot.

That is a preference with a limiter, not a SOTA claim. I have not run EvalEval in production. AISI card fields and anyone else’s leaderboard numbers stay theirs. What I am formalizing is the harness pin as a first-class part of eval-as-contract, from the same reliable-systems lane I already publish on.

What a pinned harness looks like (menu, not recipe)

Evaluation Cards earned attention because they make the boring fields visible: protocol, compute, data cut, scoring rules. You do not need their exact schema to steal the shape.

Field	Question it answers	Failure if skipped
Protocol	What procedure produced this number?	“We eval’d it” with no replay path
Compute	What budget / hardware / call limits?	Incomparable scores across teams
Dataset slice	Which queries / docs / holdouts?	Student-subset vibes framed as prod
Pipeline half	Retrieval, generation, or joint?	Green faithfulness while recall is broken (or the reverse)

The DeepEval-style lesson that keeps circulating is the same shape: a single RAG score can hide which half failed. Pin the half. Prefer two honest columns over one blended crown.

The check you can run without my internals

You do not need my private stacks (and I will not publish them). Steal this menu-only check:

Pin the harness block before the score. Same report header every time: protocol name/version, compute class, dataset slice id, and pipeline half (retrieval / generation / joint). If any field is missing, the number is not ready to celebrate.
Split halves when you report. For the same query set, publish retrieval quality and generation/faithfulness as separate columns. One green blended score is not enough — “which half is broken?” should be answerable from the table alone.
Freeze the evaluator before you chase labels. If the judge, metric, or prompt changed mid-board, say so. Moving the goalposts mid-week and claiming progress is theater.
Keep a missing-evidence arm. Same task with the gold document removed (or retrieval forced empty). If generation still narrates a confident answer, the eval contract is incomplete — regardless of how green the happy-path column looks.

No thresholds copied from anyone’s blog as “our production numbers.” No paste recipe. Calibrate cutoffs on your corpus; where missing evidence is costly, treat empty retrieval as a failure in the harness, not as a soft success.

Why query-class honesty transfers

On Portuguese clinical text, BM25 and dense retrieval solved different query classes; fusing them beat either alone on a public 500-query study. Exact terms and conceptual phrasing fail in different ways. That finding is checkable: open code at nomad-link-id/hybrid-rag-pipeline, companion write-up on Dev.to.

The lesson that transfers to Evaluation Cards is not “clone our fusion.” It is eval honesty on query classes and pipeline halves. If your harness cannot say which class and which half produced the green cell, you are not ready to ship the number into a decision.

What I will not claim

That EvalEval, UK AISI cards, or any one vendor harness is universally correct for every workload. Workload-honest eval first.
That we “ran EvalEval inside Cortexa/DocMinds.” We did not. Trend-jacking with fake usage is spam.
That a pinned harness alone proves end-to-end answer quality. Downstream faithfulness and missing-doc policy remain separate gates.
Internal thresholds, prompts, or clone guides for private products. Menu only: pin the fields, split the halves — not the spice blend.

Why this formalizes who we are

The market is amplifying Evaluation Cards and green eval dashboards. The thesis I want indexed next to that heat:

Eval as contract — protocol, compute, slice, and pipeline half are gates, not vibes.
Reliable systems — production AI fails on harness mismatch and wrong-half celebration — not only on model choice.
Retrieve-first — when the retrieval half is unpinned, generation fluency is not evidence.

If a sharp reader walks away believing this engineer will not celebrate a green score without a pinned harness, the post did its job. Follows that come from that are the point — not volume.

Soft pointers (contribution first)

Public hybrid pipeline (BM25 + dense + RRF): https://github.com/nomad-link-id/hybrid-rag-pipeline
Empirical companion (500 clinical queries, open methodology): Dev.to — Two Retrieval Methods Are Better Than One
Site / builder trail: https://igoreduardo.com
Market object that sparked this note: EvalEval × UK AISI Evaluation Cards (HF)

Stop here. No recipe. No “we proved SOTA.” Preference + limiter + public trail.

By Igor Eduardo · Austin, TX · Engineer of reliable search and AI systems for high-stakes science · https://igoreduardo.com

Ranking Isn't Judging: What the Jev Wave Still Owes Retrieval

Igor Eduardo — Thu, 24 Sep 2026 17:58:50 +0000

Ranking Isn't Judging: What the Jev Wave Still Owes Retrieval

Everyone is talking about typed judgment models for RAG — Jev-class scorers that keep or drop passages, not only reorder them. Hugging Face just got a clean write-up of jev-reranker: hybrid pool in, relevance filter out, fewer documents reaching the generator. That is a live market object, not a niche blog.

Here is what I would require before I trusted “we upgraded ranking to judging” for high-stakes retrieval — and what I would not claim yet.

The stake

Teams will ship the wrong default.

A hybrid lexical + dense pool can look fine on ranking metrics while still shipping distracting neighbors into the prompt. Rerank asks in what order? Relevance filtering asks should this context reach the generator at all? Those are different contracts. Collapsing them into one score hides the broken half.

If your domain is identifier-heavy, multi-hop, or regulated-adjacent, a fluent answer after a weak retained set is not a win. It is a silent miss.

Opinion (one sentence)

Ranking answers closeness; judging needs an explicit leave-out policy plus a faithfulness gate that is scored separately — never a single blended “quality” number.

That is a preference with a limiter, not a SOTA claim. I have not run Jev in production. Their NanoHotpotQA numbers stay theirs. What I am formalizing is the judgment type split, from the same retrieve-first / eval-as-contract lane I already publish on.

What ranking gets right — and where it stops

Hybrid retrieval earned its keep for a reason. On Portuguese clinical text, BM25 and dense retrieval solved different query classes; fusing them beat either alone on a public 500-query study. Exact terms (scores, drug names, identifiers) and conceptual phrasing fail in different ways. That finding is checkable: open code at nomad-link-id/hybrid-rag-pipeline, companion write-up on Dev.to, Zenodo preprint under CC BY.

Reranking on top of that pool is a natural next step. Cross-encoders and decision models both try to push the useful passages up. Fine — as a ranking layer.

Judging is different work:

Judgment	Question	Failure if skipped
Rank	Which of these hits is closer?	Wrong order; still often has something in context
Filter / leave-out	Should this hit reach the generator?	Distractors in the prompt; token waste
Faithfulness	Does the final answer’s claim live in the cited source?	Fluent invention with a real cite list

A leave-out policy is not “sort harder.” When the filter clears the deck, the system needs a first-class missing-evidence outcome — not a silent fallback into guessing.

The check you can run without my internals

You do not need my private stacks (and I will not publish them). Steal this menu-only check:

Split the report. For the same query set, publish (a) ranking quality on the candidate pool and (b) retained-set size / leave-out rate after the filter. One number is not enough — the HF jev-reranker table that pairs nDCG with documents retained is the right shape of honesty, independent of their thresholds.
Pin a missing-evidence arm. Same task with the filter forced to retain nothing (or with the gold document removed from the pool). If the generator still narrates a confident answer, the eval contract is incomplete.
Keep faithfulness separate. For every quoted span, verify membership against the cited source id. Fail closed on miss. Ranking/relevance can still look fine while the quote-span check fails — I have seen that failure mode in public discourse this week, and it matches retrieve-first practice.

No thresholds copied from anyone’s blog as “our production numbers.” No paste recipe. Calibrate cutoffs on your corpus; where missing evidence is costly, treat empty retained sets as failures in the harness, not as soft successes.

Hybrid pools make the split sharper

Once you run BM25 and dense in parallel, the candidate set is intentionally broad. That is a feature: complementary methods catch different query classes. It is also why a leave-out layer matters more, not less.

A wide hybrid pool without a filter ships more distractors. A filter without a missing-evidence policy ships fluent guesses when nothing survived. Rerank alone does not fix either failure — it only reorders the same set.

So when the market says “just add a judgment model,” I hear two separate jobs: upgrade the order when you still want generation, and upgrade the admit/deny policy when generation should be allowed to refuse. Publish both outcomes in the eval report. Prefer workload-honest tables over a universal crown for any one store, model, or scorer.

What I will not claim

That Jev (or any typed decision model) is universally better than a cross-encoder or a lexical filter. Workload-honest eval first.
That filter quality alone proves end-to-end answer quality. Downstream faithfulness and missing-doc policy are separate gates.
That we “shipped Jev inside Cortexa/DocMinds.” We did not. Trend-jacking with fake usage is spam.
Internal thresholds, prompts, or clone guides for private products. Menu only: what we serve is the judgment split and the eval shape — not the spice blend.

Why this formalizes who we are

The market is amplifying judgment models. The thesis I want indexed next to that heat:

Retrieve-first — evidence path before orchestration theater.
Hybrid / complementary search — lexical and dense cover different query classes; fusion needs honest eval, not a universal crown.
Eval as contract — exact-match, faithfulness, context precision/recall, paired leave-out tests — gates, not vibes.

If a sharp reader walks away believing this engineer will not collapse ranking into judging, and will fail closed when evidence is missing, the post did its job. Follows that come from that are the point — not volume.

Soft pointers (contribution first)

Public hybrid pipeline (BM25 + dense + RRF): https://github.com/nomad-link-id/hybrid-rag-pipeline
Empirical companion (500 clinical queries, open methodology): Dev.to — Two Retrieval Methods Are Better Than One
Preprint (CC BY): https://doi.org/10.5281/zenodo.19686739
Market object that sparked this note: Introducing jev-reranker (HF)

Stop here. No recipe. No “we proved SOTA.” Preference + limiter + public trail.

By Igor Eduardo · Austin, TX · Engineer of reliable search and AI systems for high-stakes science · https://igoreduardo.com

Two Retrieval Methods Are Better Than One: Evidence from 500 Clinical Queries

Igor Eduardo — Wed, 13 May 2026 19:14:30 +0000

When I set out to evaluate retrieval configurations for Portuguese clinical text, I expected one method to dominate. Instead, I found something more interesting: BM25 and dense retrieval solve different questions. Neither is a substitute for the other.

This post summarizes the methodology and results from a 500-query empirical study of hybrid retrieval for clinical question answering. All code is open source: https://github.com/nomad-link-id/hybrid-rag-pipeline

The Setup

500 clinical queries across 6 medical specialties (cardiology, endocrinology, infectology, nephrology, neurology, oncology). Each query has a single reference answer grounded in a specific passage from clinical documentation.

Four retrieval configurations were evaluated:

Config	Method
BM25-only	BM25 with Portuguese stopword removal
Dense-only	BioBERTpt embeddings, cosine similarity
Hybrid-RRF	BM25 + dense via Reciprocal Rank Fusion
Hybrid-Rerank	RRF candidates re-ranked with cross-encoder

What Is Reciprocal Rank Fusion?

RRF combines ranked lists from multiple retrievers without requiring score normalization:

def rrf_score(rankings: list[list[str]], k: int = 60) -> dict[str, float]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

Results

Config	Recall@5	MRR	Citation F1
BM25-only	0.71	0.64	0.82
Dense-only	0.68	0.61	0.78
Hybrid-RRF	0.84	0.77	0.91
Hybrid-Rerank	0.86	0.79	0.93

The Complementarity Finding

McNemar's test on BM25-only versus dense-only:

BM25 correct, dense incorrect: 89 queries
Dense correct, BM25 incorrect: 57 queries
McNemar chi2 = 39.55, p < 0.001

The asymmetry is statistically significant. Dense-only missed 22.2% of queries that BM25 solved. You need both.

Citation Verification

Deterministic approach (BM25 score threshold + exact n-gram overlap): 461/500 citations verified.

Prompt-based LLM approach (same passages, ask LLM "does this support the answer?"): 1/500.

The difference is task design, not model quality. A deterministic check measures actual textual overlap; a prompt check measures the model's opinion of the overlap.

Inter-Annotator Agreement

100 query-response pairs independently annotated by two reviewers. Cohen's kappa = 0.954 — near-perfect agreement on what constitutes correct retrieval for clinical text.

Practical Takeaway

Run both BM25 and dense retrieval
Use RRF to merge results
Implement deterministic citation verification
Measure complementarity with McNemar's test on your domain

Code and Data

Preprint: https://doi.org/10.5281/zenodo.19686739

Igor Eduardo | igoreduardo.com | ORCID: 0009-0005-6288-1135