A new benchmark paper, DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge (arXiv 2610.28340, posted 2026-10-20), measures something most RAG scoreboards skip: how often a system admits that neither its documents nor its weights can answer the question.
The setup is clever. The authors build a retrieval corpus from Minecraft Wiki pages as they stood at a 31 March 2025 cut-off, then write unanswerable questions about game features documented only after that date. The benchmark has 700 answerable and 300 unanswerable questions, and they evaluate 27 LLMs. Their headline metric is Correct Abstention Rate (CAR): the share of unanswerable questions where the model acknowledges it lacks the answer. They score "premise denial", where the model asserts without support that the item does not exist, separately from an honest "I don't know".
What they report:
- Single-shot retrieval helps. Compared with no retrieval, it raises CAR significantly in 38 of 40 model/thinking configurations. Mean CAR goes from about 17.1% to 56.4%.
- Iterative retrieval gives part of that back. When the model can request up to two more retrieval rounds, CAR drops significantly in 30 of 40 configurations, to a mean of about 46.4%. Answer Accuracy on the answerable questions barely moves (about 68.5% → 69.3%).
- Adding query rewriting plus reranking to hybrid retrieval lowered CAR for all 12 models tested, from a mean of about 54.7% to 49.8%. The authors note that only 3 of the 60 per-model paired comparisons against plain hybrid are individually significant, so the direction is consistent but each per-model gap is small.
- Retrieving more chunks also lowered it. On an 11-model subset, going from 5 to 10 chunks dropped mean CAR from about 56.8% to 51.7%.
The authors are careful about what this does and does not show. Their aggregate comparison does not isolate which part of iterative RAG causes the drop: chunk count, context length, follow-up queries and self-reflection all change together. Their corpus is one domain with one chunking scheme. Code and data are public at declinecraft-benchmark.
Those are their numbers, not mine. I have not re-run the benchmark. What I want to formalize is the eval habit the paper forces into the open.
The stake
Teams will ship the wrong upgrade.
The usual path for improving a RAG system is to add things: more chunks, a reranker, query rewriting, a second retrieval round. Each one is justified by an accuracy chart on answerable questions. In this paper, accuracy on answerable questions stays roughly flat while the system gets measurably worse at declining the questions it cannot answer. If your eval set has no unanswerable questions, that regression is invisible. You will ship it as an improvement.
AI failures rarely fail loud. A confident answer to a question your corpus cannot support looks exactly like a correct one on a dashboard that only grades answerable items.
Opinion (one sentence)
Every change to a retrieval pipeline should be graded on unanswerable questions too, because "more context" can buy a little accuracy while quietly spending the system's ability to say "I don't know."
That is a preference with a limiter, not a benchmark result. I am not saying iterative retrieval, reranking or query rewriting are mistakes. I am saying the contract for adding them must include the column that can go down.
What an abstention-aware eval contract looks like (menu, not recipe)
| Field | Question it answers | Failure if skipped |
|---|---|---|
| Answerable accuracy | Did we answer the questions the corpus supports? | Shipping on vibes |
| Correct abstention rate | Did we decline the questions nothing supports? | Confident answers with no evidence |
| Premise denial, counted separately | Did we claim something does not exist without being able to verify it? | "No such thing" passing as caution |
| Attempt rate on answerable items | Did we refuse questions we could have answered? | A system that looks safe because it answers nothing |
| Per-change deltas | Which single change moved each column? | Crediting the whole pipeline for one component's regression |
None of that needs a private harness. It needs someone to refuse a pipeline change that only reports the accuracy column.
Why "I don't know" is not the same as "no such thing"
The distinction I would steal first is premise denial.
When a corpus is older than the question, "I can't find this" and "this doesn't exist" are different claims. The first is honest. The second is a statement about the world that the system cannot back. A metric that lumps both into "declined" will reward the second. The paper's own example is a model that, asked about an item added after the cut-off, says the item is not recognized and then answers about a different item instead.
If you run retrieval over documentation that changes, such as product docs, internal wikis or release notes, this is the failure to watch. The question can be newer than your index.
Where this connects to habits I already keep public
My public work is mostly retrieve-first systems, and the same habit applies there: measure the retrieval half on its own before letting the answer vouch for it. My hybrid retrieval work is built around that habit. This week's paper adds a column I would put next to it: what the system does when the right evidence is not in the index at all.
Steal this check
Pick one RAG pipeline you are about to "improve" and add four things before you merge:
- Write a small unanswerable slice. Questions about things newer than, or missing from, your corpus. Even a few dozen is better than none.
- Grade abstention next to accuracy. Report both columns for the before and after versions of the pipeline.
- Split premise denial from abstention. "I can't find it" and "it doesn't exist" go in different buckets.
- Change one thing at a time. Extra chunks, reranking, rewriting and extra rounds each get their own before/after row.
Then ask whether the change is still an upgrade once the abstention column is visible.
What I am not claiming
- I am not reporting DeclineCraft results as my own, and I have not reproduced the benchmark.
- I am not saying iterative RAG, reranking or query rewriting should be removed. The paper's own caveat is that it cannot isolate which component drives the drop.
- I am not publishing thresholds. What counts as an acceptable abstention rate depends on the product and on what a wrong answer costs.
- One domain (a game wiki) is not every domain. The authors say so too.
The point
An accuracy chart on answerable questions cannot tell you whether your upgrade made the system more willing to bluff. Put unanswerable questions in the eval, grade "I don't know" as a real outcome, and make every pipeline change earn its place on both columns.
Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.
Top comments (1)
The detail that accuracy stays flat while abstention drops is what makes this risky, because every chart a team normally looks at says the change was fine. One practical way to build the unanswerable set without inventing questions: take real user questions and remove their supporting documents from the index for that eval run, so the answer genuinely isn't there. The phrasing stays realistic, and it also shows whether the extra retrieval rounds are pulling in near-miss chunks that make the model feel confident enough to answer anyway.