How Unanswerable Questions Can Inflate RAG Recall@K

Retrieval evaluation is one of the first things we measure when building a Retrieval-Augmented Generation (RAG) pipeline.

Metrics like Recall@K help answer a straightforward question:

Did the retriever return the documents needed to answer the question?

But there's an edge case that can make your evaluation results misleading: questions for which no relevant document exists.

The problem

Suppose you're evaluating a RAG pipeline using a golden dataset containing eight questions.

Seven questions are answerable from the document corpus. One is intentionally unanswerable.

For the unanswerable question, the golden dataset correctly contains no expected source documents.

A naive implementation of Recall@K might look like this:

def recall_at_k(retrieved_ids, expected_ids):
    if not expected_ids:
        return 1.0

    return len(
        set(retrieved_ids) & set(expected_ids)
    ) / len(set(expected_ids))

The problem is this condition:

if not expected_ids:
    return 1.0

It awards perfect retrieval recall when there are no relevant documents to retrieve.

That doesn't demonstrate successful retrieval. It means the retrieval metric is not applicable to that question.

How this changes the evaluation score

Consider the following example:

Question	Answerable?	Recall@K
Q1	Yes	1.0
Q2	Yes	1.0
Q3	Yes	0.5
Q4	Yes	1.0
Q5	Yes	0.0
Q6	Yes	0.5
Q7	Yes	1.0
Q8	No	N/A

If we incorrectly award the unanswerable question a recall score of 1.0:

Naive mean Recall@K = 0.750

If we exclude that question from the retrieval recall calculation:

Corrected mean Recall@K ≈ 0.714

The retriever hasn't changed.

The retrieved documents haven't changed.

Only the evaluation logic changed.

A small difference in this example, but one that can become more significant when a dataset contains a larger proportion of unanswerable questions.

A better approach

Instead of treating missing expected documents as perfect retrieval, return a non-applicable result.

def recall_at_k(retrieved_ids, expected_ids):
    if not expected_ids:
        return None

    expected = set(expected_ids)
    retrieved = set(retrieved_ids)

    return len(retrieved & expected) / len(expected)

When aggregating scores, exclude None values from the mean and report how many questions were eligible.

This is important because an empty expected-document set can represent different situations.

For a deliberately unanswerable question, retrieval recall is not applicable.

For a question whose relevance annotations are incomplete or missing, the evaluation dataset itself may need correction.

Ideally, your golden dataset should explicitly distinguish these cases rather than infer answerability solely from an empty source list.

But unanswerable questions still matter

Excluding unanswerable questions from retrieval recall doesn't mean excluding them from evaluation altogether.

They test a different capability:

Can the RAG system recognize insufficient evidence and avoid generating an unsupported answer?

That deserves a separate evaluation of abstention behavior.

For example:

• Did the system correctly indicate that the available documents don't support an answer?
• Did it generate an answer despite insufficient evidence?
• Did it incorrectly refuse an answerable question?

Retrieval recall and abstention correctness answer different questions. Combining them into a single retrieval score can hide important failure modes.

Why this matters in production

RAG evaluation is often used to compare chunking strategies, embedding models, retrievers and rerankers.

But before interpreting a score improvement, we need to trust the measurement itself.

A change in dataset composition, answerability labels or metric aggregation can make a system look better without any actual improvement in retrieval.

That's why I prefer reporting both the mean retrieval score and the number of eligible questions, while evaluating unanswerable cases separately.

Open-source implementation

We encountered this issue while working on an open-source RAG evaluation starter and updated its handling of unanswerable golden-set questions.

The project includes retrieval evaluation, answer-quality metrics, and optional LLM-based judging. Core metrics run locally without an API key.

GitHub: https://github.com/infrixo-systems/rag-evaluation-starter

I'm interested in how others handle this in their evaluation pipelines.

Do you explicitly label unanswerable questions in your golden datasets, and how do you evaluate abstention separately from retrieval quality?