Getting an AI app to answer a question is exciting. Getting it to recognize when the documents don’t contain the answer is a more interesting challenge.
For Day 1 of my AI challenge, I’m starting with a small problem:
How can we make a RAG application avoid answering questions its sources cannot support?
The problem
Imagine uploading a company’s leave policy and asking:
“How many days of annual leave do employees get?”
If the policy contains that information, the app should answer and show the supporting passage.
Now ask:
“Does the company pay for international travel?”
If the document says nothing about travel, the app should explain that it couldn’t find the answer in the provided material.
A confident guess could easily be mistaken for company policy.
Why RAG alone doesn’t solve this
Retrieval-Augmented Generation retrieves relevant content and passes it to a language model to help answer a question.
But the closest matching passage isn’t always a passage that answers the question.
A search system can return something even when the information we need is missing. That makes retrieval a useful starting point, but it doesn’t guarantee a supported answer.
My Day 1 approach
I’m keeping the first version small:
- Load a short document and split it into passages.
- Retrieve passages for a question.
- Ask the model to answer using only those passages.
- Require a source reference for the answer.
- Return a clear fallback when the passages don’t provide enough evidence.
The fallback can be simple:
“I couldn’t find that information in the provided documents.”
The important part is making that response a valid outcome.
What I want to test
I’ll use three types of questions:
- Answerable: The document directly contains the answer.
- Related but unsupported: The question shares the document’s topic, but the answer is missing.
- Unrelated: The question is outside the document’s scope.
That middle group is especially interesting. A passage can look relevant while still failing to support an answer.
For each response, I’ll check:
- Is the answer correct?
- Does the cited passage actually support it?
- Did the app decline when the evidence was missing?
- Did it unnecessarily decline a question it could answer?
This should help expose both unsupported answers and overly cautious responses.
One thing I’m being careful about
A valid citation ID doesn’t prove that an answer is correct. It only proves that the referenced passage exists.
I still need to check whether that passage supports the claim.
Similarly, I don’t want to choose a retrieval-score threshold and assume it works everywhere. Any cutoff needs testing with the actual documents and retrieval method.
The goal for today
Build a small baseline, inspect where it fails, and record the results before adding more complexity.
I’m starting here because knowing when to stop answering feels like a useful foundation for a document-based AI application.
If you’re building a RAG app too, try asking a question that sounds relevant but whose answer is deliberately missing from the documents.
Does your app notice?
Discussion
What’s one question you would use to test whether a RAG application is guessing?
Top comments (2)
Want to explore this topic a little deeper? Here are three research papers worth reading:
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. (2020)
A foundational paper explaining how retrieval and language generation can work together to answer knowledge-intensive questions.
arxiv.org/abs/2005.11401
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Asai et al. (2023; ICLR 2024)
Explores how a model can retrieve information when needed and assess retrieved passages and its own answers. Useful for understanding why retrieving a passage doesn’t automatically make an answer trustworthy.
arxiv.org/abs/2310.11511
Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse — Song et al. (ICLR 2025)
The closest match to this challenge: it studies appropriate refusal and citation quality, introducing Trust-Score and Trust-Align to evaluate and improve these behaviors.
arxiv.org/abs/2409.11242
tr.ee/dev-to