Day 01/90: Teaching a RAG App When to Say “I Don’t Know”

Rimsha Kanwal — Fri, 16 Oct 2026 22:55:54 +0000

Getting an AI app to answer a question is exciting. Getting it to recognize when the documents don’t contain the answer is a more interesting challenge.

For Day 1 of my AI challenge, I’m starting with a small problem:

How can we make a RAG application avoid answering questions its sources cannot support?

The problem

Imagine uploading a company’s leave policy and asking:

“How many days of annual leave do employees get?”

If the policy contains that information, the app should answer and show the supporting passage.

Now ask:

“Does the company pay for international travel?”

If the document says nothing about travel, the app should explain that it couldn’t find the answer in the provided material.

A confident guess could easily be mistaken for company policy.

Why RAG alone doesn’t solve this

Retrieval-Augmented Generation retrieves relevant content and passes it to a language model to help answer a question.

But the closest matching passage isn’t always a passage that answers the question.

A search system can return something even when the information we need is missing. That makes retrieval a useful starting point, but it doesn’t guarantee a supported answer.

My Day 1 approach

I’m keeping the first version small:

Load a short document and split it into passages.
Retrieve passages for a question.
Ask the model to answer using only those passages.
Require a source reference for the answer.
Return a clear fallback when the passages don’t provide enough evidence.

The fallback can be simple:

“I couldn’t find that information in the provided documents.”

The important part is making that response a valid outcome.

What I want to test

I’ll use three types of questions:

Answerable: The document directly contains the answer.
Related but unsupported: The question shares the document’s topic, but the answer is missing.
Unrelated: The question is outside the document’s scope.

That middle group is especially interesting. A passage can look relevant while still failing to support an answer.

For each response, I’ll check:

Is the answer correct?
Does the cited passage actually support it?
Did the app decline when the evidence was missing?
Did it unnecessarily decline a question it could answer?

This should help expose both unsupported answers and overly cautious responses.

One thing I’m being careful about

A valid citation ID doesn’t prove that an answer is correct. It only proves that the referenced passage exists.

I still need to check whether that passage supports the claim.

Similarly, I don’t want to choose a retrieval-score threshold and assume it works everywhere. Any cutoff needs testing with the actual documents and retrieval method.

The goal for today

Build a small baseline, inspect where it fails, and record the results before adding more complexity.

I’m starting here because knowing when to stop answering feels like a useful foundation for a document-based AI application.

If you’re building a RAG app too, try asking a question that sounds relevant but whose answer is deliberately missing from the documents.

Does your app notice?

Discussion

What’s one question you would use to test whether a RAG application is guessing?

DEV Community: Rimsha Kanwal