0 % vs 50 %: making a RAG agent refuse to hallucinate

2026-05-31 · LLM / RAG

A retrieval-augmented agent is only as trustworthy as its behaviour on questions whose answer
isn't in the corpus. The failure mode is quiet: instead of saying "I don't know," the model
invents a confident, well-formed, wrong answer. This post shows a single guardrail that takes
that from common to never — and, crucially, measures it.

Reference architecture:
nim-agent-blueprint — agentic RAG on
the NVIDIA NIM stack with a built-in eval harness.

The ablation

The agent loop is plan → retrieve → generate → validate. The interesting variable is the
generation prompt's contract with the retrieved context:

Configuration	Out-of-corpus hallucination rate
Generate freely from context	~50 %
Guarded prompt (answer only from context; otherwise abstain)	0 %

Same model, same retriever, same questions. The only change is a prompt that makes "I can't
answer that from the provided sources" a first-class, rewarded output — plus a validate
step that checks the answer is grounded in retrieved spans before returning it. On in-corpus
questions, retrieval recall@3 stayed at 94–100 %, so the guardrail buys safety without
costing coverage.

Why "just prompt better" isn't the lesson

The lesson isn't the prompt — it's that the difference between 50 % and 0 % is invisible
without an eval harness. A demo that only asks in-corpus questions looks perfect in both
configurations. You only see the 50 % when you deliberately ask things the corpus can't
answer and score groundedness. So the blueprint ships with:

• retrieval hit-rate (is the answer even retrievable?),
• answer groundedness via LLM-as-judge (is the answer supported by what was retrieved?),
• latency, and OpenTelemetry traces per agent step.

That's the difference between "it works on my five questions" and "here is the number a
partner can hold me to."

Takeaway

For enterprise RAG, abstention is a feature, not a failure. Make "I don't know" a rewarded
output, validate groundedness before returning, and measure the out-of-corpus rate — it's
the number that separates a demo from something you'd put in front of a customer.

→ Runnable blueprint + eval harness:
github.com/waynehacking8/nim-agent-blueprint