A retrieval system can return a confident answer to a question its documents never address. It may have retrieved a vaguely related paragraph, or the model may fill a gap from general knowledge. Showing citations helps, but a citation is not proof: the source might not support the sentence beside it.
The fix is a workflow that makes evidence explicit. Give every indexed passage a stable source ID, require the answer to point to supporting passages, and permit a clear “not enough evidence” result. Then test that behavior with questions the system should not be able to answer.
Keep provenance with every passage
At minimum, store each passage with a stable identifier, source path, title, and the source’s content hash. A record might look like this:
{
"id": "handbook-2026-04:access#p3",
"path": "handbook/access.md",
"title": "Access reviews",
"content_sha256": "<hash of the indexed source>",
"text": "Temporary access expires after seven days unless renewed."
}
Replace the hash placeholder with the source digest. A citation should identify the version and passage that was indexed, so an edit to the handbook does not erase the evidence behind an earlier answer. Keep that source text available for citation rendering.
Ollama’s embedding endpoint accepts one string or a batch and returns vectors for semantic retrieval. Similarity helps rank candidate passages, but a high-ranking passage may still lack the requested fact. See the Ollama embeddings guide.
Write down an abstention contract
Before calling the answer model, create a prompt that makes the evidence boundary concrete:
Answer using only the passages below.
For each factual claim, include one or more passage IDs that directly support it.
If the passages do not contain enough information, set answer to
INSUFFICIENT_EVIDENCE and evidence to [].
Do not use outside knowledge to fill gaps.
Question: When does temporary access expire?
[handbook-2026-04:access#p3]
Temporary access expires after seven days unless renewed.
Answer format:
answer: <one sentence or INSUFFICIENT_EVIDENCE>
evidence: [<passage ID>, ...]
For the included passage, a supported answer is “After seven days unless renewed” with the listed ID. If asked “Who approved the policy?” the correct output is answer: INSUFFICIENT_EVIDENCE and evidence: []: the passage does not say. Flag malformed output, unknown IDs, and citations that fail to support their claims for review.
Prompt wording alone cannot force every model to abstain consistently. Add a retrieval gate calibrated on your own data. For example, inspect whether the top candidates are relevant and whether at least one passage contains the kind of evidence needed. Do not copy a similarity threshold from another project: embedding models, corpus, and distance calculation affect score distributions. Store the chosen policy and test it when any of those change.
Research such as Self-RAG explores retrieval relevance and reflection as ways to improve factuality. The workflow here uses a simpler application-side contract: retain source passages, make evidence IDs explicit, and test the abstention path.
Hold out questions before tuning
Create an evaluation set with three categories: answerable questions, unanswerable questions about missing facts, and misleading questions with a related but insufficient passage. Keep some questions out of prompt and threshold tuning. For each, record expected answerability, acceptable source IDs, and the unsupported fact to watch for.
An illustrative test set might include:
| Question | Expected | What it catches |
|---|---|---|
| When does temporary access expire? | Answer with handbook-2026-04:access#p3
|
Supported fact |
| Who approved the policy? | Abstain | Missing fact |
| Does renewal extend access forever? | Abstain | Over-reading a related sentence |
Run these cases through the application's retrieval and answer path. Check both whether it abstains on missing facts and whether it answers the question the passage does support. Record unnecessary abstentions alongside unsupported answers.
Save retrieved IDs, scores, prompt version, model identifier, and final answer for each run. If “Who approved the policy?” starts producing a name after an update, those records help locate the change in indexing, retrieval, or generation.
Top comments (0)