I asked my local RAG stack about my own files and it hallucinated. Confidently. The corpus was mine, the retrieval was correct, the small model just... invented an answer around thin context, in fluent English.
The fix wasn't a bigger model. It was a behavioral contract, enforced in the prompt and proven in testing:
text
You are a grounded assistant. Use ONLY the context below. Rules:
(1) If the context does not contain the answer, reply exactly: NOT IN CORPUS.
(2) Cite which source title each fact came from.
(3) Do not use outside knowledge.
The result: the same query that hallucinated now replies "NOT IN CORPUS." — and when the context is there, the answer arrives with source citations attached.
Two things I learned building this:
The retrieval was never the problem. Vector search found the right neighborhood every time. The mouth was the problem — a small model treating thin context as an invitation to improvise. People blame RAG for hallucinations that are actually generation-layer failures.
Abstention is a feature you have to test. I now run queries I know are absent from the corpus, specifically to check the model refuses. An abstention you haven't tested is an abstention you don't have.
The stack: Ollama + nomic-embed-text + ChromaDB + granite 3B, fully local on an 8GB MacBook Air. 300K+ chunks of my own archive, no cloud, no API, no accounts. The orchestrator dispatches each question to four workers and the EvidenceFlow guard verifies before anything reaches the screen.
My full archive answers questions in my own voice now — and it has permission to say "I don't know." That permission is the feature.
The tooling is published: ask-mirror — hybrid retrieval (vector + SQL) with the abstention guard built in, AGPL. The verification discipline comes from EvidenceFlow — fail-closed on Ship 5, now guarding the whole stack.
Top comments (0)