The failure mode of a naive RAG pipeline is a confident-sounding hallucination: it retrieves some passages, ignores how weak they are, and answers anyway. Project 2 of my Agentic AI from Zero build is a RAG agent designed to never do that — every answer is backed by retrieved sources with honest confidence, hand-rolled, no framework. I ran it for real against NVIDIA NIM (meta/llama-3.1-8b-instruct, temperature 0), and the recorded run shows all three behaviours on real model output.
Retrieve locally, ground deterministically
Retrieval is a local TF-IDF + cosine step over 25 passages — six short markdown docs about a fictional product, "Nimbus Cloud" (pricing, SLA, regions, security, retention, limits). Fictional on purpose: the model can't answer from parametric memory, so a correct answer must come from retrieval, which is exactly what makes the citations load-bearing. Because retrieval is offline and deterministic, the recorded run reproduces byte-for-byte and only the generation step hits the network — three live LLM calls, each returning HTTP 200.
Confidence from two signals
The decision combines the top-1 cosine similarity with the model's own self-reported grounding:
top_score < 0.15 -> FALLBACK (retrieval too weak)
0.15 <= top_score < 0.30 -> LOW_CONFIDENCE (thin support)
top_score >= 0.30 AND model says grounded -> GROUNDED (answer + citations)
model reports it cannot support the answer -> LOW_CONFIDENCE (flag, never bluff)
Case 1 — grounded, with inline citations
"What uptime SLA does the Nimbus Enterprise plan guarantee and what is its monthly price per seat?" The top passage scored 0.6180 and the model reported supported=true, so the agent grounded and cited inline:
The Nimbus Enterprise plan guarantees 99.95 percent uptime [1] and costs 99 dollars per seat each month [2].
Each marker maps back to its source: [1] to sla.md (score 0.618), [2] to pricing.md (0.386). Both cited facts trace exactly to the cited passages — the citations aren't decoration, they're the evidence.
Case 2 — weakly supported, flagged not fabricated
"Can I pay for my Nimbus subscription using cryptocurrency such as Bitcoin or Ethereum?" The top hit scored just 0.2185 — a lexical near-miss, since nothing in the corpus mentions payment methods — and the model correctly reported supported=false. Sub-threshold similarity and the model's own honesty combine, and the agent flags instead of inventing a policy:
There is not enough information to answer this question.
That's the whole point: a plausible-but-unsupported question gets flagged, not answered with a made-up crypto-payment rule.
Case 3 — outside the corpus, fallback to search
"What are the common symptoms of a vitamin D deficiency in adults?" Every retrieved passage scored 0.0000, below the 0.15 floor, so the agent short-circuits before the grounded call and routes to a distinct, loudly-labeled web-search fallback:
FALLBACK: top retrieval similarity 0.0000 < 0.15 -> corpus has no relevant passage
The answer carries a mandatory disclaimer banner — [FALLBACK - general web knowledge, NOT from the Nimbus knowledge base; verify independently.] — and, crucially, the sources list is empty, because nothing in the trusted corpus supports it. No fake citations bolted onto a general-knowledge answer.
One detail keeps the two halves honest and swappable: retrieval is a deterministic, offline TF-IDF + cosine step, so the recorded run reproduces byte-for-byte and needs no network for the retrieval half — but retriever.embed() is written so you could drop in a NIM embeddings.create call without touching the agent. The thresholds (0.15 fallback, 0.30 confident) are tuned against this specific corpus, and the measured scores are all in the recorded run.
Those are the four ideas in one agent: retrieve context, generate with sources, flag low confidence, and fall back to search — the difference between a RAG demo and a RAG agent you'd trust. The honest move is knowing which of the three paths a query is on before you answer, and letting the retrieval score and the model's own admission decide, rather than papering over a blank corpus with fluent prose.
Top comments (0)