The first time my RAG system gave a confidently wrong answer, I did what everyone does: I blamed the model. I swapped in a bigger one. I tuned the ...
For further actions, you may consider blocking this person and/or reporting abuse
James, excellent checklist. The discussion in the comments — specifically from @mnemehq, @suraj09, and the memory-layer builder above — is circling around a single missing architectural boundary: the gap between mechanical pipeline success and semantic truth.
Your 9 steps guarantee mechanical execution. But this thread just named three distinct ways a pipeline can be mechanically perfect, pass every green dashboard check, and still be dead wrong:
exit code 0. The system records "success," fallback kicks in, and the dashboard stays green while serving pure noise.Evidence ExistsvsEvidence Entails): As @suraj09 pointed out, retrieving a valid chunk is not enough. The model often over-interprets "X is common in Y" into "X is required for Y." The citation check passes because the passage exists, but the semantic claim is false.These aren't separate edge cases — they are symptoms of a missing Semantic & Policy Verification Layer. The retrieval pipeline proves the bytes were found; it doesn't prove the claim is true, current, or permitted.
In our work with codebase intelligence (MSCodeBase Intelligence, indexing ~50K LOC), we hit this exact wall. A chunk could match with 0.95 cosine similarity, but if the underlying function was refactored yesterday, the chunk was semantically dead. Better reranking couldn't fix it — only a live verification step that validates claims against current system state before execution.
A cross-cutting question for the checklist: "Does the pipeline distinguish between 'I retrieved a matching context' and 'the claim generated from this context is currently true and actionable'?"
Without that boundary, a perfectly-tuned retrieval pipeline just produces cryptographically perfect hallucinations.
This is a great extension of the verification boundary. “Evidence exists” vs “evidence entails” is especially important — retrieval can be correct while the conclusion is wrong. And as you point out, true + relevant still doesn’t necessarily mean permitted to act.
"Cryptographically perfect hallucinations" is the phrase this whole thread was reaching for — thank you for naming it. You're right that my nine steps prove the bytes were found, and every failure surfaced here (silent absence, evidence-exists-vs-entails, the enforcement gap) is a case where the bytes are perfect and the claim is still false, stale, or unpermitted. The unifying insight is that better retrieval can't fix any of them, because they live on a different axis entirely — and worse, high retrieval accuracy launders all three by wrapping the wrong outcome in real evidence. Your MSCodeBase example is the cleanest possible proof: 0.95 cosine similarity on a function refactored yesterday isn't a retrieval miss, it's a semantically dead hit that no reranker can catch, only a live check against current system state. So your cross-cutting question is the right one to pin above the whole checklist: does the pipeline distinguish "I retrieved matching context" from "the claim from this context is currently true and actionable"? Retrieval decides what's found; a semantic-and-policy verification layer decides what's true, current, and permitted — and only the second one keeps you from shipping evidence-backed nonsense. Adding this to the follow-up with credit to you and the thread.
Thanks for the great response, James! Glad to see the distinction resonating so strongly with the community.
James, the checklist is the first one I have read that walks the whole path rather than the retriever alone, and innokentyb asked the question underneath it: whether stale records get fixed with metadata filters, reranking, or a separate policy layer. The honest answer is none of the three. Filters and reranking work after the fact, at query time, comparing dates or scores on records that already sit side by side in the index. Supersession has to be marked at the moment a new version is written, not inferred later during ranking. Five people commenting on posts of mine landed on that distinction independently, which is a stronger signal than one person making the case. One of them added a condition none of the others had: the writer must either link the new record to what it replaces or record explicitly that no predecessor was found, and that lookup is not allowed to fail silently. A silent miss there is worse than not tracking supersession at all, because it looks handled and is not.
On the context assembly item, the symptoms are named well: order sensitivity, how much fits, information lost in the middle. What is missing is the mechanism. Truncation keeps the record and drops the link between a chunk and its status, whether it is current or superseded, and nothing downstream is told that happened. Merging is fine, full replacement is fine, dropping a chunk entirely is fine. Clipping by window size is not, because that is the one operation that severs the status link without a trace.
What I do not have is a clean way to test for that loss from outside a system, short of forcing a supersession on purpose and watching whether the old chunk survives truncation.
This closes the loop innokentyb opened, and the answer — none of the three — is sharper than anything I offered in that subthread. You've named why: filters and reranking are both query-time operations comparing records that already sit side by side in the index, which means they can only act on supersession after it's been recorded. If the record was never marked stale at write time, there's nothing for a filter to filter on or a ranker to down-weight. Supersession is a write-time fact, not a query-time inference — and trying to reconstruct it during ranking is asking the retriever to guess a truth the indexer forgot to store. Five people landing on that independently is exactly the kind of signal I trust more than one confident argument.
And the added condition is the part I'd have missed: the writer must either link the new record to what it replaces or explicitly record that no predecessor was found — and that lookup can't fail silently. That's the same disease this whole comment section keeps diagnosing in different costumes: a silent miss that looks handled is worse than no handling at all, because the first one you'll trust and the second one you'll at least know to check. A supersession link that quietly fails to resolve gives you a stale record wearing a "current" badge, which is the most dangerous state in the entire pipeline.
The context-assembly point is the one I'm genuinely taking away, because you found a mechanism I stated only as a symptom. "Lost in the middle" I framed as the model ignoring buried content — but you're pointing at something worse happening before the model even reads it: truncation keeps the chunk's text and severs its status link, with nothing downstream told it happened. Merge, replace, drop-entirely — all fine, because they either preserve the association or remove the whole thing cleanly. Clipping by window size is the one operation that keeps the bytes and silently strips whether those bytes are current or retired. So a superseded chunk can survive assembly stripped of the very metadata that would have flagged it as superseded, and now it's indistinguishable from a live one. That's a real bug and I don't think I've seen it named before.
On testing it from outside — I don't have a clean answer either, and I suspect there isn't one that's purely observational, because the whole failure is the absence of a signal. The forced-supersession probe you describe is probably the honest floor: plant a known-superseded chunk, run a query that pulls it into a context that will truncate, and assert the retired chunk either carries its status through or gets dropped — never survives stripped. Which is really the red-green-red discipline from another thread applied to metadata survival: prove the status link can be observed to break, on purpose, or you have no evidence it holds. Adding all of this to the follow-up with credit — the write-time-vs-query-time framing and the truncation-severs-status mechanism are two of the most useful things anyone's contributed to this piece.
The probe inherits the failure it is built to catch. It names two conditions, that the query pulls the plant in and that assembly truncates, but names them as intentions rather than as checks, and both fail quietly. A run where the plant was never retrieved satisfies never survives stripped. So does a run where the context stayed under budget and nothing was clipped. Both come back green. Those two have to be asserted separately and loudly, or green only means nothing happened, which is the state you named as the dangerous one.
Second, the verdict turns on where the boundary landed rather than on any rule the system holds. If the status sits at the head of the chunk and the cut takes the tail, it passes by accident of byte position, and one green run is a sample of one offset. The cut has to be forced inside the plant at more than one position, and run once with the link severed on purpose, so the assertion is shown able to go red. And if the grading reads the model output rather than the assembled context, a clean drop and a carried status can look identical, which makes the pass unreadable either way.
The limitation on my own half is sharper than both. A planted chunk is the best case a corpus can offer: written through the current path, predecessor present, link fresh. The records that hurt are older than the status field, or carry the explicit no-predecessor-found marker from a lookup that came back empty. A passing probe licenses a claim about writes made today and says nothing about what is already sitting in the index. That gap does not close with this probe.
You've done to my probe what the whole thread keeps doing to everything else — turned its two intentions into two silent failures, and you're right on both counts. "Force a supersession, watch whether the old chunk survives truncation" names what should happen and quietly assumes the setup occurred. A run where the plant was never retrieved passes "never survives stripped" vacuously. So does a run that stayed under budget and clipped nothing. Both come back green, and green here means nothing happened, which is precisely the state I called the dangerous one — my own test-of-a-test fell into the exact hole the test was built to guard. So the two preconditions have to be assertions, loud and separate: assert the plant was retrieved, assert truncation actually fired inside it — a probe that can pass without both is measuring the absence of an event, not the survival of a link.
The byte-position point is the sharper one, because it means even a correctly-triggered green can be an accident. If the status sits at the head of the chunk and the cut takes the tail, the link survives by luck of offset, and one green run is a sample of one cut point. That's not evidence the system holds the status through truncation — it's evidence it held it this once, at this boundary. The fix is the ablation discipline from upthread pointed here: force the cut inside the plant at multiple positions, and run it once with the link severed on purpose so the assertion is shown able to go red. A survival check that has never been observed to fail is a green that can't fail, one floor down. And your grading-surface catch closes the last hole: if the probe reads the model output instead of the assembled context, a clean drop and a carried status can render identical, so the pass is unreadable regardless of which happened. Grade the context, not the generation.
But the limitation you put on your own half is the part I'll actually keep, because it's the most honest boundary anyone's drawn in this thread. A planted chunk is the best case the corpus can offer — written through the current path, predecessor present, link fresh. The records that hurt are the ones older than the status field itself, or the ones carrying an explicit no-predecessor-found marker from a lookup that came back empty. A passing probe licenses a claim about writes made today and says exactly nothing about the legacy already sitting in the index. That gap doesn't close with any probe of this shape, because the probe can only exercise the write path it has, and the dangerous records predate that path. Which means supersession integrity isn't one problem — it's "does the write path preserve status going forward" (probeable, with your fixes) and "what's the status of everything written before the path existed" (a backfill/audit problem the probe can't touch). Naming that second one as unclosed, instead of letting a green forward-probe imply the whole index is sound, is the difference between a check and a reassurance. Going into the revision with all of it credited — the probe-inherits-its-own-failure framing is the spine of the correction.
The split is right, and the second half is harder than a backfill. An audit can decide a fact about a record: whether it carries a status field at all. It cannot decide whether the record is current, because that requires knowing what superseded it, and for anything written before the status field existed the link was never recorded. Reconstructing it means inferring supersession at audit time from similarity or recency, which is the exact move the write-time argument says is unsound. So the legacy set is not merely expensive, it is partly undecidable by the same reasoning that made the forward case work.
What follows is a third class rather than a cleaner split. Not current and not retired, but unknown by construction, and it has to be a state the retriever can see, because both defaults are failures already named in this thread. Default unknown to current and you get the stale record wearing a current badge. Default it to retired and you silently drop facts nothing ever contradicted, which is the same silent miss pointed the other way.
The one thing that shrinks the class honestly is provenance plus time. A record written after the path existed is forward-probeable and can be proven so, which closes the unknown set at one end and makes it monotonically smaller. That is a slower claim than an audit result, and it is the one that survives being checked, because it never asserts anything about a record whose history was not written down.
One thing I’d add: retrieval success ≠ semantic correctness. A relevant chunk can still support the wrong conclusion. I’d test whether the retrieved evidence actually entails the claim, not just whether it matches the query.
Great checklist, I just published an article evaluating Self-Rag, query decomposition and HyDE and how they compare on several chunk sizes and affect retrieval quality.
If you read this blog post this will be a good follow up for a deep dive.
dev.to/shadrach1999/building-a-pro...
That sounds like exactly the deep dive this post gestures at but doesn't measure — Self-RAG, query decomposition, and HyDE compared across chunk sizes is the kind of head-to-head data I'd love to point people to instead of hand-waving "it depends." The chunk-size interaction especially interests me, since so much advice treats these techniques as independent of chunking when they clearly aren't. I'll definitely give it a read — thanks for flagging it, and for the kind words.
You could also reverse the hyde during storing embedding vectors, where you can pre-generate questions for sections, that way search becomes a bit faster. Indexing can also be progressive, so that time to search decreases, especially if you have people uploading batches of documents
Reversing HyDE at index time is clever — pre-generating likely questions per section means you're matching question-to-question instead of question-to-answer, which sidesteps the query/document shape mismatch entirely. It front-loads the LLM cost to indexing (where latency doesn't matter) instead of query time (where it does), so search gets both faster and better-aligned. And progressive indexing is the right call for batch uploads — letting documents become searchable incrementally beats blocking on a full re-index. Both are going in the follow-up; great additions.
Solid point: RAG without a checklist will confidently mislead. The fix is tighter retrieval, proven provenance, and safe fallbacks. Add adversarial prompts, post-answer citation audits, and a live user-feedback loop. Bonus: a freshness timer to flag stale sources before they masquerade as truth.
"Confidently mislead" is the whole danger in two words — a RAG system fails louder than a search box because the fluent answer hides the broken retrieval underneath. I like that your additions are all adversarial or continuous rather than one-time: post-answer citation audits and a live feedback loop assume things will drift and catch it, instead of trusting a green check from launch day. And the freshness timer is the neat fix for the "right fragment, superseded source" problem someone else raised — stale-but-relevant is exactly the failure that masquerades as truth. Great list.
Good checklist. I'd add one more line: retrieval quality and enforcement are separate failure surfaces. You can retrieve the exact right chunk of context and still let the agent act against it in a way nobody sanctioned. Fixing recall doesn't fix that.
This is a sharp addition, and it names a boundary the checklist blurs: retrieval and enforcement are separate failure surfaces, and every item in the post lives on the first one. Recall, ranking, faithfulness — all of it answers "did the right context reach the model?" None of it touches "was the model allowed to act on it the way it did?" You can nail the entire retrieval chain and still have an agent do something nobody sanctioned with a perfectly-retrieved fact.
The reason this hides so well is that a correct retrieval makes the action look justified. The chunk was right, the citation is real, the reasoning reads clean — so the unsanctioned action inherits the credibility of the good context it was built on. Fixing recall actually makes this failure more convincing, not less, because now the wrong action comes wrapped in solid evidence. Better retrieval can launder a worse enforcement gap.
It maps onto a distinction that came up in another thread here: "the model saw the right thing" and "the model was permitted to do this thing" are different claims, and only one of them is what retrieval evals measure. Enforcement is a policy question — what actions are allowed against what context, under whose authorization — and policy questions want deterministic, auditable gates, not a similarity score or a faithfulness rating. Retrieval quality is necessary and does nothing for it.
So the line I'd add to the checklist, with credit: retrieval decides what the agent knows; enforcement decides what it's allowed to do with that — and passing every retrieval check tells you nothing about the second. Which, for anything where the agent acts rather than just answers, is often the surface that actually hurts you. Great catch.
HyDE on a private corpus pulls neighbors of a parametric guess, so dense retrieval hunts documents shaped like public answers. Query expansion does the same thing when it fills in product names your index does not use, and BM25 then searches tokens that never appear. An internal runbook that answers the original question never gets retrieved. Transform the query is fine. There is still gonna be a cited answer sitting on public-shaped chunks while the internal function names never leave the index.
This is the sharpest failure here, because it attacks the fix, not the naive baseline. HyDE assumes the docs are shaped like the model's prior — on a private corpus they're not. The hypothetical is a public-text guess, so you retrieve neighbors of the public answer and sail past the internal runbook with your actual function names.
And expansion fails the same way: it invents public vocabulary, then BM25 searches tokens that appear nowhere in your index. So both channels drift public at once — dense chases public-shaped embeddings, sparse chases public-shaped tokens — and the right chunk loses on both. Nothing errors; you get a confident cited answer on the wrong chunks, which is worse than a miss because it looks retrieved.
The lesson: query transformation is only safe when it's grounded in the corpus's own vocabulary, not the model's prior. That points at corpus-mined expansion or pseudo-relevance feedback (let real retrieved chunks seed the rewrite, not a hallucinated one) — plus a lexical path that privileges exact in-corpus tokens. "Transform the query" earns the same caveat as everything else in the post: it's a liability the moment it's applied on faith without checking whether it moved you toward your corpus or toward the public web. Going in the follow-up — great catch.
Great work! Really solid checklist - I wish I'd had this before my first RAG rodeo.
One thing I didn't see mentioned, though: caching. In production, 60–80% of queries are repeats or near-duplicates. If you've already answered it once, just serve the cached response - it's cheaper, faster, and cuts LLM load dramatically. For high-traffic systems, it's a no-brainer.
Other than that, this is gold. Thanks for sharing!
Thank you — and yes, caching is a real omission, especially since you're right that in production the query distribution is brutally repetitive; 60–80% repeats or near-dupes is exactly the kind of traffic where re-running the whole retrieve-rank-generate chain every time is pure waste. The one nuance I'd add is that "near-duplicate" is where it gets interesting: exact-match caching is the easy no-brainer you described, but semantic caching (serving a cached answer for a paraphrase of a prior question) is the bigger win and the sharper knife — set the similarity threshold too loose and you'll confidently serve the answer to a slightly different question, which lands you right back in the "confident, wrong, looks retrieved" failure mode the rest of the post is about. So I'd frame it as: exact-match cache freely, semantic-cache carefully with a tight threshold and ideally a cheap verification that the cached answer actually fits the new query. Great addition — definitely earns a spot in the follow-up.
This is a genuinely a great checklist the framing of retrieval as a chain that can break at any link, not one blob called RAG is exactly right.
Good checklist - and I'd add one check that cuts across every link, because it bit us three times in production (disclosure: I build a memory/retrieval layer, so these are our own scars): can this link report absence as success? Our vector search once ran for days on zero embeddings - the embedding step had silently failed, cosine similarity over an empty index returned empty candidate sets, the keyword fallback kicked in, and every dashboard stayed green. No checklist item about chunking or reranking catches that; only an invariant like "the index is non-empty and covers ≥ N % of documents, asserted at query time" does. Same shape twice more: a coverage check that sampled 5 of 67 shards and reported fleet health, and a health endpoint whose select 1 succeeded while the application schema was broken. The pattern: each link needs not just a quality check but an existence check - "did I actually observe what I claim to summarize?"
Honest question on the 73 %: which analysis is that from? I'd genuinely like to read it - we measure our own retrieval failure split and I'd love an external baseline to compare against.
This is a fantastic addition, and "can this link report absence as success?" is a sharper framing than anything in the post. The empty-index-but-green-dashboard story is the perfect horror case: every quality check assumes there's something to assess quality on, so absence sails straight through. Cosine similarity over an empty index isn't an error, it's just an empty set — and empty looks identical to "nothing matched," which the fallback then papers over. Nightmare precisely because nothing throws.
The three cases you list are the same bug wearing three costumes, and naming the shape is the useful part: an existence check is not a quality check. "Did I observe what I claim to summarize?" is a different assertion from "is what I retrieved any good?" — and only the first catches silent absence. The shard-sampling one is especially nasty because it does observe something, just not representatively: 5 of 67 shards reporting fleet health is absence-of-coverage masquerading as presence. And select 1 succeeding while the schema is broken is the canonical version everyone's shipped at least once. I'm going to add a cross-cutting item on this and credit you — "assert the index is non-empty and covers ≥N% of docs, at query time" is exactly the invariant, and at query time is the load-bearing phrase, since a build-time check wouldn't have caught the drift.
On the 73% — fair challenge, and I should be straight with you: I can't point you to a single authoritative study behind that number. It recurs across vendor writeups and practitioner analyses on the retrieval-vs-generation failure split, but the ones I've seen don't publish methodology I'd want to stake a claim on, and "retrieval failure" isn't defined consistently between them (some count "right doc, wrong rank" as retrieval; some fold it into generation). So treat it as a directional "most failures are upstream" signal, not a measured constant — I flagged it loosely as "industry analysis" in the post for exactly that reason, but your comment makes me think I should soften it further or drop the specific figure. If you're measuring your own split with a real definition, your internal number is worth more than the folklore one — and I'd genuinely rather cite a rigorous source than a sticky stat. If you ever publish your failure-split methodology, I'd link to it in a heartbeat.
Here's our split, with the definition attached so you can reject or reuse it. We measure four nested gates on a frozen benchmark: in-pot (did the candidate set contain the right answer at all - nomination), @10, @3, @1 (ranking). Failure attribution falls out of the nesting: on our house benchmark the floors sit at 97 / 72 / 55 / 41 - so nomination loses ~3 %, and burial, not absence, is the giant: the right answer is in the candidate set but ranked out of sight for ~40 % of questions at @3. "Right doc, wrong rank" is retrieval failure in our book, and it dwarfs everything else - which is why I'd gently push back on folding it into generation: the model never saw what it needed, that's not its failure. Generation failure we only count when the right doc was in the delivered context AND the answer still went wrong. I'm writing the methodology up properly as part of a coming piece - happy to ping you when it's out, and thanks for being straight about the 73 %; "directional, not measured" is exactly the honest label.
This is exactly the rigor the folklore number was missing — thank you for putting real definitions behind it. The nested gates (in-pot → @10 → @3 → @1) are clean precisely because attribution falls out of the nesting instead of being argued after the fact.
And your floors settle the question I was hand-waving: 97/72/55/41 means nomination barely loses (~3%), and burial is the giant — the right answer sits in the candidate set but ranked out of sight for ~40% of questions at @3. That's a fantastic argument for reranking being the highest-ROI link, made with numbers instead of vibes.
Fully agree on not folding "right doc, wrong rank" into generation. The model never saw what it needed — charging it for that is blaming the reader for a book that was left on the wrong shelf. Your generation-failure definition (right doc in delivered context AND still wrong) is the right gate, because it's the only one where the model actually had a fair shot.
One preliminary data point on "burial is the giant," since that exchange sent us after it directly: we identified ~45 boilerplate stems that dominate question phrasing (template words carrying no discriminative signal) and trimmed them before embedding. Exploration set: @3 went 59.3 → 60.4 (+1.1 points, n=1999, p=0.096 - not yet significant). The confirmation set is running now, paired McNemar, and we'll report the result either way, including a null.
Worth noting how it relates to your reranking point: this is upstream of reranking - making the embedding see the words that matter instead of the template - but it attacks the same ~40% burial share. If both land, they should stack, since one fixes what the ranker sees and the other fixes how the shortlist is ordered.
Confirmation set is in, and the honest report turned out to be the interesting one: the effect flipped sign. Exploration set: +1.1 @3 (n=1999, p=0.096). Confirmation set: −1.2 @3 (n=2001, paired McNemar, p=0.0043) - significantly worse, not just a null.
Root cause, we're fairly confident: the ~45 trimmed stems were derived from the exploration set's own question frequencies - so the rule was fitted to the very set that scored it. The confirmation set has a harder baseline (44.2 vs 59.3 @3), and there the trimmed words carried signal for twice as many questions as they were noise for (+21 improved, −45 got worse).
So: candidate rejected, nothing ships. The pre-registered confirmation hurdle is the only reason this didn't reach production wearing a "+1.1 improvement" label. If there's a takeaway for the checklist: never derive a trimming or stop rule from the same set that grades it - and a confirmation set isn't bureaucracy, it's the thing that catches the sign flip. Burial is still the giant; this just wasn't the weapon.
The separation between retrieval quality and answer faithfulness is probably the most important part of this checklist.
I’d add one more check between the two: evidence verification.
A retrieved chunk can be relevant, and the generated answer can still misrepresent it. So instead of only asking “did we retrieve the right chunk?” and “is the answer grounded?”, I’d want the system to verify that each important claim can actually be traced back to a specific passage in the retrieved source.
In other words:
retrieve → rank → generate → independently verify claims
That makes retrieval and generation less of a trust chain and more of a verification pipeline. It also gives you a much better failure signal: wrong retrieval, unsupported claim, or correct evidence but incorrect interpretation.
Yes — evidence verification is the link I underweighted, and you've named it precisely. "Relevant chunk, misrepresented in the answer" is a real failure mode that both my checks miss: retrieval recall says the right passage was there, faithfulness says the answer sounds grounded, and neither confirms that this specific claim traces to that specific span.
What I like most is that per-claim tracing gives you a three-way failure signal instead of a binary. "Wrong retrieval / unsupported claim / correct evidence but wrong interpretation" are three different bugs with three different fixes, and lumping them into one "it was wrong" tells you nothing about where to look. That third category especially — right source, wrong reading — is invisible to everything upstream of it.
And your reframe is the real point: retrieve → rank → generate → verify turns a trust chain into a verification pipeline. Every arrow before the last one is hope; the last one is a check. Adding it to the checklist with credit.
That “right source, wrong reading” category is the interesting one.
It makes me think verification shouldn't only ask whether a claim has supporting evidence, but whether the claim actually follows from that evidence.
So perhaps there are two different checks:
Evidence exists → Evidence entails the claim
The first catches unsupported claims. The second catches cases where the model cites a real passage but over-interprets it.
That distinction could make the failure taxonomy even more actionable: retrieval errors need better search, unsupported claims need grounding controls, while evidence-but-wrong-interpretation needs a reasoning/entailment check.
At that point the RAG pipeline starts looking less like “retrieve context for an LLM” and more like a system for constructing and verifying an evidence chain.
The evidence-exists / evidence-entails split is the sharper version of what I was groping at — those really are two different checks, and collapsing them is why "faithfulness" always felt too coarse to act on.
The distinction is that they fail for opposite reasons. Evidence exists is a retrieval-and-citation question: is there a passage backing this claim at all? Evidence entails is a logic question: given that passage, does the claim actually follow, or did the model stretch "X is common in Y" into "X is required for Y"? A citation check passes the second case — the passage is real, it's cited, and the reasoning on top of it is still wrong. Only an entailment check catches over-interpretation.
And you're right that this is what makes the taxonomy actionable, because each failure now routes to a different fix: retrieval errors → better search; unsupported claims → grounding controls; correct-evidence-wrong-reading → an entailment/reasoning check. Three bugs, three owners, three interventions — instead of one undifferentiated "the answer was wrong" that tells you nothing about where to go.
Your last line is the real reframe, though. Once you're verifying an evidence chain link by link — retrieved, ranked, cited, entailed — it stops being "retrieve some context for an LLM" and becomes a system for constructing and checking a chain of evidence, where the LLM is one step in the chain rather than the thing you blindly trust at the end. That's a genuinely different design posture, and I think it's where robust RAG is heading. Might have to write the follow-up around exactly this.
Another failure mode is retrieving the right fragment from the wrong version of the truth.
A document may be highly relevant and faithfully quoted, but already superseded by a newer decision. I have found it useful to track the owner, effective date, status, and scope of each source before ranking its content.
Where would you resolve conflicts between two relevant sources: metadata filtering, reranking, or a separate policy layer?
One thing I’d add to the checklist is testing for stability, not just accuracy. In our AI work at IT Path Solutions, we’ve found that a retrieval change can improve benchmark results while still being surprisingly sensitive to small query or corpus changes. Measuring that sensitivity alongside accuracy can expose brittle retrieval pipelines that a single recall score might otherwise hide.
This is a genuinely important addition, and it's the axis a single recall score is blindest to. Accuracy tells you how high the pipeline scored; stability tells you how much you can trust that score to survive contact with reality — and those come apart more often than people expect. A change that bumps the benchmark but is wildly sensitive to a reworded query or a slightly different corpus isn't an improvement, it's a number that happened to land well on the exact inputs you measured.
What I like about framing it as its own test is that brittleness hides inside a good average. A pipeline can post strong recall while a handful of near-duplicate queries swing between hit and miss depending on phrasing — the mean looks healthy, the variance is quietly terrifying, and a single score never shows you the second one. It's the retrieval version of a metric that helps a few queries a lot while hurting many a little: same headline number, opposite reliability.
Practically, I take this as: perturb and re-measure. Paraphrase the query set, add or drop a slice of the corpus, reorder ties — and watch how much the ranking moves. If small perturbations produce large swings, the pipeline is overfit to your test conditions no matter how good the recall looks. Measuring that sensitivity alongside accuracy is exactly the kind of thing that separates "passed the benchmark" from "will hold up in production." Adding this to the follow-up with credit — thanks for surfacing it.