The first time my RAG system gave a confidently wrong answer, I did what everyone does: I blamed the model. I swapped in a bigger one. I tuned the prompt. I added "only answer from the context provided" in bold. The answer got no better.
The problem was never the model. The model was faithfully summarizing the context it was handed — the context was just wrong. It had retrieved the wrong chunks, so it answered the wrong question, fluently.
This turns out to be the norm, not the exception. Industry analysis in 2026 keeps landing on the same number: when RAG fails, the failure is in retrieval roughly 73% of the time, not generation. The LLM gets blamed for a mistake that happened several steps upstream, before it ever saw a token.
So here's the checklist I wish someone had handed me before I shipped — organized as a walk through the whole pipeline, because retrieval isn't one step, it's a chain, and it can break at any link. Naive RAG ("chunk, embed, cosine similarity, stuff into prompt") was always a prototype. This is the gap between that and production.
Let's go link by link.
First, the mental model: two paths, not one pipeline
The mistake underneath a lot of RAG pain is treating RAG as a single flow. It's actually two separate paths that most people accidentally couple together.
The indexing path (offline). Runs when documents are added or changed: parse the source → clean the text → chunk → (optionally) enrich each chunk with context → embed → write to the vector store and a keyword index. This can take minutes per document and runs in the background.
The query path (online). Runs on every user request, in real time, under a latency budget (aim for under ~3 seconds end to end): take the query → optionally rewrite it → retrieve candidates → rerank → assemble the prompt with citations → generate → log the trace.
The most common architectural mistake is coupling these. If re-indexing forces the query path offline, you can't iterate on chunking or swap embedding models without downtime — so you stop iterating, and a frozen pipeline is a stale pipeline. Keep them independent from day one.
Check: Can you re-chunk and re-embed your whole corpus without taking live search down? If not, decouple the paths before anything else.
☐ 1. Is your chunking splitting ideas in half?
Chunking is where pipelines silently fail, because bad chunks don't throw errors — they just quietly return technically-relevant, practically-useless context.
The naive default — "split every 1,000 characters with 100 overlap" — is a fast start and a slow ceiling. Fixed-size splitting cuts sentences mid-thought, tables mid-row, and code mid-function. The retrieved chunk looks relevant and is missing the half that mattered.
Better options, roughly in order of effort:
-
Structure-aware splitting — split on the document's own boundaries:
##headings for docs, per-function or per-class for code, per-row for tables. Low effort, big payoff, respects how the content is actually organized. - Semantic chunking — compute similarity sentence-to-sentence and start a new chunk where the meaning shifts, so each chunk is one complete thought. More compute, but a published comparison reported it lifting accuracy meaningfully over fixed-size on the same dataset.
The rule to hold onto: each chunk should be able to answer a question on its own. If a chunk only makes sense next to its neighbor, your splitting is too aggressive. Also mind chunk size — too small and you fragment ideas; too large and you dilute the signal, forcing the model to average across a wall of mostly-irrelevant text.
Check: Pull ten random chunks and read them cold. Does each stand on its own, or are half of them sentence fragments and orphaned table rows?
☐ 2. Are you embedding the chunk — or the chunk in context?
A subtle, high-impact one. If you embed only the raw body text of a chunk, you strip away the context that told a human what it meant — which section it's under, which product it's about, what came before it.
Two fixes, both cheap relative to their payoff:
- Embed context, not just body. Prepend the heading, a short document summary, or a one-line description of what the chunk is about before embedding. This aligns the chunk's vector with how people actually phrase questions. (This is the core idea behind "contextual retrieval" — giving each chunk a little situating context before it's indexed measurably improves recall.)
- Keep metadata attached. Every document arrives with structure — author, date, source, section, product version, document type, access level. Store it alongside the chunk. You'll use it in the next step.
Check: Does an isolated chunk in your index carry any signal about where it came from, or is it a naked paragraph with no situating context?
☐ 3. Are you using hybrid search — or just vector search?
This is the single most common retrieval mistake, and it hides in plain sight because vector search usually works.
Pure vector (semantic) search is great at meaning. Ask "how do I fix login problems?" and it'll surface chunks about authentication, OAuth, and session timeouts even if none use the word "login." That's the magic.
But it falls on its face the moment a query contains something exact. A user searches for the error code ERR_SSL_PROTOCOL_ERROR, a SKU like WX-4200, or a specific function name — and vector search has no idea what to do, because semantic similarity is meaningless for a serial number. It returns things "sort of about errors" and misses the exact match sitting right there in your corpus.
The fix is hybrid search: run keyword search (BM25/full-text) and vector search on the same query, then fuse the results — Reciprocal Rank Fusion (RRF) is the standard merge. Keyword catches exact matches; vector catches meaning. The consensus across 2024–2026 benchmarks (BEIR, MTEB, and others) is blunt: BM25 + dense embeddings fused with RRF beats either one alone, on basically every public benchmark. Dense-only retrieval lost that argument.
Check: Does your retrieval handle a literal error code and a vague conceptual question equally well? If not, you're probably vector-only, and adding a keyword index is your highest-leverage change.
☐ 4. Are you transforming the query — or retrieving the user's raw words?
Here's a link most people skip entirely: the user's literal question is often not what the retriever wants. People ask vague, compound, context-dependent questions; your index holds precise, standalone statements. Bridging that gap is query transformation, and it's one of the biggest quiet wins available.
The main patterns, each solving a different problem:
- Query rewriting / expansion — clean up and enrich the raw query before retrieval so it matches the corpus better. Especially important in multi-turn chat, where "what about the second one?" is meaningless without rewriting it into a standalone query.
- HyDE (Hypothetical Document Embeddings) — have the LLM generate a hypothetical answer to the question, then embed that and retrieve against it. A fake answer is often shaped much more like the real documents than the question is, which boosts precision.
- Step-back prompting — rewrite a narrow question into a more general one, retrieve the background, then specialize. Good for ambiguous queries where the literal phrasing misses the corpus.
- Decomposition — split a multi-part question into independent sub-queries, retrieve each separately, then synthesize. "How does our refund policy differ between B2B and B2C, and what are the exceptions?" is really three retrievals, not one.
You don't need all of these. But if you're feeding raw user text straight into the retriever, you're leaving a lot of recall on the table — especially for compound and conversational questions.
Check: Take your ten hardest real user questions. How many would retrieve better if they were rephrased, split, or expanded first? If it's most of them, add a transformation step.
☐ 5. Are you reranking — or trusting first-pass retrieval order?
The mistake that cost me the most quality for the least obvious reason: I assumed that if the right chunk was retrieved, the model would use it. But where it lands in the list matters enormously.
Vector search uses a bi-encoder — it encodes the query and each chunk separately and compares vectors. Fast, but it trades away fine-grained relevance. So the genuinely best chunk often gets retrieved... at position 8, buried under seven "pretty relevant" ones. And models demonstrably ignore information stranded in the middle of a long list — the "lost in the middle" problem. The right answer is in the context and the model still misses it.
Reranking fixes this. Retrieve a broad candidate set with hybrid search (top 20–50), then run a cross-encoder reranker that scores each (query, chunk) pair jointly — seeing query and chunk together, which makes it far better at fine-grained relevance than first-pass retrieval. Keep the top 3–8.
The impact is large, not marginal: a cross-encoder reranker commonly adds 5–15 points of MRR on hard sets, and on some reasoning-heavy benchmarks reranking pushed nDCG@10 from ~0.13 to ~0.40 — roughly 3x, just from reordering the same candidates you already retrieved.
The recipe that beats ~80% of production deployments: retrieve ~20 via hybrid search → rerank to ~5 → send 3–5 to the LLM. Reranking 100+ candidates rarely pays; the signal lives at the head.
Check: Is there a reranking step between retrieval and the prompt? If chunks go straight from vector similarity into the context window, add one — it's often the highest-ROI change in the whole pipeline.
☐ 6. Is your context assembly helping the model — or dumping on it?
You've retrieved and reranked the right chunks. You can still lose here, at the stage nobody talks about: how you actually assemble the prompt.
Things that quietly hurt:
- Order. Because of "lost in the middle," put the strongest chunks at the very start and end of the context, not buried in the center.
- Volume. More chunks is not better. Stuffing 30 chunks in "to be safe" dilutes the signal and invites the model to average across noise. Send the few that earned their place.
- No citations. Ask the model to cite which chunk supports each claim. This both discourages free-floating fabrication and gives you a way to verify the answer against its sources.
- Long-context ≠ skip retrieval. Frontier models have million-token windows now, and the reflex is "just dump everything in, who needs retrieval." Resist it. Dumping the whole corpus is slower, more expensive, and less accurate than sending a few well-chosen chunks, because the model still has to find the needle. Use the big window for genuine synthesis (long reports, whole codebases), not as a substitute for retrieval.
Check: How many chunks do you send, and in what order? If the answer is "as many as fit, in retrieval order," you're leaving quality on the floor.
☐ 7. Do you need agentic RAG — or are you bolting complexity onto a broken pipeline?
Everything above describes a single retrieve-then-generate pass. That has a ceiling: it works on simple questions and falls apart on nuanced, multi-hop ones where the answer isn't in any single chunk. Enter agentic RAG.
The shift is structural. Classic RAG does one retrieval call, up front, stateless — if it misses, there's no recovery. Agentic RAG moves retrieval inside a reasoning loop (the ReAct pattern: the model alternates between thinking and calling tools). Now the model can retrieve, look at what it got, decide it's not enough, rewrite its query, retrieve again, and stop when it has what it needs. Retrieval becomes a tool the agent uses repeatedly, not a fixed step in front of it.
This unlocks multi-hop questions — "what's the GDP of the country that hosted the 2024 Olympics?" needs hop one (Olympics → France) before hop two (France → GDP). A single retrieval can't do that; an agent that retrieves, reasons, and retrieves again can. Related advanced patterns include GraphRAG (build a knowledge graph from your docs to answer questions that require connecting entities across many documents) and giving the agent explicit tools: search, fetch-full-document-by-id, exact-match/regex, and even prune-context-to-discard-junk.
The honest caveat — and this ties straight back to over-engineering: agentic RAG costs more (more calls, more latency, more nondeterminism) and is worth it for genuinely complex or high-stakes retrieval (legal, medical, financial, multi-hop). It is not a fix for a broken basic pipeline. If your chunking is bad and you have no reranking, an agent will just make bad retrieval calls, repeatedly, more expensively. Get single-pass retrieval solid first. Add the agent loop only when the questions genuinely need multiple hops.
Check: Do your failing questions actually require chaining facts across documents — or would they be answered fine by hybrid search + reranking you haven't implemented yet?
☐ 8. Can you measure retrieval in isolation — and catch a fabrication?
This is the meta-mistake that hides all the others. Most teams evaluate RAG end-to-end: read the final answer, decide it "seems good," ship. But an end-to-end answer blends retrieval and generation, so when it's wrong you can't tell which half failed. You'll spend a week tuning prompts to fix a chunking bug.
Two things you need to measure separately:
Retrieval quality on its own. Given a query, did the right chunk make it into the retrieved set (recall), and how high did it rank (rank / nDCG / MRR)? For multi-hop and agentic setups, recall has to be measured across the whole chain, not one call. Frameworks like RAGAS exist, but even a hand-built set of real queries mapped to their correct source chunks beats vibes.
Faithfulness / groundedness of the answer. Is every claim in the final answer actually supported by the retrieved context? This is the check that catches the scariest failure: the agent that retrieves 8 chunks, uses 6, and invents the 7th fact entirely. Without a faithfulness score or a judge gating the output, that fabrication ships and a customer finds it two days later.
One hard-won caution: a tiny eval set will lie to you. If your test set is small and easy, every method scores near-perfect and they all look equally good — the differences that matter on real data are invisible on a toy set. A retrieval eval is only an eval if methods can actually fail on it. If everything scores 95%, you've built a smoke test, and a smoke test will happily bless the broken layer you were hoping to justify.
Check: If retrieval regressed tomorrow, would a number tell you — or would a user? If it's the user, you can't measure retrieval yet, and everything above this line is guesswork.
The whole checklist, in one screen
- Decouple the indexing path from the query path so you can iterate without downtime.
- Chunk so each piece stands alone — structure-aware or semantic, never blind fixed-size.
- Embed in context — prepend headings/summaries, keep metadata attached.
- Hybrid search — BM25 + vector, fused with RRF. Never vector-only.
- Transform the query — rewrite, HyDE, step-back, or decompose before retrieving.
- Rerank with a cross-encoder — retrieve ~20, rerank to ~5, send 3–5.
- Assemble context deliberately — best chunks first and last, few not many, with citations.
- Go agentic only when needed — multi-hop and high-stakes, on top of a solid base.
- Evaluate both retrieval recall and answer faithfulness — on a set hard enough to fail.
The One Line to Remember
When RAG gives a bad answer, suspect retrieval first — it's the culprit far more often than the model.
The instinct to reach for a bigger model is almost always wrong. The bigger model will summarize the wrong chunks just as fluently as the small one did. The leverage is upstream — in finding the right chunk, making it usable, ranking it where the model will see it, and being able to tell when any link in the chain breaks.
I learned this checklist one confidently-wrong answer at a time. You don't have to.
What's the retrieval bug that fooled you the longest? Mine was a chunking issue I spent two weeks blaming the model for. Share yours — and any checklist items I missed — in the comments.
Top comments (37)
James, excellent checklist. The discussion in the comments — specifically from @mnemehq, @suraj09, and the memory-layer builder above — is circling around a single missing architectural boundary: the gap between mechanical pipeline success and semantic truth.
Your 9 steps guarantee mechanical execution. But this thread just named three distinct ways a pipeline can be mechanically perfect, pass every green dashboard check, and still be dead wrong:
exit code 0. The system records "success," fallback kicks in, and the dashboard stays green while serving pure noise.Evidence ExistsvsEvidence Entails): As @suraj09 pointed out, retrieving a valid chunk is not enough. The model often over-interprets "X is common in Y" into "X is required for Y." The citation check passes because the passage exists, but the semantic claim is false.These aren't separate edge cases — they are symptoms of a missing Semantic & Policy Verification Layer. The retrieval pipeline proves the bytes were found; it doesn't prove the claim is true, current, or permitted.
In our work with codebase intelligence (MSCodeBase Intelligence, indexing ~50K LOC), we hit this exact wall. A chunk could match with 0.95 cosine similarity, but if the underlying function was refactored yesterday, the chunk was semantically dead. Better reranking couldn't fix it — only a live verification step that validates claims against current system state before execution.
A cross-cutting question for the checklist: "Does the pipeline distinguish between 'I retrieved a matching context' and 'the claim generated from this context is currently true and actionable'?"
Without that boundary, a perfectly-tuned retrieval pipeline just produces cryptographically perfect hallucinations.
This is a great extension of the verification boundary. “Evidence exists” vs “evidence entails” is especially important — retrieval can be correct while the conclusion is wrong. And as you point out, true + relevant still doesn’t necessarily mean permitted to act.
"Cryptographically perfect hallucinations" is the phrase this whole thread was reaching for — thank you for naming it. You're right that my nine steps prove the bytes were found, and every failure surfaced here (silent absence, evidence-exists-vs-entails, the enforcement gap) is a case where the bytes are perfect and the claim is still false, stale, or unpermitted. The unifying insight is that better retrieval can't fix any of them, because they live on a different axis entirely — and worse, high retrieval accuracy launders all three by wrapping the wrong outcome in real evidence. Your MSCodeBase example is the cleanest possible proof: 0.95 cosine similarity on a function refactored yesterday isn't a retrieval miss, it's a semantically dead hit that no reranker can catch, only a live check against current system state. So your cross-cutting question is the right one to pin above the whole checklist: does the pipeline distinguish "I retrieved matching context" from "the claim from this context is currently true and actionable"? Retrieval decides what's found; a semantic-and-policy verification layer decides what's true, current, and permitted — and only the second one keeps you from shipping evidence-backed nonsense. Adding this to the follow-up with credit to you and the thread.
Thanks for the great response, James! Glad to see the distinction resonating so strongly with the community.
James, the checklist is the first one I have read that walks the whole path rather than the retriever alone, and innokentyb asked the question underneath it: whether stale records get fixed with metadata filters, reranking, or a separate policy layer. The honest answer is none of the three. Filters and reranking work after the fact, at query time, comparing dates or scores on records that already sit side by side in the index. Supersession has to be marked at the moment a new version is written, not inferred later during ranking. Five people commenting on posts of mine landed on that distinction independently, which is a stronger signal than one person making the case. One of them added a condition none of the others had: the writer must either link the new record to what it replaces or record explicitly that no predecessor was found, and that lookup is not allowed to fail silently. A silent miss there is worse than not tracking supersession at all, because it looks handled and is not.
On the context assembly item, the symptoms are named well: order sensitivity, how much fits, information lost in the middle. What is missing is the mechanism. Truncation keeps the record and drops the link between a chunk and its status, whether it is current or superseded, and nothing downstream is told that happened. Merging is fine, full replacement is fine, dropping a chunk entirely is fine. Clipping by window size is not, because that is the one operation that severs the status link without a trace.
What I do not have is a clean way to test for that loss from outside a system, short of forcing a supersession on purpose and watching whether the old chunk survives truncation.
This closes the loop innokentyb opened, and the answer — none of the three — is sharper than anything I offered in that subthread. You've named why: filters and reranking are both query-time operations comparing records that already sit side by side in the index, which means they can only act on supersession after it's been recorded. If the record was never marked stale at write time, there's nothing for a filter to filter on or a ranker to down-weight. Supersession is a write-time fact, not a query-time inference — and trying to reconstruct it during ranking is asking the retriever to guess a truth the indexer forgot to store. Five people landing on that independently is exactly the kind of signal I trust more than one confident argument.
And the added condition is the part I'd have missed: the writer must either link the new record to what it replaces or explicitly record that no predecessor was found — and that lookup can't fail silently. That's the same disease this whole comment section keeps diagnosing in different costumes: a silent miss that looks handled is worse than no handling at all, because the first one you'll trust and the second one you'll at least know to check. A supersession link that quietly fails to resolve gives you a stale record wearing a "current" badge, which is the most dangerous state in the entire pipeline.
The context-assembly point is the one I'm genuinely taking away, because you found a mechanism I stated only as a symptom. "Lost in the middle" I framed as the model ignoring buried content — but you're pointing at something worse happening before the model even reads it: truncation keeps the chunk's text and severs its status link, with nothing downstream told it happened. Merge, replace, drop-entirely — all fine, because they either preserve the association or remove the whole thing cleanly. Clipping by window size is the one operation that keeps the bytes and silently strips whether those bytes are current or retired. So a superseded chunk can survive assembly stripped of the very metadata that would have flagged it as superseded, and now it's indistinguishable from a live one. That's a real bug and I don't think I've seen it named before.
On testing it from outside — I don't have a clean answer either, and I suspect there isn't one that's purely observational, because the whole failure is the absence of a signal. The forced-supersession probe you describe is probably the honest floor: plant a known-superseded chunk, run a query that pulls it into a context that will truncate, and assert the retired chunk either carries its status through or gets dropped — never survives stripped. Which is really the red-green-red discipline from another thread applied to metadata survival: prove the status link can be observed to break, on purpose, or you have no evidence it holds. Adding all of this to the follow-up with credit — the write-time-vs-query-time framing and the truncation-severs-status mechanism are two of the most useful things anyone's contributed to this piece.
The probe inherits the failure it is built to catch. It names two conditions, that the query pulls the plant in and that assembly truncates, but names them as intentions rather than as checks, and both fail quietly. A run where the plant was never retrieved satisfies never survives stripped. So does a run where the context stayed under budget and nothing was clipped. Both come back green. Those two have to be asserted separately and loudly, or green only means nothing happened, which is the state you named as the dangerous one.
Second, the verdict turns on where the boundary landed rather than on any rule the system holds. If the status sits at the head of the chunk and the cut takes the tail, it passes by accident of byte position, and one green run is a sample of one offset. The cut has to be forced inside the plant at more than one position, and run once with the link severed on purpose, so the assertion is shown able to go red. And if the grading reads the model output rather than the assembled context, a clean drop and a carried status can look identical, which makes the pass unreadable either way.
The limitation on my own half is sharper than both. A planted chunk is the best case a corpus can offer: written through the current path, predecessor present, link fresh. The records that hurt are older than the status field, or carry the explicit no-predecessor-found marker from a lookup that came back empty. A passing probe licenses a claim about writes made today and says nothing about what is already sitting in the index. That gap does not close with this probe.
You've done to my probe what the whole thread keeps doing to everything else — turned its two intentions into two silent failures, and you're right on both counts. "Force a supersession, watch whether the old chunk survives truncation" names what should happen and quietly assumes the setup occurred. A run where the plant was never retrieved passes "never survives stripped" vacuously. So does a run that stayed under budget and clipped nothing. Both come back green, and green here means nothing happened, which is precisely the state I called the dangerous one — my own test-of-a-test fell into the exact hole the test was built to guard. So the two preconditions have to be assertions, loud and separate: assert the plant was retrieved, assert truncation actually fired inside it — a probe that can pass without both is measuring the absence of an event, not the survival of a link.
The byte-position point is the sharper one, because it means even a correctly-triggered green can be an accident. If the status sits at the head of the chunk and the cut takes the tail, the link survives by luck of offset, and one green run is a sample of one cut point. That's not evidence the system holds the status through truncation — it's evidence it held it this once, at this boundary. The fix is the ablation discipline from upthread pointed here: force the cut inside the plant at multiple positions, and run it once with the link severed on purpose so the assertion is shown able to go red. A survival check that has never been observed to fail is a green that can't fail, one floor down. And your grading-surface catch closes the last hole: if the probe reads the model output instead of the assembled context, a clean drop and a carried status can render identical, so the pass is unreadable regardless of which happened. Grade the context, not the generation.
But the limitation you put on your own half is the part I'll actually keep, because it's the most honest boundary anyone's drawn in this thread. A planted chunk is the best case the corpus can offer — written through the current path, predecessor present, link fresh. The records that hurt are the ones older than the status field itself, or the ones carrying an explicit no-predecessor-found marker from a lookup that came back empty. A passing probe licenses a claim about writes made today and says exactly nothing about the legacy already sitting in the index. That gap doesn't close with any probe of this shape, because the probe can only exercise the write path it has, and the dangerous records predate that path. Which means supersession integrity isn't one problem — it's "does the write path preserve status going forward" (probeable, with your fixes) and "what's the status of everything written before the path existed" (a backfill/audit problem the probe can't touch). Naming that second one as unclosed, instead of letting a green forward-probe imply the whole index is sound, is the difference between a check and a reassurance. Going into the revision with all of it credited — the probe-inherits-its-own-failure framing is the spine of the correction.
The split is right, and the second half is harder than a backfill. An audit can decide a fact about a record: whether it carries a status field at all. It cannot decide whether the record is current, because that requires knowing what superseded it, and for anything written before the status field existed the link was never recorded. Reconstructing it means inferring supersession at audit time from similarity or recency, which is the exact move the write-time argument says is unsound. So the legacy set is not merely expensive, it is partly undecidable by the same reasoning that made the forward case work.
What follows is a third class rather than a cleaner split. Not current and not retired, but unknown by construction, and it has to be a state the retriever can see, because both defaults are failures already named in this thread. Default unknown to current and you get the stale record wearing a current badge. Default it to retired and you silently drop facts nothing ever contradicted, which is the same silent miss pointed the other way.
The one thing that shrinks the class honestly is provenance plus time. A record written after the path existed is forward-probeable and can be proven so, which closes the unknown set at one end and makes it monotonically smaller. That is a slower claim than an audit result, and it is the one that survives being checked, because it never asserts anything about a record whose history was not written down.
One thing I’d add: retrieval success ≠ semantic correctness. A relevant chunk can still support the wrong conclusion. I’d test whether the retrieved evidence actually entails the claim, not just whether it matches the query.
Great checklist, I just published an article evaluating Self-Rag, query decomposition and HyDE and how they compare on several chunk sizes and affect retrieval quality.
If you read this blog post this will be a good follow up for a deep dive.
dev.to/shadrach1999/building-a-pro...
That sounds like exactly the deep dive this post gestures at but doesn't measure — Self-RAG, query decomposition, and HyDE compared across chunk sizes is the kind of head-to-head data I'd love to point people to instead of hand-waving "it depends." The chunk-size interaction especially interests me, since so much advice treats these techniques as independent of chunking when they clearly aren't. I'll definitely give it a read — thanks for flagging it, and for the kind words.
You could also reverse the hyde during storing embedding vectors, where you can pre-generate questions for sections, that way search becomes a bit faster. Indexing can also be progressive, so that time to search decreases, especially if you have people uploading batches of documents
Reversing HyDE at index time is clever — pre-generating likely questions per section means you're matching question-to-question instead of question-to-answer, which sidesteps the query/document shape mismatch entirely. It front-loads the LLM cost to indexing (where latency doesn't matter) instead of query time (where it does), so search gets both faster and better-aligned. And progressive indexing is the right call for batch uploads — letting documents become searchable incrementally beats blocking on a full re-index. Both are going in the follow-up; great additions.
Solid point: RAG without a checklist will confidently mislead. The fix is tighter retrieval, proven provenance, and safe fallbacks. Add adversarial prompts, post-answer citation audits, and a live user-feedback loop. Bonus: a freshness timer to flag stale sources before they masquerade as truth.
"Confidently mislead" is the whole danger in two words — a RAG system fails louder than a search box because the fluent answer hides the broken retrieval underneath. I like that your additions are all adversarial or continuous rather than one-time: post-answer citation audits and a live feedback loop assume things will drift and catch it, instead of trusting a green check from launch day. And the freshness timer is the neat fix for the "right fragment, superseded source" problem someone else raised — stale-but-relevant is exactly the failure that masquerades as truth. Great list.
Good checklist. I'd add one more line: retrieval quality and enforcement are separate failure surfaces. You can retrieve the exact right chunk of context and still let the agent act against it in a way nobody sanctioned. Fixing recall doesn't fix that.
This is a sharp addition, and it names a boundary the checklist blurs: retrieval and enforcement are separate failure surfaces, and every item in the post lives on the first one. Recall, ranking, faithfulness — all of it answers "did the right context reach the model?" None of it touches "was the model allowed to act on it the way it did?" You can nail the entire retrieval chain and still have an agent do something nobody sanctioned with a perfectly-retrieved fact.
The reason this hides so well is that a correct retrieval makes the action look justified. The chunk was right, the citation is real, the reasoning reads clean — so the unsanctioned action inherits the credibility of the good context it was built on. Fixing recall actually makes this failure more convincing, not less, because now the wrong action comes wrapped in solid evidence. Better retrieval can launder a worse enforcement gap.
It maps onto a distinction that came up in another thread here: "the model saw the right thing" and "the model was permitted to do this thing" are different claims, and only one of them is what retrieval evals measure. Enforcement is a policy question — what actions are allowed against what context, under whose authorization — and policy questions want deterministic, auditable gates, not a similarity score or a faithfulness rating. Retrieval quality is necessary and does nothing for it.
So the line I'd add to the checklist, with credit: retrieval decides what the agent knows; enforcement decides what it's allowed to do with that — and passing every retrieval check tells you nothing about the second. Which, for anything where the agent acts rather than just answers, is often the surface that actually hurts you. Great catch.
HyDE on a private corpus pulls neighbors of a parametric guess, so dense retrieval hunts documents shaped like public answers. Query expansion does the same thing when it fills in product names your index does not use, and BM25 then searches tokens that never appear. An internal runbook that answers the original question never gets retrieved. Transform the query is fine. There is still gonna be a cited answer sitting on public-shaped chunks while the internal function names never leave the index.
This is the sharpest failure here, because it attacks the fix, not the naive baseline. HyDE assumes the docs are shaped like the model's prior — on a private corpus they're not. The hypothetical is a public-text guess, so you retrieve neighbors of the public answer and sail past the internal runbook with your actual function names.
And expansion fails the same way: it invents public vocabulary, then BM25 searches tokens that appear nowhere in your index. So both channels drift public at once — dense chases public-shaped embeddings, sparse chases public-shaped tokens — and the right chunk loses on both. Nothing errors; you get a confident cited answer on the wrong chunks, which is worse than a miss because it looks retrieved.
The lesson: query transformation is only safe when it's grounded in the corpus's own vocabulary, not the model's prior. That points at corpus-mined expansion or pseudo-relevance feedback (let real retrieved chunks seed the rewrite, not a hallucinated one) — plus a lexical path that privileges exact in-corpus tokens. "Transform the query" earns the same caveat as everything else in the post: it's a liability the moment it's applied on faith without checking whether it moved you toward your corpus or toward the public web. Going in the follow-up — great catch.
Great work! Really solid checklist - I wish I'd had this before my first RAG rodeo.
One thing I didn't see mentioned, though: caching. In production, 60–80% of queries are repeats or near-duplicates. If you've already answered it once, just serve the cached response - it's cheaper, faster, and cuts LLM load dramatically. For high-traffic systems, it's a no-brainer.
Other than that, this is gold. Thanks for sharing!
Thank you — and yes, caching is a real omission, especially since you're right that in production the query distribution is brutally repetitive; 60–80% repeats or near-dupes is exactly the kind of traffic where re-running the whole retrieve-rank-generate chain every time is pure waste. The one nuance I'd add is that "near-duplicate" is where it gets interesting: exact-match caching is the easy no-brainer you described, but semantic caching (serving a cached answer for a paraphrase of a prior question) is the bigger win and the sharper knife — set the similarity threshold too loose and you'll confidently serve the answer to a slightly different question, which lands you right back in the "confident, wrong, looks retrieved" failure mode the rest of the post is about. So I'd frame it as: exact-match cache freely, semantic-cache carefully with a tight threshold and ideally a cheap verification that the cached answer actually fits the new query. Great addition — definitely earns a spot in the follow-up.
This is a genuinely a great checklist the framing of retrieval as a chain that can break at any link, not one blob called RAG is exactly right.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.