The first time my RAG system gave a confidently wrong answer, I did what everyone does: I blamed the model. I swapped in a bigger one. I tuned the prompt. I added "only answer from the context provided" in bold. The answer got no better.
The problem was never the model. The model was faithfully summarizing the context it was handed — the context was just wrong. It had retrieved the wrong chunks, so it answered the wrong question, fluently.
This turns out to be the norm, not the exception. Industry analysis in 2026 keeps landing on the same number: when RAG fails, the failure is in retrieval roughly 73% of the time, not generation. The LLM gets blamed for a mistake that happened several steps upstream, before it ever saw a token.
So here's the checklist I wish someone had handed me before I shipped — organized as a walk through the whole pipeline, because retrieval isn't one step, it's a chain, and it can break at any link. Naive RAG ("chunk, embed, cosine similarity, stuff into prompt") was always a prototype. This is the gap between that and production.
Let's go link by link.
First, the mental model: two paths, not one pipeline
The mistake underneath a lot of RAG pain is treating RAG as a single flow. It's actually two separate paths that most people accidentally couple together.
The indexing path (offline). Runs when documents are added or changed: parse the source → clean the text → chunk → (optionally) enrich each chunk with context → embed → write to the vector store and a keyword index. This can take minutes per document and runs in the background.
The query path (online). Runs on every user request, in real time, under a latency budget (aim for under ~3 seconds end to end): take the query → optionally rewrite it → retrieve candidates → rerank → assemble the prompt with citations → generate → log the trace.
The most common architectural mistake is coupling these. If re-indexing forces the query path offline, you can't iterate on chunking or swap embedding models without downtime — so you stop iterating, and a frozen pipeline is a stale pipeline. Keep them independent from day one.
Check: Can you re-chunk and re-embed your whole corpus without taking live search down? If not, decouple the paths before anything else.
☐ 1. Is your chunking splitting ideas in half?
Chunking is where pipelines silently fail, because bad chunks don't throw errors — they just quietly return technically-relevant, practically-useless context.
The naive default — "split every 1,000 characters with 100 overlap" — is a fast start and a slow ceiling. Fixed-size splitting cuts sentences mid-thought, tables mid-row, and code mid-function. The retrieved chunk looks relevant and is missing the half that mattered.
Better options, roughly in order of effort:
-
Structure-aware splitting — split on the document's own boundaries:
##headings for docs, per-function or per-class for code, per-row for tables. Low effort, big payoff, respects how the content is actually organized. - Semantic chunking — compute similarity sentence-to-sentence and start a new chunk where the meaning shifts, so each chunk is one complete thought. More compute, but a published comparison reported it lifting accuracy meaningfully over fixed-size on the same dataset.
The rule to hold onto: each chunk should be able to answer a question on its own. If a chunk only makes sense next to its neighbor, your splitting is too aggressive. Also mind chunk size — too small and you fragment ideas; too large and you dilute the signal, forcing the model to average across a wall of mostly-irrelevant text.
Check: Pull ten random chunks and read them cold. Does each stand on its own, or are half of them sentence fragments and orphaned table rows?
☐ 2. Are you embedding the chunk — or the chunk in context?
A subtle, high-impact one. If you embed only the raw body text of a chunk, you strip away the context that told a human what it meant — which section it's under, which product it's about, what came before it.
Two fixes, both cheap relative to their payoff:
- Embed context, not just body. Prepend the heading, a short document summary, or a one-line description of what the chunk is about before embedding. This aligns the chunk's vector with how people actually phrase questions. (This is the core idea behind "contextual retrieval" — giving each chunk a little situating context before it's indexed measurably improves recall.)
- Keep metadata attached. Every document arrives with structure — author, date, source, section, product version, document type, access level. Store it alongside the chunk. You'll use it in the next step.
Check: Does an isolated chunk in your index carry any signal about where it came from, or is it a naked paragraph with no situating context?
☐ 3. Are you using hybrid search — or just vector search?
This is the single most common retrieval mistake, and it hides in plain sight because vector search usually works.
Pure vector (semantic) search is great at meaning. Ask "how do I fix login problems?" and it'll surface chunks about authentication, OAuth, and session timeouts even if none use the word "login." That's the magic.
But it falls on its face the moment a query contains something exact. A user searches for the error code ERR_SSL_PROTOCOL_ERROR, a SKU like WX-4200, or a specific function name — and vector search has no idea what to do, because semantic similarity is meaningless for a serial number. It returns things "sort of about errors" and misses the exact match sitting right there in your corpus.
The fix is hybrid search: run keyword search (BM25/full-text) and vector search on the same query, then fuse the results — Reciprocal Rank Fusion (RRF) is the standard merge. Keyword catches exact matches; vector catches meaning. The consensus across 2024–2026 benchmarks (BEIR, MTEB, and others) is blunt: BM25 + dense embeddings fused with RRF beats either one alone, on basically every public benchmark. Dense-only retrieval lost that argument.
Check: Does your retrieval handle a literal error code and a vague conceptual question equally well? If not, you're probably vector-only, and adding a keyword index is your highest-leverage change.
☐ 4. Are you transforming the query — or retrieving the user's raw words?
Here's a link most people skip entirely: the user's literal question is often not what the retriever wants. People ask vague, compound, context-dependent questions; your index holds precise, standalone statements. Bridging that gap is query transformation, and it's one of the biggest quiet wins available.
The main patterns, each solving a different problem:
- Query rewriting / expansion — clean up and enrich the raw query before retrieval so it matches the corpus better. Especially important in multi-turn chat, where "what about the second one?" is meaningless without rewriting it into a standalone query.
- HyDE (Hypothetical Document Embeddings) — have the LLM generate a hypothetical answer to the question, then embed that and retrieve against it. A fake answer is often shaped much more like the real documents than the question is, which boosts precision.
- Step-back prompting — rewrite a narrow question into a more general one, retrieve the background, then specialize. Good for ambiguous queries where the literal phrasing misses the corpus.
- Decomposition — split a multi-part question into independent sub-queries, retrieve each separately, then synthesize. "How does our refund policy differ between B2B and B2C, and what are the exceptions?" is really three retrievals, not one.
You don't need all of these. But if you're feeding raw user text straight into the retriever, you're leaving a lot of recall on the table — especially for compound and conversational questions.
Check: Take your ten hardest real user questions. How many would retrieve better if they were rephrased, split, or expanded first? If it's most of them, add a transformation step.
☐ 5. Are you reranking — or trusting first-pass retrieval order?
The mistake that cost me the most quality for the least obvious reason: I assumed that if the right chunk was retrieved, the model would use it. But where it lands in the list matters enormously.
Vector search uses a bi-encoder — it encodes the query and each chunk separately and compares vectors. Fast, but it trades away fine-grained relevance. So the genuinely best chunk often gets retrieved... at position 8, buried under seven "pretty relevant" ones. And models demonstrably ignore information stranded in the middle of a long list — the "lost in the middle" problem. The right answer is in the context and the model still misses it.
Reranking fixes this. Retrieve a broad candidate set with hybrid search (top 20–50), then run a cross-encoder reranker that scores each (query, chunk) pair jointly — seeing query and chunk together, which makes it far better at fine-grained relevance than first-pass retrieval. Keep the top 3–8.
The impact is large, not marginal: a cross-encoder reranker commonly adds 5–15 points of MRR on hard sets, and on some reasoning-heavy benchmarks reranking pushed nDCG@10 from ~0.13 to ~0.40 — roughly 3x, just from reordering the same candidates you already retrieved.
The recipe that beats ~80% of production deployments: retrieve ~20 via hybrid search → rerank to ~5 → send 3–5 to the LLM. Reranking 100+ candidates rarely pays; the signal lives at the head.
Check: Is there a reranking step between retrieval and the prompt? If chunks go straight from vector similarity into the context window, add one — it's often the highest-ROI change in the whole pipeline.
☐ 6. Is your context assembly helping the model — or dumping on it?
You've retrieved and reranked the right chunks. You can still lose here, at the stage nobody talks about: how you actually assemble the prompt.
Things that quietly hurt:
- Order. Because of "lost in the middle," put the strongest chunks at the very start and end of the context, not buried in the center.
- Volume. More chunks is not better. Stuffing 30 chunks in "to be safe" dilutes the signal and invites the model to average across noise. Send the few that earned their place.
- No citations. Ask the model to cite which chunk supports each claim. This both discourages free-floating fabrication and gives you a way to verify the answer against its sources.
- Long-context ≠ skip retrieval. Frontier models have million-token windows now, and the reflex is "just dump everything in, who needs retrieval." Resist it. Dumping the whole corpus is slower, more expensive, and less accurate than sending a few well-chosen chunks, because the model still has to find the needle. Use the big window for genuine synthesis (long reports, whole codebases), not as a substitute for retrieval.
Check: How many chunks do you send, and in what order? If the answer is "as many as fit, in retrieval order," you're leaving quality on the floor.
☐ 7. Do you need agentic RAG — or are you bolting complexity onto a broken pipeline?
Everything above describes a single retrieve-then-generate pass. That has a ceiling: it works on simple questions and falls apart on nuanced, multi-hop ones where the answer isn't in any single chunk. Enter agentic RAG.
The shift is structural. Classic RAG does one retrieval call, up front, stateless — if it misses, there's no recovery. Agentic RAG moves retrieval inside a reasoning loop (the ReAct pattern: the model alternates between thinking and calling tools). Now the model can retrieve, look at what it got, decide it's not enough, rewrite its query, retrieve again, and stop when it has what it needs. Retrieval becomes a tool the agent uses repeatedly, not a fixed step in front of it.
This unlocks multi-hop questions — "what's the GDP of the country that hosted the 2024 Olympics?" needs hop one (Olympics → France) before hop two (France → GDP). A single retrieval can't do that; an agent that retrieves, reasons, and retrieves again can. Related advanced patterns include GraphRAG (build a knowledge graph from your docs to answer questions that require connecting entities across many documents) and giving the agent explicit tools: search, fetch-full-document-by-id, exact-match/regex, and even prune-context-to-discard-junk.
The honest caveat — and this ties straight back to over-engineering: agentic RAG costs more (more calls, more latency, more nondeterminism) and is worth it for genuinely complex or high-stakes retrieval (legal, medical, financial, multi-hop). It is not a fix for a broken basic pipeline. If your chunking is bad and you have no reranking, an agent will just make bad retrieval calls, repeatedly, more expensively. Get single-pass retrieval solid first. Add the agent loop only when the questions genuinely need multiple hops.
Check: Do your failing questions actually require chaining facts across documents — or would they be answered fine by hybrid search + reranking you haven't implemented yet?
☐ 8. Can you measure retrieval in isolation — and catch a fabrication?
This is the meta-mistake that hides all the others. Most teams evaluate RAG end-to-end: read the final answer, decide it "seems good," ship. But an end-to-end answer blends retrieval and generation, so when it's wrong you can't tell which half failed. You'll spend a week tuning prompts to fix a chunking bug.
Two things you need to measure separately:
Retrieval quality on its own. Given a query, did the right chunk make it into the retrieved set (recall), and how high did it rank (rank / nDCG / MRR)? For multi-hop and agentic setups, recall has to be measured across the whole chain, not one call. Frameworks like RAGAS exist, but even a hand-built set of real queries mapped to their correct source chunks beats vibes.
Faithfulness / groundedness of the answer. Is every claim in the final answer actually supported by the retrieved context? This is the check that catches the scariest failure: the agent that retrieves 8 chunks, uses 6, and invents the 7th fact entirely. Without a faithfulness score or a judge gating the output, that fabrication ships and a customer finds it two days later.
One hard-won caution: a tiny eval set will lie to you. If your test set is small and easy, every method scores near-perfect and they all look equally good — the differences that matter on real data are invisible on a toy set. A retrieval eval is only an eval if methods can actually fail on it. If everything scores 95%, you've built a smoke test, and a smoke test will happily bless the broken layer you were hoping to justify.
Check: If retrieval regressed tomorrow, would a number tell you — or would a user? If it's the user, you can't measure retrieval yet, and everything above this line is guesswork.
The whole checklist, in one screen
- Decouple the indexing path from the query path so you can iterate without downtime.
- Chunk so each piece stands alone — structure-aware or semantic, never blind fixed-size.
- Embed in context — prepend headings/summaries, keep metadata attached.
- Hybrid search — BM25 + vector, fused with RRF. Never vector-only.
- Transform the query — rewrite, HyDE, step-back, or decompose before retrieving.
- Rerank with a cross-encoder — retrieve ~20, rerank to ~5, send 3–5.
- Assemble context deliberately — best chunks first and last, few not many, with citations.
- Go agentic only when needed — multi-hop and high-stakes, on top of a solid base.
- Evaluate both retrieval recall and answer faithfulness — on a set hard enough to fail.
The One Line to Remember
When RAG gives a bad answer, suspect retrieval first — it's the culprit far more often than the model.
The instinct to reach for a bigger model is almost always wrong. The bigger model will summarize the wrong chunks just as fluently as the small one did. The leverage is upstream — in finding the right chunk, making it usable, ranking it where the model will see it, and being able to tell when any link in the chain breaks.
I learned this checklist one confidently-wrong answer at a time. You don't have to.
What's the retrieval bug that fooled you the longest? Mine was a chunking issue I spent two weeks blaming the model for. Share yours — and any checklist items I missed — in the comments.
Top comments (0)