A retrieval-augmented generation system can return fluent answers from the wrong evidence. RAG connects a language model to retrieved documents before it answers a question. Think of it as handing a researcher a folder: a confident summary is only useful if the folder contains the right material.
This is an abridged adaptation of RAG Works in Theory. Here's Why It Fails in Production, originally published on TheQuery by Addy.
Check the evidence before changing the model
When a generated answer is wrong, inspect the retrieved passages alongside the question. Do they actually contain the information needed to answer it? Are they from the right customer, product version, contract, or period?
A passage can be about the right topic and still answer a different question. A contract clause about termination may be irrelevant if it belongs to another agreement. Giving a stronger model that same passage does not correct the document mismatch.
Save representative questions and the evidence they require. Include difficult cases: exact identifiers, old and new policy versions, conflicting documents, and questions that the collection cannot answer. This gives you something concrete to compare when changing the retrieval pipeline.
Treat chunk boundaries as part of the meaning
Chunking divides documents into pieces small enough to retrieve and send to a model. Cutting at a fixed token count is like slicing a manual every few inches: an instruction may end up separated from the condition that makes it apply.
Check whether a retrieved piece preserves its heading, subject, exceptions, and relevant metadata. A paragraph that says “this is permitted” is not useful evidence if the previous paragraph explains who “this” applies to.
Document-aware splitting can preserve useful boundaries. It still needs evaluation on your documents. There is no universal chunk size that establishes relevance by itself.
Similarity is a candidate signal
Embedding search represents text numerically so related passages can be retrieved. It resembles finding books on the same shelf: proximity suggests a relationship, but does not establish that a book answers a particular question.
Exact strings can require a different retrieval signal. A section number, error code, or product identifier may be the decisive part of the query. Hybrid retrieval combines semantic search with keyword matching, much like consulting both a library's subject catalog and its alphabetical index.
A reranker can then compare candidate passages more closely with the question. Think of it as a second reader inspecting the shortlist. Test whether it improves retrieval on the difficult cases; adding another stage also adds latency and cost.
Evaluate retrieval and answers separately
A single end-to-end score can hide where the failure occurred. Track whether retrieval found the required evidence, then whether the generated answer stayed supported by that evidence. An answer should be able to acknowledge when the supplied documents do not establish a conclusion.
Inspect the final context sent to the model, not just the search results. Selection and assembly can remove qualifiers or combine contradictory passages. More retrieved text is not automatically better evidence.
The full guide on TheQuery covers context assembly and production failure modes in more detail. Start with the documents the system actually retrieved. The answer cannot repair evidence it never received.
Top comments (1)
"Check the evidence before changing the model" is the rule I'd tattoo on every RAG project's README. In practice the failure modes need sorting first, because each has a different fix: wrong chunk entirely (query understanding), right topic wrong neighbor (chunk boundaries — the manual-slicing analogy is exactly right), right document wrong version (metadata). People reflexively reach for a bigger model or a re-embed when the actual bug is a stale index or a chunk that lost the paragraph that qualifies it.
The versioning point deserves more weight than it usually gets. Storing source URL and fetch date on every chunk sounds bureaucratic until the day the index answers from last year's policy doc while the corpus has been updated for months. In my experience stale-index drift produces more confident wrong answers than embedding quality ever does — precisely because the retrieval genuinely is relevant to the question.
Do you take a position on rerankers in the longer guide? I've been treating them as the cheap middle signal: a cross-encoder over the top-k candidates catches a lot of the "relevant but not answering" cases hybrid retrieval alone misses, without touching the embedding layer.