A RAG demo almost never fails. The questions in a demo are written by the same person who indexed the documents, in the same vocabulary those documents use, about content everyone in the room already knows is in there.
Production is a different problem. Real users ask sideways, in their own words, about documents they have never seen. Roughly 30 percent of those queries come back with an answer that is fluent, confident, and built on the wrong passage. Nothing errors. Latency looks normal. The dashboard stays green.
That gap is not a model problem. It is a retrieval problem, and it comes in a small number of specific shapes.
The Four Failure Modes
The first is the wrong chunk. The retriever returns something about the right topic that does not contain the answer, and the model writes around it.
The second is scatter. The answer exists, but it is spread across three documents, and top-k retrieval hands over one of them.
The third is the missing check. Nothing between retrieval and generation ever asks whether the retrieved text supports an answer at all, so an empty result and a perfect result take the same path to the user.
The fourth is amnesia. The same question fails the same way forty times and the system learns nothing from any of it.
Each of these has a fix, and the fixes are independent, which is good news if you are trying to decide what to do first.
Similarity Is Not An Answer Check
Cosine similarity between embeddings measures vocabulary and topic overlap. That is all it measures. A paragraph of background about your refund policy and the paragraph that states the actual refund window are neighbors in embedding space, because they talk about the same thing in the same words.
This is why hybrid search helps. Keyword matching catches the exact identifiers, product codes, and version numbers that embeddings smooth over, and the two together reach documents neither one finds alone.
Reranking helps more. A cross encoder reads the query and a candidate passage together and scores whether that passage answers that query, which is a different question from whether they are about the same subject. Running it over the top 50 from the first pass and keeping the best 5 is usually the single largest quality jump available for the cost.
The full breakdown of next generation retrieval walks through where each of these sits in a production pipeline, including the scoring signals that go beyond similarity.
Adding A Verification Layer
The cheapest reliability win in RAG is a grader that runs after retrieval and before generation, and answers one question: does this context support an answer to this query.
If it does not, the system has options that are all better than guessing. It can rewrite the query and search again. It can widen to a different index. It can tell the user it does not have that information, which sounds like a failure and is actually the feature. An assistant that says "I do not have this documented" is trusted the second time. An assistant that invents a plausible refund window is not.
The reason this layer matters more than it looks is that a wrong answer and a right answer are indistinguishable from the outside. Both are fluent. Both cite something. The only place the difference is visible is between retrieval and generation, and if nothing is looking there, nothing catches it.
When Retrieval Becomes Memory
A stateless retriever treats every query as the first one it has ever seen. It cannot know that this document has answered forty questions well and that one has never been useful. It cannot know that the policy it just returned was superseded in March.
Once you start scoring passages on how often they have actually helped, how recently they were confirmed, and how authoritative their source is, retrieval stops being a search index and starts behaving like memory. Usage becomes a signal. Staleness becomes a penalty. The system gets better at the questions it gets asked most, which is exactly the behavior you want and exactly what pure similarity search cannot do.
What To Fix First
If you have a pipeline that is quietly wrong a third of the time, the order that pays off fastest is: add reranking, add a grader that is allowed to say no, then add scoring signals beyond similarity.
None of that requires a bigger model. All of it requires accepting that the retrieval step, not the generation step, is where the answer was decided.
Top comments (0)