The latest articles on DEV Community by mofuteq (@mofuteq). https://dev.to/mofuteq https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3986517%2F1522fd82-a3f8-4372-b241-0750977f2488.JPG https://dev.to/mofuteq en mofuteq Fri, 19 Jun 2026 12:35:09 +0000 https://dev.to/mofuteq/your-rag-retrieved-the-right-documents-but-still-gave-the-wrong-answer-5fdo https://dev.to/mofuteq/your-rag-retrieved-the-right-documents-but-still-gave-the-wrong-answer-5fdo <p>Your retriever returned the right documents. The similarity scores look fine. The answer is still wrong. If you've shipped RAG, you've seen this — and it's the failure that survives every retrieval upgrade.</p> <h2> What everyone tries </h2> <p>Reranker. Higher top-k. Hybrid search. A better embedding model. All of these chase the same goal: <em>documents more similar to the query.</em> They help when the right document wasn't being retrieved. They do nothing when the right document <strong>was</strong> retrieved and the answer is still wrong.</p> <h2> Why it doesn't work </h2> <p>Similarity answers "is this chunk about the same topic?" It does not answer "does this chunk contain the facts needed to support the answer?" Those come apart constantly. A chunk can be highly similar — same vocabulary, same subject — and contain nothing that actually grounds the answer. Hand the model a pile of on-topic text and it will produce a fluent, plausible, even cited-looking answer. The grounding is cosmetic: the text was nearby, not load-bearing.</p> <p>High similarity with a wrong answer isn't a contradiction. You asked retrieval to find related text. It did. Nobody asked whether the text was <em>enough.</em></p> <h2> The one shift </h2> <p>Stop treating retrieval output as evidence. Treat it as candidate material that has to pass an explicit evidence check before it can support an answer. Put a step between retrieval and generation: <em>does the retrieved set actually contain the facts this answer requires? If not, abstain.</em> When the documents don't contain the facts, the system should return nothing rather than a confident guess.</p> <p>Relevant context in, only sufficient evidence allowed through. That's the line between a RAG demo and a RAG system you can trust in production.</p> <p>I write about the three boundaries where production RAG dies — query, evidence, output — from the angle of shipping under security and model constraints. <a href="https://blog.mofuteq.space/your-rag-retrieved-documents-not-evidence" rel="noopener noreferrer">Read the full version on my blog</a>, where this connects to the practical <strong>RAG Failure Diagnosis Kit</strong> for teams debugging production RAG.</p> rag llm ai machinelearning