RAG Retrieval Can Look Relevant and Still Be Wrong

Addy | TheQuery — Tue, 08 Sep 2026 20:50:39 +0000

A retrieval-augmented generation system can return fluent answers from the wrong evidence. RAG connects a language model to retrieved documents before it answers a question. Think of it as handing a researcher a folder: a confident summary is only useful if the folder contains the right material.

This is an abridged adaptation of RAG Works in Theory. Here's Why It Fails in Production, originally published on TheQuery by Addy.

Check the evidence before changing the model

When a generated answer is wrong, inspect the retrieved passages alongside the question. Do they actually contain the information needed to answer it? Are they from the right customer, product version, contract, or period?

A passage can be about the right topic and still answer a different question. A contract clause about termination may be irrelevant if it belongs to another agreement. Giving a stronger model that same passage does not correct the document mismatch.

Save representative questions and the evidence they require. Include difficult cases: exact identifiers, old and new policy versions, conflicting documents, and questions that the collection cannot answer. This gives you something concrete to compare when changing the retrieval pipeline.

Treat chunk boundaries as part of the meaning

Chunking divides documents into pieces small enough to retrieve and send to a model. Cutting at a fixed token count is like slicing a manual every few inches: an instruction may end up separated from the condition that makes it apply.

Check whether a retrieved piece preserves its heading, subject, exceptions, and relevant metadata. A paragraph that says “this is permitted” is not useful evidence if the previous paragraph explains who “this” applies to.

Document-aware splitting can preserve useful boundaries. It still needs evaluation on your documents. There is no universal chunk size that establishes relevance by itself.

Similarity is a candidate signal

Embedding search represents text numerically so related passages can be retrieved. It resembles finding books on the same shelf: proximity suggests a relationship, but does not establish that a book answers a particular question.

Exact strings can require a different retrieval signal. A section number, error code, or product identifier may be the decisive part of the query. Hybrid retrieval combines semantic search with keyword matching, much like consulting both a library's subject catalog and its alphabetical index.

A reranker can then compare candidate passages more closely with the question. Think of it as a second reader inspecting the shortlist. Test whether it improves retrieval on the difficult cases; adding another stage also adds latency and cost.

Evaluate retrieval and answers separately

A single end-to-end score can hide where the failure occurred. Track whether retrieval found the required evidence, then whether the generated answer stayed supported by that evidence. An answer should be able to acknowledge when the supplied documents do not establish a conclusion.

Inspect the final context sent to the model, not just the search results. Selection and assembly can remove qualifiers or combine contradictory passages. More retrieved text is not automatically better evidence.

The full guide on TheQuery covers context assembly and production failure modes in more detail. Start with the documents the system actually retrieved. The answer cannot repair evidence it never received.

DEV Community: Addy | TheQuery

RAG Retrieval Can Look Relevant and Still Be Wrong

Check the evidence before changing the model

Treat chunk boundaries as part of the meaning

Similarity is a candidate signal

Evaluate retrieval and answers separately