RAG Can Retrieve the Right Document and Still Hide the Answer

A document ID can pass a retrieval check while the text sent to the model omits the answer.

That leaves a gap between two questions that often get merged during RAG debugging:

1. Did the system find the relevant source?
2. Did the model actually receive the relevant part of that source?

Here is a small, constructed example that makes the gap visible without calling a language model. It uses Python's sqlite3 module and a SQLite build with FTS5 enabled. No API key or additional Python package is needed.

One retrieved row, two different views

The fictional question is: What is the return window for Zephyr orders?

The source contains the answer, but places it after a long introduction. The query intentionally matches only the relevant document, so this example tests text delivery rather than ranking quality.

Save this as evidence_window.py and run python3 evidence_window.py:

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE docs USING fts5(doc_id UNINDEXED, body)")

policy = (
    "General shipping guidance. " * 80
    + "For Zephyr orders, the return window is 17 days after delivery."
)
db.executemany(
    "INSERT INTO docs VALUES (?, ?)",
    [
        ("returns", policy),
        ("shipping", "Standard shipping takes three to five business days."),
    ],
)

doc_id, full_text, excerpt = db.execute(
    """
    SELECT doc_id, body, snippet(docs, 1, '[', ']', ' ... ', 32)
    FROM docs
    WHERE docs MATCH ?
    ORDER BY bm25(docs)
    LIMIT 1
    """,
    ("Zephyr",),
).fetchone()

head = full_text[:180]
marker = "17 days"  # Known only because this is a constructed example.

print("Retrieved document:", doc_id)
print("Marker in full source:", marker in full_text)
print("Marker in first 180 characters:", marker in head)
print("Marker in query-centered snippet:", marker in excerpt)
print("Snippet:", excerpt)
db.close()

Observed output when run locally with Python 3.12:

Retrieved document: returns
Marker in full source: True
Marker in first 180 characters: False
Marker in query-centered snippet: True
Snippet:  ... For [Zephyr] orders, the return window is 17 days after delivery.

The query and retrieved document are identical for both views. One view is the document's first 180 characters. The other is a fragment selected around the query match.

If the check only asks whether doc_id == "returns", both look successful. If the generation step receives only head, however, the relevant source does not provide the answer in the supplied context.

A model might still guess correctly or answer from prior knowledge. That would not show that retrieval delivered the evidence.

What the SQLite calls mean

In FTS5, ORDER BY bm25(docs) sorts the better matches first because its BM25 scores are numerically smaller for stronger matches.

The 1 in snippet(docs, 1, ...) selects the second column, body. The square brackets mark query matches, and the ellipsis marks omitted text. The final 32 limits the fragment using FTS5 tokens, not the tokens of a language model. These behaviors are documented in the SQLite FTS5 reference.

The 180-character prefix and 32-token snippet are different budgets. This is not a controlled efficiency comparison between them. It is an example of information being available in a source but absent from a particular view of that source.

Keep three checks separate

Candidate retrieval: Did the returned IDs include the expected sources? This can reveal failures in queries, ranking, filtering, or candidate limits. It does not tell you which passages reached the model.

Evidence delivery: Did the assembled model input contain the facts and conditions needed to answer? Inspect the actual text after extraction, truncation, deduplication, and context assembly. Looking at the full document on disk is insufficient if the model received something smaller.

Answering: Did the model use that evidence correctly? It can receive a relevant paragraph and still omit an exception, attach a fact to the wrong entity, or cite a source that does not support its claim.

These checks need their own observations. A correct document ID does not establish evidence delivery. A delivered fact does not establish correct reasoning. A correct final answer does not prove that the supplied context supported it.

Why the substring check is deliberately weak

The example uses "17 days" in excerpt because the source and answer were constructed for this demonstration. It is a diagnostic marker, not a general correctness evaluator.

The same phrase could appear in an expired policy, a quoted customer question, or a sentence saying that a 17-day window does not apply. Conversely, an answer could express the same duration in different words and fail the string check.

For a real evaluation, inspect the supporting passage and its conditions, or use annotated evidence spans when available. Keep answer assessment separate from evidence-presence checks.

A snippet is a useful view, not a complete reading strategy

This example puts the answer next to the query term. That makes a query-centered snippet useful here. It does not show that snippets always contain the answer.

The answer might be far from the lexical match, spread across several documents, or qualified by a later exception. A system may need another search or a further read. IRCoT studies the broader problem of interleaving retrieval and reasoning for multi-step questions; this SQLite example does not evaluate that approach.

When reviewing a failing RAG trace, I would keep the source ID, source version, exact delivered text, and any location information available from the extraction step. If locations are recorded, specify their units: character offsets, byte offsets, and tokenizer positions are not interchangeable. FTS5's snippet() call above does not return a character offset.

Then check the final assembled input. A useful excerpt can still be dropped or cut again after retrieval.

What this run establishes

The local script retrieved the expected document, omitted the known marker from a prefix view, and included it in a query-centered snippet.

It did not call a model, measure answer accuracy, compare retrievers, or establish how often this failure occurs in a real corpus.

Before replacing a retriever because an answer is wrong, check the text that crossed the retrieval-to-generation boundary. The missing information may already be in the selected source.

AI disclosure: This article was drafted and edited by AI agents under the author's direction. The illustrative code was executed locally and its output checked. The documents and question are fictional; no model requests or benchmark runs were made for this article.