A retrieval-augmented generation system can return a fluent answer to the wrong question, cite an outdated document, or invent a detail that its sources never supplied. Those failures can look similar in the chat window. They need different fixes.
Before changing the model, follow one request through three checkpoints: indexed evidence, retrieved evidence, and context actually delivered to the generator.
A small example
Suppose an internal assistant is asked:
How long are application audit logs retained?
Imagine two documents in its knowledge base. An older handbook says 30 days; a current policy says 90 days for application audit logs, with separate rules for other categories. The assistant answers “30 days.” These numbers are illustrative policy values, not retention recommendations.
A stronger model might still repeat the old policy if that is the only evidence it receives. First determine where the current information disappeared.
1. Check the evidence before the search
Locate the current source document. Confirm that ingestion succeeded and inspect the extracted text, not just the original PDF or page.
- Does the extracted text include the 90-day rule?
- Is “application audit logs” still attached to that rule?
- Are headings, tables or exceptions split in a way that changes the meaning?
- Can each chunk be traced to a document version and section?
A successfully uploaded file is not proof that the right passage is searchable. If extraction lost the table containing the answer, changing a generation prompt will not recover it.
2. Inspect retrieval separately
For the exact failing question, inspect the returned passages and their identifiers before generating an answer.
If only the old handbook appears, investigate collection selection, metadata filters, query wording, chunk boundaries and any reranking step.
Do not assume that increasing the number of results will solve the problem. It may introduce more obsolete or irrelevant passages. Compare the actual passages returned before and after each change.
Version metadata is useful only when your application applies it correctly. A recent ingestion timestamp is not necessarily a policy's effective date.
3. Inspect the assembled context
The search results are not always the same as the model input. A later stage may drop, shorten, reorder or summarize passages.
| Checkpoint | What to inspect |
|---|---|
| Retrieval output | Was the current passage returned? |
| Context assembly | Did it survive filtering and the context budget? |
| Generator input | Did the model receive the rule, category and effective version together? |
If retrieval returns the current policy but prompt assembly omits it, the immediate problem is between retrieval and generation.
Use access-controlled traces. Avoid copying confidential passages or personal information into unrestricted logs.
4. Evaluate the answer against its evidence
Once the correct passage reaches the generator, inspect what the answer does with it.
“All logs are retained for 90 days” is still incorrect in this example: it expands a category-specific rule into a universal statement.
Check whether each important claim is supported by the supplied passage. Inspect citations directly; a citation marker alone does not establish that a source supports the sentence.
For conflicting or insufficient evidence, define how the assistant should respond. It might explain the conflict, ask which log category the user means, or say it cannot determine the answer. A fallback instruction defines intended behavior; it does not guarantee compliance.
5. Keep a small regression set
Save representative questions with expected evidence and answer requirements. Include:
- A straightforward answerable question.
- A question affected by an old and a current document.
- A question with an important exception.
- A question the available documents cannot answer.
Record retrieval quality and answer support separately. Correct-looking answers can hide poor retrieval, while relevant passages can still produce unsupported answers.
Change one component at a time and compare the same cases. A useful experiment might adjust a version filter, a chunking rule or context assembly. Keep other settings stable enough to interpret the difference.
A practical debugging order
Start with source availability, then extraction and indexing, retrieval, context assembly, and finally generation. This is a diagnostic order, not a rule that every failure belongs to exactly one stage.
Identify the first point where the evidence stops supporting the requested answer. That gives you a concrete change to test instead of another blind model swap.
Further reading
- LangChain: Evaluate a RAG application — evaluation dimensions and dataset-based comparisons.
- NVIDIA RAG Blueprint: Pipeline debugging guide — component-level troubleshooting for that implementation.
Top comments (1)
The extraction checkpoint is the one I'd underline. We hit the web-page version of your lost table: the HTML-to-text step kept the nav and dropped a table that only appears after JavaScript runs, so the passage was never indexed and every later check looked fine. Diffing the extracted text against what a browser shows, for a handful of failing questions, caught it fast. Do you keep the extracted text alongside each chunk version so you can rerun step 1 after a re-ingest?