Follow a wrong answer through sources, retrieval, context and generation, and learn how to isolate the failing stage.
Originally published on Stack & Craft.
What you will learn: Use a repeatable investigation to distinguish missing facts, search failures, context assembly mistakes and unsupported generation.
Prerequisites: Start with the Building and Evaluating a RAG System guide. Worked exercises use Python’s standard library and need no paid model.
Start with a reproducible case
Save the exact question, expected answer and source version. Keep a minimal reproduction with safe data. Do not copy customer secrets into public debugging logs.
Expected evidence
Write the passage that should support the answer. If the collection has no such passage, record that the question is unanswerable.
For the fictional Atlas annual plan, v2 says 14 days.
Observed output
Record the actual response and the evidence IDs it cited. Separate the generated text from the search results.
The answer says 30 days, with a citation to an old policy.
Configuration
Record index version, retrieval settings, model version and prompt template. Otherwise two apparently identical tests may use different systems.
A cached answer can hide an index refresh.
Inspect the source first
Read the source of truth. Confirm its date, audience and applicability. Then inspect the extracted text and indexed passage. A current PDF does not help if the index still contains an old version.
Source failure
If the current policy is missing or the extractor lost “annual”, repair ingestion or source selection. The generator cannot reliably recover evidence it never received.
Inspect retrieval and context separately
Search can find the right document while the application sends the wrong fragment to the model. Trace both stages. Preserve permissions through filtering and do not rely on the model to remove restricted content.
Four questions for one wrong answer
- Source — Does the trusted collection contain the current applicable fact, correctly extracted?
- Retrieval — Do the returned candidate passages include that fact? Inspect IDs, ranks and passage text.
- Context — Was the relevant passage actually included in the request? Check truncation, deduplication, ordering and source labels.
- Answer — Does each important claim follow from those passages? Check source references and handling of missing information.
Map symptoms to checks
Treat these as hypotheses to investigate, not automatic diagnoses. More than one failure can be present.
| Situation | What it means | Next check |
|---|---|---|
| No passage contains the expected fact | The source may be absent, outdated or poorly extracted. | Inspect raw and extracted sources. |
| Right fact exists but is not retrieved | Search, chunking or filtering may be responsible. | Run a known query and inspect candidates. |
| Right passage retrieved but absent from model request | Context assembly may drop or truncate it. | Inspect the final safe request payload. |
| Evidence says 14 days; answer says 30 | Generation, a stale cache or conflicting context may be responsible. | Trace the exact request, response and cache keys. |
| Right answer, broken citation | Source mapping or citation rendering may be wrong. | Resolve the displayed reference to its actual passage. |
Run an evidence trace
Run exercises/debugging_demo.py in the starter. It intentionally keeps a stale index after changing a temporary policy, then rebuilds it. The output shows the source text and the passage returned by search before and after rebuilding. It never changes the project’s sample documents.
Expected conclusion
A new source can coexist with an old in-memory index. Rebuilding replaces that index in this exercise. In a larger system, verify background jobs, caches and deletion handling too. This experiment does not prove a production model will answer correctly.
Change one variable and rerun
After identifying a likely cause, compare one controlled change against the same evaluation set. A prompt rewrite that fixes one example may break other cases.
Keep a failure record
Record question, expected evidence, failing stage, proposed change, measured result and regression checks. Keep the old baseline until the new behavior has been verified.
A practical workflow
- Capture a safe reproduction and expected evidence.
- Trace source → retrieval → context → answer.
- Identify the smallest change likely to address the failure.
- Recheck the failing case and the held-out evaluation set.
Practice
Write a debugging record for a stale policy answer and a missing-citation answer. Name the evidence that would confirm or disprove your hypothesis in each case.
Check your understanding
Should every wrong answer trigger a prompt rewrite?
No. First locate the failure. Source, retrieval, context or caching problems may happen before generation.
Does retrieving the right document prove the model saw its relevant passage?
No. Context assembly may omit, truncate or mislabel it.
Is one corrected answer enough to prove improvement?
No. Recheck representative cases and regressions under controlled settings.
Sources
- Original RAG research — Lewis et al.: Research background on combining retrieval and generation. Examples and exercises in this guide are original teaching material.
Continue learning
Read the original guide on Stack & Craft for the illustrated version and related lessons.
The Local RAG Document Assistant includes the companion exercises.
Top comments (0)