Imagine a customer support assistant answering a routine question: "When must a return request reach quality control?" In this invented scenario, the current company policy states that the request must arrive "within three business days after the complete request is received." An older version of the policy allowed five calendar days. If the assistant replies "within five days" or simply "within three days," the answer fails to reproduce the current rule. From the final output alone, you cannot tell whether retrieval fetched the outdated five-day policy, context assembly dropped the business-day qualifier and starting condition, or the model misread valid text. (The return policy examples throughout this article are fictional illustrations, not legal advice or results from a client test.)
In a retrieval-augmented generation (RAG) system, the application searches an external document index, passes the retrieved excerpts into the prompt as context, and asks a language model to generate a response. When the final answer fails, swapping the model is often the first instinct. A different model can sometimes help, especially with complex reasoning or strict instruction following. However, if the search step fetched an obsolete policy or a truncation script cut off the key condition, a new model will still receive broken evidence.
Before changing models, take one incorrect answer and walk through five checks to localize where the evidence failed and choose your next experiment.
Capture One Redacted Trace First
You cannot debug a RAG failure from a screenshot of a fluent but wrong answer. You need a minimal trace of what the pipeline actually saw at each step. Before storing or sharing a trace for debugging, redact personal data, customer identifiers, and API secrets, replacing them with safe synthetic values. Make sure your redaction keeps the non-sensitive policy wording and conditions intact, while noting that a redacted trace is for logical inspection rather than a byte-identical production replay.
A compact diagnostic record captures the user question, the actual search query and filters executed, the retrieved document and chunk identifiers with versions, the raw retrieved excerpts, the exact assembled context sent to the model, the final answer, and any returned citations or source mappings (or their explicit absence):
{
"question": "When must a return request reach quality control?",
"search_query": "return request reach quality control deadline",
"filters": {"status": "active"},
"retrieved_chunks": [
{
"doc_id": "pol-returns-2023",
"chunk_id": "pol-returns-2023#c04",
"version": "1.4",
"excerpt": "Requests must reach QC within five calendar days."
}
],
"prompt_context": "Source [pol-returns-2023#c04]: Requests must reach QC within five calendar days.",
"answer": "A return request must reach quality control within five calendar days.",
"citations": ["pol-returns-2023#c04"]
}
With a trace like this, you can inspect the pipeline stage by stage.
1. Did retrieval find the passage that can answer the question?
Start by comparing the authoritative source document, its parsed or indexed representation, and the retrieved_chunks in your trace. Check whether the specific passage containing "within three business days after the complete request is received" was indexed intact and selected by search.
Topical similarity alone does not guarantee good retrieval quality. A vector search might return several paragraphs about quality control and returns in general without surfacing the deadline clause. In Anthropic's guide to Contextual Retrieval, the authors note that splitting documents into isolated chunks can strip away the company name, document title, or time period needed to retrieve and interpret a passage, and that combining semantic embeddings with lexical search is an option worth evaluating when queries depend on exact identifiers or policy codes.
It also helps to separate retrieval checks from generation checks. The Amazon Bedrock evaluation metrics documentation distinguishes retrieve-only evaluation, such as context relevance and coverage, from end-to-end generation evaluation. If the right passage is missing from retrieved_chunks, your parser, ingestion pipeline, index contents, query formulation, metadata filters, and ranking logic are candidate areas to inspect before changing the generation prompt.
As a quick diagnostic check, hold the question, model, and system instructions constant, manually paste the known-good policy passage into prompt_context, and run the generation step again. If the answer improves, prioritize investigating how evidence is retrieved and assembled rather than treating that single run as proof of a single root cause. If the answer still fails, the model or prompt may be contributing, though one manual test is never a formal benchmark.
2. Is that passage current and applicable?
Sometimes search finds a passage that matches the topic, but it brings back stale documents or rules meant for a different scenario. In the sample trace above, the system retrieved pol-returns-2023, which specifies "five calendar days," instead of the current policy requiring "three business days."
When you see an outdated or mismatched answer, inspect the provenance, version metadata, effective dates, and applicability of the retrieved chunks (such as whether the rule applies to standard customer returns). Do not assume a document is current or active just because its filename contains "latest" or a filter tag looks right at a glance.
Some pipelines sync changes automatically while others require scheduled or manual runs, so verify your connector's actual ingestion logs and failure counts. For example, in Amazon Bedrock Knowledge Bases, the Amazon Bedrock data source sync documentation explains that after adding, modifying, or deleting source files, you run a sync job to process those changes incrementally and can inspect the job statistics to confirm completion.
You do not always need to delete older policies globally, since historical documents may remain in storage for audit or past-date queries. What matters is verifying that the current source finished ingestion and that your query filters or ranking rules select the version applicable to a present-day question.
3. Did the complete rule reach the model?
Even when retrieval selects the right current document, the full rule might not survive the transition from the source file to prompt_context. Compare the authoritative source text against both the excerpt in retrieved_chunks and the final prompt_context string.
Suppose the current policy splits the rule across two adjacent sentences: "A return request must reach quality control within three business days. This period begins only after the complete request is received." If a chunk boundary separates those two sentences during indexing, retrieved_chunks will only contain the first sentence. Conversely, if retrieved_chunks contains both sentences but a prompt-formatting or token-trimming script cuts the text before calling the model, the loss happened during context assembly.
Chunk boundaries, overlap settings, contextual headers, and the number of selected chunks all influence whether qualifiers stay attached to a passage. At the same time, retrieving more chunks or making chunks much larger can introduce irrelevant text that distracts the model. Rather than hunting for a universal chunk size or top-k value, check whether your chunking and prompt assembly preserve conditions, exceptions, and table headers alongside the rules they modify.
4. Does the answer follow from the supplied evidence?
If your trace confirms that the current, complete rule reached prompt_context intact, move to the generation step. Compare every claim in the answer against the supplied context. For our scenario, a well-supported answer must keep both "three business days" and "after the complete request is received."
Here it is important to distinguish between retrieval relevance, groundedness (often called faithfulness), and overall correctness, even though evaluation tools sometimes define these operational metrics differently. Retrieval metrics look at whether the fetched passages cover the needed evidence. As the Microsoft Azure RAG evaluation guide explains, groundedness checks whether the generated response is supported by the provided context, while correctness evaluates factual accuracy against a verified source of truth. If prompt_context contains the outdated five-day policy and the model repeats it accurately, the response is grounded in the supplied context but still gives the user the wrong current rule.
Also check how citations behave. A citation array in the response only shows that a source identifier was attached; citation precision and coverage measure whether the cited passage actually supports the specific claim. Because automated LLM judges can miss subtle qualifier drops and cannot certify real-world truth on their own, combine automated scores with human review of the trace to verify that the model did not drop conditions, merge distinct policies, or overstate certainty.
5. What happens when the documents cannot answer?
Finally, check how the pipeline behaves when the user asks about a case that the supplied context cannot answer, such as an undocumented exception for international hazardous-material returns.
First, remember that an empty retrieval result or missing clause in prompt_context does not automatically prove the policy is absent from your company's records. Distinguish between a true knowledge-base coverage gap and an ingestion or retrieval failure where the document exists in storage but was never indexed or fetched.
When the supplied context genuinely lacks the required fact, a reliable assistant should state what cannot be verified from the available documents, ask a clarifying question when missing user details would select between known rules, or hand the conversation off to human support. Asking the user for clarification cannot invent a policy that is not in the evidence. You can specify this fallback behavior in your system prompt, though instructions alone never guarantee that a model will abstain every time.
Whenever you tighten prompts or retrieval thresholds to reduce unsupported guessing, pair your unanswerable test cases with answerable control questions. Otherwise, you can easily push the system into over-refusal, where it replies "I don't have enough information" even when the complete three-business-day rule is present in prompt_context.
Turning One Fix into a Regression Check
Once you localize the failure, change one component at a time where feasible, such as running an index sync, adjusting chunk boundaries, or refining the prompt. To validate a fix, re-run the same questions through the updated pipeline, capture new traces, and compare them against your baseline. Replaying a saved prompt_context from an old trace only tests generation; it will not verify whether an indexing or retrieval change worked.
A small, targeted regression set helps verify that fixing one issue does not break another behavior:
| Test case type | Example scenario | What to inspect in the new trace |
|---|---|---|
| Answerable baseline control | User asks "When must a return request reach quality control?" | Does prompt_context include the current rule, and does answer preserve both "three business days" and "after the complete request is received" (plus accurate citations if used)? |
| Paraphrased question | User asks "How long do we have to get a return over to QC?" | Does retrieval still surface the current rule when the wording changes, and does answer retain all conditions? |
| Exact identifier | User asks about policy ID pol-returns-2026
|
Does your search setup locate the exact document ID and pass the matching passage into prompt_context? |
| Outdated version trap | Both 2023 (five calendar days) and current (three business days) policies exist in storage | Does the pipeline select the current applicable version for a present-day question so answer reflects the three-business-day rule? |
| Split condition | The deadline and "after the complete request is received" sit in adjacent sentences | Do retrieved_chunks and prompt_context keep both sentences together, and does answer include the starting condition? |
| Unanswerable question | User asks about an undocumented international hazardous-material exception | When the evidence is absent from prompt_context, does answer state the limitation or hand off without inventing a deadline? |
If your new traces show that retrieval, document currency, and context assembly are delivering the complete current rules, yet the model still drops qualifiers or ignores abstention instructions, you have clear evidence to justify testing a different model. You can then run this exact regression set across candidate models under comparable prompts and settings, comparing accuracy alongside any latency and cost measurements you collect.
For readers who also work in Russian and want a broader foundational walkthrough of how document retrieval, context passing, and response generation fit together, see https://timofei-ermolaev.ru/llm/chto-takoe-rag/.
Drafted and edited with Gemini, with additional AI-assisted editorial review.
Top comments (0)