Updated October 29, 2026. The taxonomy now separates unsupported factual claims from instruction, role and consistency failures. The categories are a practical review framework, not a universal standard.
Calling every rejected answer a hallucination makes diagnosis harder. An unsupported factual claim, an ignored instruction and an inconsistent conversation can require different evidence and different corrections.
Quick answer: Classify the observed failure before selecting a check. Compare facts with appropriate references, answer support with supplied context, instruction adherence with stated requirements and conversational consistency with relevant prior turns. A category helps organize investigation; it does not prove a single root cause.
Five useful categories for review
| Category | Illustrative failure | Evidence needed |
|---|---|---|
| Unsupported factual claim | The agent invents a product capability | A current authoritative reference and the claim |
| Context-support failure | An answer gives a return window absent from the supplied policy | The actual supplied policy and response |
| Instruction-adherence failure | The agent performs an action outside its assigned scope | Relevant instructions and captured action |
| Role or style failure | A technical assistant abandons a required response style | Task, role requirements and a reviewed rubric |
| Conversational inconsistency | The agent makes incompatible claims in relevant turns | Those turns and their surrounding context |
Instruction and style failures are not necessarily factual hallucinations. Conversely, an answer can follow the assigned style while inventing a fact.
Support in context is not independent truth
Suppose a document states that returns are accepted within 14 days. An answer saying 30 days contradicts the supplied evidence. A response repeating 14 days may be faithful to the document, but that does not prove the document is current or applies to the requested product.
Use different checks for different questions. A system prompt defines expected behavior. It does not establish a factual reference. A check without an expected answer can still need correctly mapped source context or task fields.
Investigate the point where the evidence changes
For the policy example, inspect the requested product, the retrieved source and the material delivered to generation. If the wrong product’s policy was retrieved, a prompt-only change may leave the underlying issue untouched.
If the source was correct but truncated, inspect preparation and context selection. If adequate context arrived but the answer departed from it, test candidate changes to the response behavior. These are hypotheses to compare, not diagnoses established by the final answer alone.
Keep a regression case for the correction
Test a relevant source, an irrelevant source, a missing source and an ambiguous request. Define the intended behavior for each. Review some passing cases to find errors a scorer missed.
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. My hallucination investigation guide develops the evidence workflow; the eval-driven development guide explains how to keep the cases useful over time.
Follow my work on LinkedIn. This refresh was drafted with AI assistance.
Top comments (0)