Where in Your Pipeline Should Hallucination Detection Actually Live?
Ask ten teams how they detect hallucinations and most will describe a test suite they ran before launch. Ask them how they'd catch a hallucination happening right now, in a live session, before it reaches the user, and the room usually gets quieter. That gap, between pre-launch testing and runtime detection, is where most of the hallucinations that actually cause damage slip through.
Detection isn't one technique. It's a set of techniques that belong at different stages of the pipeline, and the mistake I see constantly is teams investing everything into one stage (usually pre-launch testing, because it's the most familiar) while leaving the other two nearly empty. A system can pass every offline test and still hallucinate its way through a live customer conversation, because offline testing and runtime behavior aren't the same problem.
I think about detection across three stages: before deployment, during live inference, and after the fact in production monitoring. Each one catches things the others structurally cannot.
Stage 1: Pre-Deployment Detection
This is the stage most teams already have some version of, and it's where you build the evaluation infrastructure everything else depends on.
The core technique here is reference-based evaluation: run a curated set of test queries through the system, compare outputs against known-correct answers, and flag divergence. For RAG-based systems, pair this with source-grounding verification, checking whether each claim in the output can be traced back to retrieved context rather than the model's own training data. An answer that can't be traced to a source, even if it happens to be factually correct, is a signal worth flagging, because you're relying on something you can't consistently reproduce.
Adversarial test design matters more here than most teams initially budget for. Easy questions with well-covered answers won't surface hallucinations, because that's exactly the condition under which models perform best. Build test cases specifically designed to be ambiguous, to sit outside the knowledge base's coverage, or to invite the model into territory where a plausible-sounding wrong answer is genuinely tempting to generate.
What this stage catches well: systemic issues in prompt design, retrieval configuration, and model behavior on known failure categories.
What it structurally can't catch: anything specific to the long tail of real user phrasing, multi-turn conversation drift, or edge cases nobody thought to write a test for. Pre-deployment testing is bounded by the imagination of whoever wrote the test set, and real users are more creative than test authors, every time.
Stage 2: Runtime Detection
This is the stage most enterprise deployments skip entirely, usually because it's harder to build and there's no obvious off-the-shelf tool that does it well for every use case. It's also the stage that catches hallucinations before a user ever sees them, which makes it worth the extra effort.
A few techniques that work here in practice:
Confidence and uncertainty signals. Some model providers expose token-level probability data, and low-confidence spans in a generated answer correlate reasonably well with hallucination risk, though not perfectly. This isn't a standalone detector, but it's a useful signal to route uncertain outputs toward additional checks or human review before they reach a user.
Real-time groundedness checks. For RAG systems specifically, you can run a lightweight verification pass at inference time, checking whether the generated answer's claims align with the retrieved context, before the answer is returned. This adds latency, so it needs to be scoped carefully (more on that trade-off below), but for high-stakes applications it's often worth the cost.
Guardrail rules for known-risky patterns. If certain query types have historically produced hallucinations (a specific product category, a certain kind of numeric question, requests outside the system's documented scope), route those queries through stricter validation or a fallback response instead of a direct model answer.
A callout worth remembering: runtime detection isn't about catching every hallucination. It's about catching the ones with the highest cost if they reach a user, and doing it fast enough not to break the experience. Trying to catch everything at this stage usually means adding so much latency that the feature becomes unusable.
Stage 3: Post-Production Monitoring
This stage answers a different question than the first two: not "did this specific answer hallucinate," but "is the system's hallucination rate changing over time, and why."
The core practice here is sampling and reviewing live production outputs on an ongoing basis, not just when something breaks. A useful pattern: run a percentage of real production queries through your groundedness and reference-based checks after the fact, on a rolling basis, and track the trend. A sudden shift, more flagged outputs this week than last, is often your earliest signal of model drift, a knowledge base update that degraded retrieval quality, or a prompt change that shipped without anyone connecting it to the regression.
This stage is also where user feedback signals matter, when you have them: corrections, complaints, or explicit "this was wrong" flags. These are noisy and incomplete on their own, most users don't report a bad answer, they just stop trusting the system, but combined with sampled automated review, they help validate whether your detection methods are actually catching what matters to real users.
What this stage catches well: drift, degradation from upstream changes, and failure patterns that only show up at real production scale and query diversity.
What it structurally can't catch: anything before the fact. This is detection after the user has already seen the answer, which makes it a monitoring and improvement loop, not a prevention mechanism.
The Trade-Off Nobody Likes Talking About
Runtime detection adds latency, and for some applications, that trade-off genuinely isn't worth it. A lightweight groundedness check might add a few hundred milliseconds. For a real-time chat interface, that's often acceptable. For a high-throughput batch process generating thousands of summaries, it can meaningfully change the economics of running the system at all.
There's no universal answer here, and I'd be skeptical of anyone who gives you one. The right call depends on the cost of a hallucination reaching a user versus the cost of the added latency or compute, and that's a business decision as much as a technical one. What I'd push back on is skipping runtime detection entirely for high-stakes use cases just because it's the harder stage to build. That's usually a decision made under deadline pressure that gets revisited after the first serious incident, and by then the fix costs more than it would have upfront.
Common Mistakes Across All Three Stages
Building Stage 1 thoroughly and stopping there. This is the most common gap I see, and it's understandable, pre-deployment testing is the most mature, best-documented part of the process. It's also the stage least connected to what actually happens once real users start interacting with the system.
Treating post-production monitoring as incident response instead of a routine practice. Waiting for a complaint before sampling production outputs means you're always finding out about drift after it's already affected real users. Rolling, scheduled review catches it earlier.
Assuming one detection method covers all three stages. A reference-based evaluation set built for pre-deployment testing doesn't translate directly into a runtime check, the latency budget and the available context are completely different. Each stage needs its own tooling, even when the underlying detection logic overlaps.
No clear ownership of the runtime and monitoring stages. Pre-deployment testing usually has an obvious owner: whoever built the test suite before launch. Runtime detection and post-production monitoring often don't, and undefined ownership is exactly how a detection gap survives for months without anyone noticing it's there.
A Starting Checklist
- Pre-deployment evaluation set includes adversarial and out-of-scope test cases, not just well-covered questions
- Source-grounding verification is in place for any RAG-based system
- At least one runtime detection method exists for high-stakes query paths, even if it's a simple guardrail rule
- Production outputs are sampled and reviewed on a recurring schedule, not only after an incident
- A regression trigger connects knowledge base and prompt changes to renewed detection testing
- Ownership is assigned for each of the three stages individually, not just for "hallucination testing" as one undifferentiated task
My Honest Take
Most of the hallucination incidents I've been called in to review after the fact weren't failures of pre-deployment testing. The test suite had been fine. They were failures of never building the other two stages at all, so nothing caught the problem between launch and the moment a customer or a compliance reviewer did. If I had to guess where the next wave of AI governance requirements lands, it's here: not demanding better pre-launch test coverage, which most enterprise teams already have some version of, but demanding evidence of ongoing runtime and production-level detection, because that's the gap regulators and customers alike are going to notice first.
At PrimeQA Solutions, this three-stage approach shapes how we scope AI Model Testing for clients moving from pilot to production, because a strong pre-launch report has never been the thing that prevented a real incident. The stages after launch are where that actually happens.
Top comments (0)