Hallucination Testing: Check What AI Does When the Answer Is Missing

Kartik Sharma — Thu, 03 Sep 2026 09:32:57 +0000

Hallucination testing becomes more useful when the system is tested with questions it cannot confidently answer from the available information.

A normal AI Testing flow may check whether generated responses are correct. Hallucination testing goes further by checking whether the model invents facts, sources, names, numbers, or explanations when evidence is missing.

Create situations where the AI should say "I don't know"

Suppose an AI assistant can access three internal policy documents.
Ask:

What percentage of customers requested refunds last quarter?

If the documents do not contain that number, the system should not create one.

Depending on the product requirements, acceptable behaviour may include:

stating that the information is unavailable
asking for another source
explaining what information is available
clearly marking uncertainty

A confident but unsupported percentage should be treated as a failure.

Test fabricated entities

Another useful approach is to introduce believable but nonexistent information.

Examples include:

fake product names
fictional employees
nonexistent research papers
made-up API methods
invented company policies

Ask the AI to explain or reference them.

The test checks whether the system challenges the false assumption or begins building an answer around something that does not exist.

Verify citations separately

If the AI provides citations, do not assume the response is grounded simply because links or references appear.

Check:

Does the source exist?
Does it contain the claimed information?
Does the cited section actually support the statement?

A real source attached to an unsupported claim is still a problem.

Repeat important scenarios

Generative systems may behave differently across repeated runs.

A model may correctly refuse to invent information several times and then fabricate an answer later.

Run important hallucination cases repeatedly, especially for financial, healthcare, security, or business-critical workflows.

Hallucination testing is only one part of evaluating AI behaviour. Teams building broader coverage can use the full AI Testing Checklist to connect these checks with factual accuracy, RAG testing, context handling, regression, security, and response quality.

What We Learned Running Managed QA Across 3 Different Industries

Jayesh Jain — Wed, 12 Aug 2026 12:29:57 +0000

Over the last few years my team has owned the QA function for an edtech platform, a food delivery app, and a non-profit system. Same service model each time. Wildly different problems.

Here's what actually transferred between them, and what didn't.

The setup

In all three cases we ran managed QA, meaning we owned the whole testing function, not just supplied testers. Test strategy, the team, execution, environments, and the SLAs we were held to. Ownership sat with us, not the client.

That part stayed constant. Almost nothing else did.

Edtech: the problem was scale, not features

The edtech platform served 5,000 concurrent users, and load was the thing that broke it. Functional bugs mattered, but performance under real concurrency mattered more, because a slow platform during peak class hours is a churned customer.

What we learned:

Load testing isn't a phase, it's a heartbeat. We ran it continuously, not once before release. The bottlenecks that mattered only showed up under sustained concurrency, not in a one-off test.
Response time is a feature. After isolating the bottlenecks, response time dropped 65%. Nobody wrote a "make it 65% faster" ticket. It came out of treating performance as a first-class test target.
We caught 150+ critical bugs before the first production deploy, and the ones that would have hurt most were all concurrency-related, not logic errors.

Food delivery: the problem lived in the APIs

The food delivery app's pain was different. The user-facing flow looked fine. The APIs underneath it were slow, and in a delivery app latency compounds across every order, driver update, and status change.

What we learned:

Test where the latency compounds, not where it's visible. The UI wasn't the problem. Targeted API performance testing was, and it drove a 60% latency reduction.
Coverage numbers only mean something when they're on the right surface. We pushed test coverage to 95%, but the number mattered because it was concentrated on the ordering and delivery paths, not spread thin across the whole app.
Mobile plus API is a different testing discipline than web. Treating it like web testing would have missed the defects that actually cost money.

Non-profit: the problem was permissions

The non-profit platform had almost no performance pressure. Its risk was entirely in access control. 900 role-based permission configurations, and every wrong combination was a data exposure waiting to happen.

What we learned:

Some systems are all edge cases. There was no "happy path" worth much here. The whole risk surface was permission combinations, so that's where the entire effort went.
Systematic beats clever. We tested all 900 configurations methodically and resolved every one before launch. No shortcut would have been safe.
Low-traffic doesn't mean low-risk. This was the lowest-load system of the three and by far the highest-stakes per defect.

What actually transferred

Three engagements, three completely different failure modes. The thing that carried across all of them wasn't a tool or a test type. It was owning the risk model.

The risk lives somewhere different every time. Scale, then APIs, then permissions. Walking in with a fixed checklist would have missed the real threat in at least two of the three.
Match the testing to the risk, not to a template. The edtech playbook would have been useless for the non-profit. The value was figuring out where each system actually breaks, then pointing everything there.
Ownership is what makes that possible. When you own the outcome instead of a task list, you're free to put the effort where the risk is. That's the whole argument for managed QA over just renting testers.

If you're weighing managed QA against staff augmentation or an in-house build, that ownership question is the one that decides whether it works. I wrote up the full breakdown of the models, the SLAs worth demanding, and when to skip managed QA entirely in a complete guide to managed QA services.

DEV Community: Testscenario

Hallucination Testing: Check What AI Does When the Answer Is Missing

Create situations where the AI should say "I don't know"

Test fabricated entities

Verify citations separately

Repeat important scenarios

What We Learned Running Managed QA Across 3 Different Industries

The setup

Edtech: the problem was scale, not features

Food delivery: the problem lived in the APIs

Non-profit: the problem was permissions

What actually transferred