How Do You Test AI That Never Gives the Same Answer Twice?

Traditional testing rests on an assumption nobody states out loud: same input, same output. assertEqual(add(2,2), 4). LLMs break that assumption on purpose — they're non-deterministic, and the "correct" answer is a fuzzy region, not a value. So how do you test a system that doesn't repeat itself, especially when it's answering questions about someone's insurance coverage?

The scariest part: traditional bugs throw. LLM failures look like confident, well-written, incorrect answers. Nothing crashes. Your test suite is green and the model just told a customer their flood damage is covered when it isn't.

Here's what actually works.

1. Stop asserting exact strings

assertEqual(output, "Your excess is £250") fails the moment the model says "Your excess is 250 pounds." Test properties and facts, not phrasing:

result = agent.answer("What's my excess for water damage?")
assert "250" in extract_amounts(result)          # the fact is present
assert not contradicts(result, policy.facts)     # nothing false asserted
assert cites_source(result)                        # grounded, not invented

2. Build an eval set, treat it like a test suite

Curate a golden dataset of input → acceptable-output-criteria pairs, versioned in the repo. Run it on every prompt/model change. This is your regression suite for behavior. When a "prompt improvement" silently breaks 8% of cases, the eval catches it — the same way a unit test catches a refactor gone wrong.

3. LLM-as-judge for the fuzzy middle

For open-ended answers, use a separate model to grade against a rubric ("Is the answer factually consistent with these policy facts? Does it avoid giving advice it shouldn't?"). Not perfect, but it scales judgment across thousands of cases. Keep a human-labeled slice to validate the judge itself.

4. Adversarial and guardrail tests

The failure modes that matter in insurance aren't "slightly wrong wording" — they're:

• Hallucinated coverage. Feed it edge cases and assert it says "I don't know / escalate" instead of inventing.
• Prompt injection. Include hostile documents in your test set; assert the model doesn't act on embedded instructions.
• Refusal boundaries. Assert it refuses to give regulated advice, quote binding prices, etc.

@pytest.mark.parametrize("doc", INJECTION_CORPUS)
def test_ignores_embedded_instructions(doc):
    out = agent.process(doc)
    assert not took_privileged_action(out)   # reading ≠ acting

5. Test the system, not just the model

Most production LLM failures are retrieval failures, not generation failures — the model answered fine given bad context. So test the pipeline: did retrieval surface the exclusion and not just the grant of cover? Is the context complete? A perfect model on wrong context is confidently wrong.

The mindset shift

You're not testing for the answer; you're testing that the output stays inside a boundary of acceptable, grounded, non-harmful responses — statistically, over an eval set, with adversarial cases and a human-checked slice. It's closer to fuzzing and monitoring than to assertEqual.

Full write-up with the insurance framing (why confident-and-wrong is the real risk):

How Do You Test AI That Doesn't Give the Same Answer Twice? →

From IntelliBooks' series on the data foundation under insurance AI.

How's your team testing LLM features — eval sets, LLM-judge, something else? What caught your worst confident-wrong bug?