How Do You Test an AI Agent That Gives a Different Answer Every Time?

You spent three days building an AI agent that rewrites user emails in a friendlier tone. It works beautifully — on Tuesday. On Wednesday, the same input produces something weirdly formal. On Thursday, it apologizes for no reason and signs off as "Warmly, Your AI."

You stare at your test suite. Every assertion is red. But the agent isn't broken. It's just… different today.

Welcome to the most maddening problem in agentic AI: how do you test something that gives a different answer every time you ask?

Why Traditional Tests Fail Immediately

If you've written unit tests before, you know the drill: given input X, expect output Y. Clean. Deterministic. Beautiful.

AI agents laugh at this.

An agent backed by an LLM might rephrase the same sentence ten different ways — all correct, all passing the vibe check, none matching your hardcoded expected string. Writing assertEqual(output, "Sure, I'd love to help!") is like testing whether your friend will use the exact same words every time they say hello.

It's not flaky tests. It's a fundamentally wrong testing paradigm.

The Counterintuitive Fix: Stop Testing Outputs, Start Testing Constraints

Here's the mental shift that saves your sanity: you don't test what the agent says. You test what it must never do, always do, and structurally produce.

Think of it like testing a chef. You wouldn't check if they plate the garnish at exactly 43 degrees. You'd check: Is the chicken cooked? Is there no glass in the food? Did they use the ingredients the customer ordered?

Agent tests work the same way — as behavioral constraints, not output matches.

There are three layers that actually work:

Layer 1: Structural Contracts

Does the output have the right shape? If your agent returns JSON, validate the schema. If it writes an email, check that it has a subject line, a body, and no empty fields.

import jsonschema

agent_output_schema = {
    "type": "object",
    "required": ["subject", "body", "tone"],
    "properties": {
        "subject": {"type": "string", "minLength": 3},
        "body": {"type": "string", "minLength": 20},
        "tone": {"enum": ["friendly", "professional", "casual"]}
    }
}

def test_email_rewrite_structure(agent, sample_email):
    result = agent.rewrite(sample_email)
    jsonschema.validate(result, agent_output_schema)  # Throws if wrong shape

This test passes whether the agent writes "Hey there!" or "Hi friend!" — because you're testing the contract, not the words.

Layer 2: Guardrail Assertions

These are your "never do this" and "always do that" rules. They catch the stuff that actually matters:

def test_email_guardrails(agent, sample_email):
    result = agent.rewrite(sample_email)

    # Never leak the original sender's private data
    assert sample_email["sender_ssn"] not in result["body"]

    # Always preserve the original recipient
    assert result["to"] == sample_email["to"]

    # Never exceed token budget (cost control!)
    assert len(result["body"].split()) < 500

    # Tone must match what was requested
    assert result["tone"] == "friendly"

Notice: not a single hardcoded phrase. Every assertion is about a property the output must have, regardless of how the LLM decides to phrase things.

Layer 3: LLM-as-Judge (Yes, Really)

For the subjective stuff — "is this actually friendly?" — use a second LLM call as a grader. It sounds circular, but it works surprisingly well when you give the judge a rubric:

def test_tone_quality(agent, judge_llm, sample_email):
    result = agent.rewrite(sample_email)

    verdict = judge_llm.evaluate(
        prompt=f"""Rate this rewritten email on friendliness (1-5).
        Original: {sample_email['body']}
        Rewritten: {result['body']}
        Score only. No explanation."""
    )

    score = int(verdict.strip())
    assert score >= 4, f"Friendliness score {score}/5 — too low"

Run this 5 times. If 4 out of 5 pass, your agent is solid. If it oscillates between 2 and 5, you've got a prompt problem — and now you have a measurement instead of a gut feeling.

The Test Pyramid for Agents

Steal this structure for every agent you build:

• Base (run on every commit): Structural schema validation. Fast, deterministic, catches regressions in output format.
• Middle (run nightly): Guardrail assertions across 20+ diverse inputs. Catches safety/policy violations and drift.
• Top (run weekly): LLM-as-judge evaluations on a curated benchmark set. Catches subtle quality degradation.

The base layer runs in milliseconds. The top layer costs a few cents in API calls. Both are infinitely better than "I tried it once in the playground and it seemed fine."

The One Thing Nobody Tells You

Agent tests aren't about proving your agent is correct. They're about proving it's safe to deploy. That's a lower bar — and a much more useful one.

Your agent will surprise you. It will rephrase things in ways you didn't expect. Sometimes better than what you'd write. The test suite's job isn't to prevent surprises — it's to prevent disasters.

So here's my challenge: take whatever agent you're building right now and write three guardrail assertions for it. Not output matches. Not vibes. Three concrete properties it must always satisfy.

What's the first guardrail you'd write? Drop it in the comments — I'll tell you if it's tight enough.