How To Evaluate An AI Agent

In classical test-driven development, we deal with deterministic outcomes. We write assertions against values and types we already know in advance — assert result == expected, and we move on.

AI agents don't play by those rules. Because the underlying LLM is non-deterministic, giving it the same input twice can produce two differently-worded outputs. That alone breaks the classical assertion model.

But there's a second problem, beyond just consistency. Say we have a customer support agent, and we want it to be helpful, empathetic, professional, and friendly when it talks to customers. Even if the output were consistent, how do we test for qualities like that? There's no fixed value or type to assert against — "empathetic" isn't a type or an exact string, it's a subjective judgment call.

So we're dealing with two distinct problems: unpredictable output, and qualities that are inherently subjective. Testing an AI agent means solving for both.

Enter Pydantic Evals

Pydantic Evals is Pydantic AI's recommended way to test the output of your AI agents. It's an evaluation library that gives you a set of evaluators purpose-built for exactly this kind of testing — some for the deterministic parts of your agent's behavior, and some for the parts that require judgment.

Testing Deterministic Behavior: IsInstance

Not everything about an AI agent is unpredictable. If your agent is set up with a structured output type, you can — and should — still test that the shape of the response is correct, even if the content varies.

For example, say we have an agent that returns a structured output type:

from pydantic_ai import Agent
from dataclasses import dataclass

# Response Model
@dataclass
class Response:
    message: str
    sentiment: str

# Agent Setup
support_agent = Agent(
    model="openai/gpt-5.5",
    name="customer_support_agent",
    instructions="""
        You are a customer support agent for an online store.
    """,
    output_type=Response
)

We can define a test case that checks the output is an instance of that Response type, using the IsInstance evaluator:

from pydantic_evals import Case
from pydantic_evals.evaluators import IsInstance

test_case = Case(
    name="check_agent_response",
    inputs="where is my order?",
    evaluators=[
        IsInstance(type_name='Response')
    ]
)

This is a classic assertion in spirit — we know exactly what type we expect back, so we test for it directly. No judgment call required.

Testing Non-Deterministic Behavior: LLMJudge

Structural checks like IsInstance don't help us with the harder problem: was the response actually good? Was it empathetic? Professional? Did it address the customer's question?

This is where LLMJudge comes in. It uses an LLM as a judge to evaluate your agent's output against a set of criteria you define — scoring qualitative aspects of the response that a type check could never catch.

from pydantic_evals.evaluators import LLMJudge

quality_case = Case(
    name="check_response_quality",
    inputs="where is my order?",
    evaluators=[
        LLMJudge(
            rubric="Response should be empathetic, professional, and directly address the customer's question about their order status."
        )
    ]
)

Under the hood, LLMJudge sends your agent's output and your rubric to a judge model, which returns a score (and often a reason) reflecting how well the response meets your criteria. Rather than asserting on an exact value, you're asserting on a standard.

Together, IsInstance and LLMJudge cover both problems from earlier: structural correctness for the deterministic parts, and rubric-based scoring for everything else.

Watch the Full Walkthrough

I go deeper into LLMJudge — including setting up rubrics, reading scores, and combining multiple evaluators in a single test suite — in this video, part of the Master Pydantic AI series on my YouTube channel.

If you find it useful, I'd really appreciate a like and a subscribe on the channel.