Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Evaluate AI Agents in Production: 3-Level Framework [2026]

AI agent evaluation in production is the practice of systematically testing whether your agent completes real tasks correctly, safely, and efficiently — not just whether the underlying LLM generates plausible text. It's the difference between knowing your agent sounds smart and knowing it works. And in 2026, with teams shipping AI agents into customer-facing workflows at unprecedented scale, getting evals right is the single highest-leverage investment you can make.

Key takeaways:

• Agent evals are fundamentally different from LLM evals — you're testing multi-step tool use, planning, and task completion, not just text quality.
• The 3-level eval framework (unit tests → LLM-as-judge → online evaluation) gives teams a concrete progression path from zero coverage to production-grade confidence.
• LLM-as-judge is the most scalable eval method for agents, but it requires calibration against human labels before you can trust it.
• Tracking latency and token counts without measuring task completion creates false confidence — the most dangerous kind of metric.
• Start with 50 golden traces from production traffic and grow the dataset weekly through a structured review ritual.

Why Agent Evals Are Fundamentally Different from LLM Evals

Here's the thing nobody's saying about agent evaluation: most teams treat it like LLM evaluation with extra steps. It's not. It's a fundamentally different problem.

When you evaluate a standalone LLM, you're asking: "Given this input, is this output good?" It's a single-turn, input-output assessment. You can use benchmarks like MMLU or HELM, run BLEU/ROUGE scores, or do pairwise comparisons. The problem is bounded.

AI agents break every assumption in that model. An agent doesn't just generate text. It plans. It selects tools. It calls APIs. It reads results and decides what to do next. It might spawn sub-agents. A customer support agent might look up an order, check a refund policy, calculate a partial credit, and draft a response — a 4-step chain where failure at any step cascades. Evaluating just the final response misses the 3 intermediate decisions that actually determine quality.

An eval that only checks the final output is like grading a math exam by looking at the answer without checking the work.

As Hamel Husain, independent AI consultant and former lead of the CodeSearchNet project (a precursor to GitHub Copilot), puts it: "Unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems." He's seen this pattern repeatedly — teams that nail their eval loop iterate 5-10x faster than teams flying blind.

The distinction matters practically. LLM evals test generation quality. Agent evals test behavior — did the agent complete the task, did it use the right tools in the right order, and did it handle edge cases without going off the rails? If you're building agents with frameworks like LangGraph or CrewAI, you need evals designed for multi-step execution, not single-turn text scoring.

The 3-Level Eval Framework

The framework I use — adapted from Hamel Husain's practitioner hierarchy and battle-tested on this site's own agent pipeline — breaks agent evaluation into three levels. Each level builds on the previous one, and you should implement them in order.

Level 1: Assertion-based unit tests — fast, deterministic, run on every commit. These catch the obvious regressions.

Level 2: Trace-based evaluation with LLM-as-judge — slower, probabilistic, run on curated datasets. These catch the subtle quality issues humans would notice.

Level 3: Online evaluation and A/B testing — continuous, production-traffic-based. These catch the things your dataset doesn't cover yet.

The key insight is that each level serves a different purpose. Level 1 prevents shipping known-broken behavior. Level 2 measures quality across the distribution. Level 3 detects drift and surprises in the wild. Most teams try to skip straight to Level 3 ("we'll just monitor production") and wonder why their agent ships garbage for two weeks before anyone notices.

Running this blog's own 7-agent publishing pipeline taught me this progression the hard way. Deterministic gates before LLM review catch more issues than doubling the review model's size. The boring Level 1 assertions — checking that slugs are valid, that category mappings exist, that word counts fall within range — prevented more broken publishes than any amount of clever LLM judging.

Level 1: Assertion-Based Unit Tests for Agent Behaviors

Level 1 is where most teams should start, and where most teams skip. These are traditional assertion-based tests adapted for agent behaviors — they run fast, they're deterministic, and they catch regressions immediately.

For agents, unit tests should cover three categories:

1. Tool selection tests — given a specific user intent, does the agent call the correct tool? If a user says "cancel my order," the agent should invoke the order cancellation API, not the refund calculator. This sounds trivial, but tool-selection regressions are the #1 failure mode I've seen when swapping underlying models.

2. Output format tests — does the agent's response conform to the expected structure? If your agent returns structured JSON for downstream processing, a format violation breaks everything downstream silently.

3. Guardrail tests — does the agent refuse to do things it shouldn't? If a user asks your customer support agent to write Python code, it should decline. These are your safety boundary assertions.

4. State transition tests — for multi-turn conversations, does the agent maintain context correctly across turns? Does it remember what the user said 3 messages ago?

The beauty of Level 1 is speed. These tests run in seconds, not minutes. You can gate every pull request on them. The Applied LLMs collaborative guide — authored by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu, and Shreya Shankar — recommends creating assertion-based unit tests directly from real input/output samples. Don't invent test cases from imagination. Pull them from actual production logs.

A practical starting point: write 10-20 unit tests covering your agent's most common intents. If your agent handles 5 primary tasks, write 2-4 tests per task. This takes a day, not a sprint. And it immediately tells you whether a model swap or prompt change broke something fundamental.

For teams using CI/CD pipelines, these tests should run on every commit, blocking merge if any assertion fails. No exceptions.

Level 2: Trace-Based Evaluation and LLM-as-Judge

What Is Trace-Based Evaluation and How to Set It Up

Trace-based evaluation for AI agents means capturing the full execution path of an agent — every LLM call, tool invocation, intermediate reasoning step, and final output — as a structured trace, then evaluating the entire trajectory rather than just the endpoint.

A good trace should capture:

• The user's input and the agent's final response
• Every intermediate LLM call (prompt and completion)
• Every tool call (name, arguments, return value)
• Token counts and latency per step
• The agent's planning or reasoning steps (if using chain-of-thought)
• Any retrieval results (for RAG-based agents)

This is where tools like LangSmith, Arize Phoenix, and similar observability platforms earn their keep. LangSmith's documentation distinguishes between offline evaluation (pre-deployment, on curated datasets) and online evaluation (post-deployment, sampling live traffic). The trace is the atomic unit for both.

The workflow looks like this: instrument your agent to emit traces → store them in your eval platform → select traces that represent interesting behaviors → label them as pass/fail → use these labeled traces as your regression dataset.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, the cost difference between running a fast evaluation model (like Claude Haiku) versus a frontier model as your LLM judge is roughly 10-20x — which matters enormously when you're evaluating hundreds of traces per deploy.

How LLM-as-Judge Works (and When to Trust It)

LLM-as-judge is the practice of using a large language model to evaluate another model's output against a natural-language rubric. Instead of writing code to check "is this response correct," you write a rubric — "Rate whether this customer support response accurately addresses the user's billing question, uses a professional tone, and provides actionable next steps" — and let an LLM score the output.

Jeffrey Ip, co-founder at Confident AI and creator of DeepEval, argues that LLM-as-judge is the most reliable evaluation method for LLM outputs when paired with calibration techniques like G-Eval. G-Eval uses chain-of-thought prompting to make the judge's reasoning explicit before scoring, which significantly improves consistency.

But here's the critical part everyone glosses over: you must calibrate your judge before trusting it.

The calibration loop works like this:

1. Take 50 production traces and have a human label them pass/fail.
2. Run your LLM judge on the same 50 traces with your rubric.
3. Compute the agreement rate between human labels and judge labels.
4. If agreement is below 80%, your rubric needs sharpening. Rewrite it, re-run, repeat.
5. Once agreement exceeds 85%, you can start trusting the judge for automated decisions.

The Applied LLMs authors propose what they call the "intern test" — if a new intern, given the same rubric and examples, would arrive at the same pass/fail conclusion, the eval is well-specified. If the intern would be confused, your rubric is too vague to automate.

A common mistake: relying on traditional scorers like BLEU or ROUGE for agent outputs. These metrics measure surface-level text overlap and completely miss semantic correctness. An agent might rephrase a perfect answer in different words and score 0.2 on ROUGE while being completely correct. LLM-as-judge handles this naturally because it evaluates meaning, not tokens.

One warning from the Applied LLMs authors that I've seen validated repeatedly: "Overemphasizing certain evals can hurt overall performance." If you optimize your agent narrowly for one metric — say, response conciseness — you might degrade helpfulness, completeness, or empathy. Balance your rubric across multiple dimensions.

Level 3: Regression Suites and Online Evaluation in Production

Level 3 is where your eval system becomes continuous. Instead of testing before deployment, you're evaluating in production — sampling real user interactions and running your LLM judge against live traffic.

LangSmith's evaluation framework implements this as online evaluation with configurable sampling rates. You don't evaluate every trace in production (that's expensive and slow). Instead, you sample 5-10% of traffic, run your evaluators asynchronously, and alert when quality drops below your threshold.

The feedback loop that makes this powerful: failing production traces get added to your offline dataset. You create targeted evaluators for the failure mode. You validate the fix against the expanded dataset. You deploy. The cycle repeats. Over weeks, your regression suite grows organically from real failures rather than imagined scenarios.

For detecting regressions when upgrading your agent's underlying model — say, moving from one Claude Sonnet version to the next — the pattern is straightforward:

1. Run your full Level 1 unit test suite. If anything breaks, stop.
2. Run your Level 2 LLM-as-judge evaluation against your golden dataset. Compare scores to the previous model version.
3. Set a regression threshold: if overall pass rate drops by more than 3%, block the deploy.
4. If the new model passes both gates, deploy to 10% of traffic with online evaluation active.
5. Monitor for 48-72 hours. If online eval scores hold, ramp to 100%.

This is the same pattern used for any canary deployment, adapted for probabilistic systems. The difference is that your "health check" isn't HTTP 200s — it's task completion rate.

Building Your Eval Dataset: Curating Golden Traces

The Cold-Start Problem

How do you build an eval dataset when you have zero production traffic? This is the #1 practical blocker for teams just starting out, and none of the major guides address it well.

Three approaches that work:

Synthetic scenario generation. Write 20-30 representative user scenarios by hand. For a customer support agent, this means crafting realistic tickets: "I was charged twice for order #12345," "My package shows delivered but I never got it," "Can I change the shipping address on an in-transit order?" Run your agent against these scenarios, manually grade the outputs, and you have your seed dataset.

Adversarial inputs. Deliberately try to break your agent. Ask it to do things outside its scope. Feed it contradictory information. Give it edge cases like empty inputs, extremely long messages, or requests in unexpected languages. These traces test your guardrails and graceful degradation.

Persona-based testing. Create 5-7 user personas with different communication styles and needs. A frustrated customer who uses caps lock. A technical user who includes error codes. A non-native English speaker. Run each persona through your top 5 workflows. That's 25-35 traces with realistic variety.

Combine all three and you'll have 50-100 traces before a single real user touches your agent. That's enough for a meaningful Level 2 eval suite.

Growing Your Golden Set from Production

Once you have production traffic, your eval dataset should grow every week. Here's the ritual I recommend for small teams (2-5 engineers):

Weekly trace review (30 minutes). Every Monday, pull 20 random traces from the past week. Each engineer reviews 5-10 traces and labels them pass/fail with a one-sentence reason for any failure. This is the Applied LLMs advice to "look at samples every day" operationalized into a sustainable cadence.

Failure-first curation. When a user reports a bad agent response, immediately add that trace to your golden set. These are your most valuable examples because they represent real disappointment.

Stratified sampling. Don't just pull random traces. Sample across your agent's different capabilities. If 80% of traffic is simple FAQ answers but 20% is complex multi-step workflows, your golden set should over-represent the 20% — that's where failures live.

A golden set of 50-200 traces, reviewed and re-validated monthly, is enough to catch most regressions for a single-purpose agent. Multi-agent systems with orchestrator-subagent architectures need larger datasets — 100-300 traces per sub-agent plus 50-100 end-to-end trajectories.

Metrics That Actually Matter vs. Metrics That Create False Confidence

Quality Metrics: Task Completion, Correctness, Grounding

The Arize AI team frames this perfectly: "A useful agent metric should answer one of three questions: did the agent complete the task, why did it fail, or what should we change next?"

The metrics that matter:

• Task completion rate — the percentage of interactions where the agent fully resolved the user's request without human escalation. This is your north star. Everything else is supporting evidence.
• Correctness — for agents that retrieve or compute information, is the answer factually right? This requires ground-truth labels for at least a sample of production traces.
• Grounding — for RAG-powered agents, does the response faithfully reflect the retrieved documents, or does the agent hallucinate beyond its sources?
• Tool selection accuracy — did the agent pick the right tool for the job? A coding agent using a web search tool when it should be reading local files is a tool selection failure even if the final answer is okay.

Job-specific metrics matter enormously here. Arize AI correctly points out that resolution rate matters for customer support agents, test pass rate for coding agents, and citation accuracy for research agents. There is no universal "agent quality score" — you have to define what success looks like for your specific use case.

Cost and Latency Metrics

These are real metrics, but they're secondary to task completion. The metrics to track:

• Cost per resolution — total API spend divided by successfully completed tasks. Not cost per request (that ignores multi-turn conversations and retries).
• Tokens per task — measures efficiency. If your agent uses 15,000 tokens to answer a question that should take 3,000, your context engineering needs work.
• P95 latency — the 95th percentile end-to-end response time. P50 is misleading for agents because multi-step executions create a long tail. I've written extensively about agent latency budgets — the short version is that each tool call adds 200-800ms, and users notice.

The operational pattern: set SLO-style gates that combine quality and cost. For example: task completion rate > 85% AND p95 latency < 8 seconds AND cost per resolution < $0.50 = green. If any threshold breaks, the deploy is blocked. This turns soft metrics into hard CI gates.

Safety Metrics

Anthropic's eval guidance makes an important point: even "hazy" criteria like safety can be quantified. Their example — "less than 0.1% of outputs out of 10,000 trials flagged for toxicity" — turns a vague goal into a measurable threshold.

Safety metrics for agents include:

• Refusal rate — how often does the agent correctly refuse out-of-scope requests? Too low means weak guardrails. Too high means the agent is over-refusing legitimate requests. Both are problems.
• Escalation rate — how often does the agent hand off to a human? Track this over time. A rising escalation rate might mean the agent is getting worse, or it might mean you're handling harder cases. Segment by intent to tell the difference.
• Unsafe tool actions — did the agent ever invoke a destructive action (delete, modify, send) without proper confirmation? This is critical for agents with write access to production systems.

Metrics That Create False Confidence

The Arize AI team warns that tracking latency, tokens, traces, and tool calls without measuring task completion "creates false confidence." I've seen this firsthand. A team proudly showed me their agent dashboard: p50 latency 1.2 seconds, 100% uptime, 50,000 traces per day. Beautiful graphs. But they had no idea what percentage of those 50,000 interactions actually resolved the user's issue. The dashboard was measuring activity, not quality.

Other false-confidence metrics: response length (longer ≠ better), tool call count (more tools ≠ more helpful), and conversation turns (fewer turns doesn't mean faster resolution — it might mean the agent gave up).

How to Build an Agent Eval CI/CD Pipeline

Integrating evals into CI/CD is where this framework becomes enforceable rather than aspirational. Here's the concrete pattern:

On every pull request:

• Run Level 1 unit tests (tool selection, output format, guardrails). These take 10-30 seconds.
• Block merge if any assertion fails.

On merge to main (pre-deploy):

• Run Level 2 eval suite against your golden dataset (50-200 traces).
• Compare pass rate to the last successful deploy's baseline.
• Block deploy if regression exceeds your threshold (I recommend starting at 3% — tighten as your dataset matures).
• Log all results for trend tracking.

Post-deploy (continuous):

• Enable Level 3 online evaluation at 5-10% sampling rate.
• Run LLM-as-judge asynchronously on sampled traces.
• Alert if rolling 24-hour quality score drops below threshold.
• Auto-add failing traces to the offline dataset for the next regression suite update.

The total eval overhead per deploy: 3-8 minutes for Level 2 (depending on dataset size and judge model speed), plus ongoing background cost for Level 3 sampling. This is negligible compared to the cost of shipping a broken agent to production for a week.

Model-per-job-shape matters here. From running this site's own agent pipeline, I learned that using a fast, cheap model (like Claude Haiku) for high-volume Level 3 online judging and a stronger model (like Claude Sonnet 4) for detailed Level 2 offline analysis beats using one model everywhere — on both cost and quality.

Evaluating Multi-Step and Multi-Turn Agent Conversations

Single-turn evaluation misses most of what makes agents hard. A real production agent conversation might span 5-8 turns, with the agent maintaining context, updating its plan, and recovering from misunderstandings.

For multi-step evaluation, you need trajectory-level scoring — evaluating the entire execution path, not just individual steps. This means:

• Step-by-step correctness: was each intermediate action appropriate given the context at that point?
• Plan coherence: did the agent's sequence of actions form a logical plan toward the user's goal?
• Recovery behavior: when the agent made a mistake or received unexpected tool output, did it adapt intelligently?
• Context retention: across a 5-turn conversation, does the agent remember and correctly reference information from turn 2?

This is where component-level evals complement end-to-end trajectory evals. Component evals test individual pieces — did the retrieval step return relevant documents? Did the tool call use correct parameters? — while trajectory evals test whether the whole sequence produced the right outcome. You need both. A trajectory can fail even when every individual component succeeds (the wrong combination of correct actions), and a trajectory can succeed despite a component-level failure (the agent recovered gracefully).

For teams building agentic AI systems with delegation chains — a parent agent spawning child agents — evaluation gets harder still. You need traces that capture the full delegation tree, and your eval rubric needs to assess whether delegation decisions themselves were appropriate.

Tooling Landscape: LangSmith, Arize Phoenix, DeepEval, and More

The agent eval tooling landscape in 2026 has matured significantly. Here's how the major options compare for agent-specific evaluation:

Tool	Type	Agent Eval Strength	Best For
LangSmith	Platform (SaaS)	Online + offline eval, trace capture, dataset management, LLM-as-judge	Teams already using LangChain/LangGraph
Arize Phoenix	Platform (open-source + SaaS)	Trace visualization, span-level analysis, eval metrics	Teams needing deep observability
DeepEval	Library (open-source)	14+ SOTA metrics, G-Eval implementation, CI integration	Teams wanting code-first eval in test suites
Braintrust	Platform (SaaS)	Prompt playground, scoring, experiment tracking	Teams iterating heavily on prompts
Anthropic Eval Guidance	Documentation	Success criteria framework (Specific, Measurable, Achievable)	Teams building on Claude

LangSmith's recent updates include an AI-powered engine that monitors traces and detects issues automatically — a significant upgrade from manual threshold-based alerting. DeepEval's open-source library implements SOTA LLM evaluation metrics with minimal setup code, making it the fastest path to adding LLM-as-judge to an existing test suite.

For teams evaluating which agent framework to use, trace instrumentation quality should be a selection criterion. If your framework doesn't emit structured traces, you'll spend weeks building custom instrumentation before you can even start evaluating.

Anthropic's research team raises a sobering point that applies to all these tools: "Many of today's existing evaluation suites are limited in their ability to serve as accurate indicators of model capabilities or safety." The tooling helps operationalize evals, but the hard work — defining what "good" means for your specific agent, curating representative datasets, calibrating your judges — remains fundamentally a human problem.

The Intern Test: Verifying Your Eval Rubric

Before you automate any LLM-as-judge evaluation, run the intern test. This concept from the Applied LLMs authors (including Eugene Yan, now a Member of Technical Staff at Anthropic) is the simplest quality gate for your rubric.

Take your eval rubric. Hand it to someone unfamiliar with the project — an intern, a colleague from another team, a friend. Give them 10 agent traces. Ask them to grade each one pass/fail using only the rubric.

If their grades match your grades 80%+ of the time, your rubric is specific enough to automate with an LLM judge. If they diverge significantly, the rubric is ambiguous and an LLM judge will produce unreliable scores.

Common rubric failures:

• "The response should be helpful" — too vague. What does helpful mean? Does it require a specific action, or is empathy enough?
• "The agent should use appropriate tools" — which tools are appropriate for which intents? List them.
• "The tone should be professional" — give examples of professional vs. unprofessional responses.

Sharpening a rubric usually takes 2-3 iterations. Each iteration, you identify the cases where human graders disagreed, clarify the rubric for those edge cases, and re-test. By the third round, agreement rates typically stabilize above 85%, and your LLM judge becomes trustworthy.

This is the step that separates teams with reliable automated evals from teams that trust scores from a poorly-specified rubric and wonder why their agent keeps shipping bad experiences.

What Comes Next

The teams that will win the agent race in 2026 aren't the ones with the fanciest models or the most complex architectures. They're the ones with the tightest eval loops. Every model upgrade, every prompt change, every new tool integration gets tested against a golden dataset, judged by a calibrated LLM evaluator, and monitored in production with automated alerts.

If you're building production AI systems today and you don't have at least Level 1 evals running in CI, stop building features and build evals first. I'm serious. Every hour you spend on evals now saves ten hours of debugging mysterious production failures later.

My prediction: by the end of 2026, "eval coverage" will be as standard a metric as test coverage is today. Teams will report it in sprint reviews. Hiring managers will ask about it in interviews. And the teams without it will be the ones still shipping agents that sound smart but don't actually work.

Start with 10 unit tests. Add 50 golden traces. Calibrate your LLM judge against human labels. Gate your deploys. Review traces weekly. The framework isn't complicated. The hard part is doing it consistently, every single week, even when the pressure to ship new features is relentless. That discipline is the difference between an agent demo and an agent product.

---

Originally published on kunalganglani.com