Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.

How to Evaluate AI Agents: Regression Tests, Trajectory Eval, and Reliability in 2026

Shipping an AI agent that demos well is easy. Knowing whether it still works after you swap a model, edit a prompt, or add a tool is the hard part. By 2026 the teams running agents in production have converged on a few non-obvious practices: you grade the whole trajectory rather than the final answer, you build your test set from real failures instead of imagined ones, and you stop trusting a single passing run because agents are statistically unreliable across repeated attempts.

The difference between a suite that catches a regression the morning you bump a model and one that floods you with false alarms comes down to a handful of design decisions — what you grade, where your cases come from, and how many times you run each one. Those decisions, and a setup you can stand up this week, are what follows.

What you are actually evaluating: the trajectory, not the answer

When an agent answers a question, the final text is a tiny slice of what happened. It may have called five tools, retried two of them, hallucinated an intermediate result, and recovered. A single end-state check hides all of that.

Anthropic's agent-eval guide names the unit of evaluation precisely: a transcript (also called a trace or trajectory) is "the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions." That is what you store, replay, and grade. If you only log the final message, you cannot debug a regression — you will see that a case failed but not where the agent went wrong.

So the first engineering task is plumbing: capture full trajectories for every trial, in a structured, replayable format. Everything downstream depends on it.

The trap: grading the exact path

The intuitive next step is to assert the agent took a specific sequence of tool calls. Resist it. As the same guide warns, "There is a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order. We've found this approach too rigid and results in overly brittle tests," because "agents regularly find valid approaches that eval designers didn't anticipate." The recommendation: "it's often better to grade what the agent produced, not the path it took."

In practice this means checking outcomes and state, not steps. Did the refund get issued for the right amount? Is the database row in the expected state? Did the agent avoid the forbidden action? A test that demands search() -> filter() -> book() in that exact order will go red the day the agent discovers a one-call shortcut — a false alarm that trains your team to ignore the suite.

Grading approach	What it checks	When it helps	Failure mode
Exact tool-call sequence	Path matches a script	Almost never; tightly constrained flows only	Brittle; breaks on valid alternative paths
Final-answer only	Output string/JSON	Simple Q&A, classification	Misses silent errors and unsafe intermediate actions
Outcome / state check	World ended up correct	Most agent tasks	Needs an inspectable environment
LLM-as-judge on transcript	Subjective quality of reasoning	Tone, helpfulness, safety	Needs calibration; can hallucinate grades

The right answer is usually a combination of the bottom two rows, which we get to below.

Build your eval set from real failures

A good eval set is not invented at a whiteboard. The advice is blunt: "Begin with the manual checks you run during development... If you're already in production, look at your bug tracker and support queue." Every user-reported failure becomes a structured case. This does two things at once — it grounds your suite in problems that actually happen, and it guarantees you never reintroduce a bug a customer already hit.

A workable intake loop:

User reports broken behavior
  -> reproduce, capture the failing trajectory
  -> minimize to a deterministic test case (inputs + expected outcome)
  -> add to regression suite
  -> fix the agent
  -> case now passes; it guards that fix forever

This is the same discipline as a code regression test, applied to non-deterministic systems. If you run a content or RAG pipeline, the same loop applies — see our companion guide on RAG evaluation for the document-grounded version of this.

Two kinds of evals: regression vs. capability

Keep these separate, because they have opposite targets.

Regression evals ask, in Anthropic's framing, "Does the agent still handle all the tasks it used to?" and "should have a nearly 100% pass rate." These are your safety net before every release. But there is a subtlety worth internalizing: "An eval at 100% tracks regressions but provides no signal for improvement." A suite stuck at 100% tells you nothing about whether the agent got better — only that it did not get worse.

Capability evals are the opposite. They should be hard enough that the agent fails some of them, so changes produce measurable movement. You want headroom.

Dimension	Regression eval	Capability eval
Question answered	Did we break anything?	Did we get better?
Target pass rate	~100%	Deliberately below 100%
Source	Past bugs + shipped behaviors	Hard/novel tasks
Runs when	Every CI gate, pre-release	Periodic, model upgrades
If it moves	Block the release	Quantify the improvement

Grade with deterministic checks plus an LLM judge

For most real agents, neither pure code assertions nor pure model grading is enough on its own. The recommended pattern combines them. For a coding agent the deterministic question is concrete: "does the code run and do the tests pass?" Unit tests validate correctness cheaply and without ambiguity. Then you "also grade the transcript" with an LLM rubric to capture overall quality — was the approach reasonable, did it explain itself, did it avoid unsafe shortcuts that happened to pass the tests.

Use deterministic graders wherever a check can be expressed as code or a state assertion: tests pass, the row exists, the total equals the expected amount, no guardrail was tripped. Reserve the LLM judge for the genuinely subjective dimensions.

Making the LLM judge trustworthy

LLM-as-judge is powerful and easy to get wrong. Three rules from the guidance:

1. Calibrate against humans. Graders "should be closely calibrated with human experts." Before you trust a judge, have humans label a sample and confirm the judge agrees. An uncalibrated judge is a random number generator with good grammar.
2. Give it an out. To avoid hallucinated grades, "give the LLM a way out, like providing an instruction to return 'Unknown' when it doesn't have enough information." A judge forced to pick pass/fail will fabricate a verdict.
3. One judge per dimension. "Grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions." Asking one call to score correctness, tone, and safety at once produces muddy, correlated scores. Split them.

Reliability: why one passing run is a lie

Here is the finding that reshapes how you read your numbers. Agents are not consistent across attempts, and a single green run badly overstates how good they are.

Two metrics make this precise. pass@k measures whether an agent gets at least one correct solution in k attempts — an optimistic, best-of view. pass^k measures whether all k trials of the same task succeed — the reliability view your users actually experience. The τ-bench benchmark introduced pass^k specifically, in its authors' words, "to evaluate the reliability of agent behavior over multiple trials."

The gap is brutal. On τ-bench, state-of-the-art function-calling agents score below 50% task success and are "quite inconsistent (pass^8 <25% in retail)." Sierra puts a number on it: "the agent powered by GPT-4o drops to ~25% on pass^8 in τ-retail, which is a staggering 60% drop compared to its corresponding pass^1 score" — meaning roughly a one-in-four chance of correctly handling the same issue across eight different customers.

Why does this happen? Errors compound multiplicatively. If each of n steps succeeds independently with probability p, end-to-end success is p^n. The arithmetic is sobering:

Per-step accuracy	10 steps	20 steps	50 steps
95%	~60%	~36%	~8%
99%	~90%	~82%	~61%
99.9%	~99%	~98%	~95%

Even at an optimistic 99% per step, a 20-step agent succeeds only ~82% of the time, and 0.99^100 degrades to ~36.6%. Better models shift the curve, but the multiplication never goes away. The takeaway for your eval harness: run every important case k times and report pass^k, not the result of a single lucky sample. τ²-bench extends this to dual-control settings where both user and agent call tools, precisely because small per-step error rates compound into large end-to-end failures over a multi-step conversation. The Agent Reliability Calculator reproduces this exact table for any per-step rate and step count, plus a reverse mode that solves for the per-step accuracy a target pass^k rate requires.

Most failures are context failures

When you start reading failing trajectories, you will notice a pattern. The model is usually capable; the inputs were broken. Phil Schmid frames the era as a shift from prompting to context engineering, and states the diagnosis plainly: "Most agent failures are not model failures anymore, they are context failures." The relevant information was missing, the tools were poorly defined, or the context window was cluttered with noise the model had to fight through.

A large-scale benchmark adds hard numbers here, though they complicate the qualitative claim more than they confirm it. MCP-Atlas — a 2026 benchmark evaluating 13 frontier and open models across 1,000 tasks spanning 36 real MCP servers and 220 tools, using a four-category failure taxonomy (Tool Usage, Task Understanding, Response Quality, Logical Errors) — found that Tool Usage failures are the single largest category, averaging 56.7% of failures across models (a 47.5–68.5% range), ahead of Task Understanding (30.3%), Response Quality (8.5%), and Logical Errors (4.5%). On this benchmark, in other words, the largest failure mode is still mechanical — wrong tool, wrong parameters, wrong sequencing — rather than comprehension or synthesis. The split varies by benchmark and task type, so treat this as a caution against assuming context failures dominate everywhere: when you triage a failing trajectory, check the tool call itself before you assume the problem was comprehension.

Context here is everything the model sees before it acts: the system prompt, the user input, retrieved documents, conversation history, tool definitions, and long-term memory. Schmid defines the discipline as "designing and building dynamic systems that provides the right information and tools, in the right format, at the right time, to give a LLM everything it needs to accomplish a task." One concrete lever this opens up is applying retrieval to tool selection — fetching only the most relevant tool descriptions for the current step instead of dumping every tool into the prompt, which both cuts noise and frees up the context budget. So when an eval case fails, your first question should be "what was in the context window?" before "is the model dumb?"

The metrics worth tracking

Quality is necessary but not sufficient; agents also have to be affordable and fast. A rounded scorecard spans both:

• Quality: task success rate, trajectory analysis, tool-selection and tool-call accuracy.
• Efficiency: step efficiency η = optimal steps / actual steps; cost per task (input + output tokens × model price); end-to-end latency (planning + tool execution + reflection).
• Safety: guardrail-violation rate and graceful-failure rate (does it fail safely when it cannot succeed?). The guardrails being measured here — tiered permissions, human approval, sandboxing — are the subject of the companion guide on AI agent safety guardrails.

A 99% success rate that takes 40 steps and costs ten times a competitor is not a win. Track the efficiency and safety columns alongside accuracy or you will optimize the wrong thing.

Wiring it into CI

The end state is an automated gate. A GitHub Action runs the replay regression suite — on the order of dozens of golden cases — on every pull request, each case executed k times, graded by the deterministic + judge stack, and the build blocks if the regression pass rate slips below threshold. New production failures flow back in as fresh golden cases. That loop is the whole discipline in one sentence: capture trajectories, mine real failures, replay k times, grade outcomes plus quality, gate, repeat.

What backs the numbers

The reliability figures — τ-bench pass^8 under 25% in retail, GPT-4o's ~25% pass^8 and ~60% relative drop, sub-50% task success — come straight from the τ-bench paper and Sierra's write-up in Sources, and the τ²-bench extension from its arXiv paper. One thing to keep in mind reading them: each reflects the specific model version those authors evaluated, so the absolute scores belong to that snapshot, not to whatever model you are running today. The vocabulary of transcripts, regression versus capability evals, and the calibrate/give-an-out/one-judge-per-dimension rules are all quoted from Anthropic's engineering guide; the "context failures, not model failures" framing and the definition of context engineering are Phil Schmid's. The pass@k and pass^k definitions are our own paraphrase — only the "reliability of agent behavior over multiple trials" wording is quoted, from the τ-bench abstract. The compounding-accuracy table is plain p^n arithmetic — a worked illustration, not data from a run. The MCP-Atlas failure-category breakdown and percentages come from that paper's Section 5.2 and Table 4 (arXiv 2602.00933).

Verification note (2026-06-28): All quoted figures were checked against primary sources. The τ-bench abstract (arXiv 2406.12045) confirms verbatim that state-of-the-art agents "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)" and that pass^k was proposed "to evaluate the reliability of agent behavior over multiple trials." Sierra's blog confirms the "~25% on pass^8 ... staggering 60% drop compared to its corresponding pass^1 score" line. Phil Schmid's page supports only the qualitative claim "Most agent failures are not model failures anymore, they are context failures" and the quoted definition of context engineering — no numeric failure-rate or tool-selection-improvement figures are claimed here because the source does not state any. The pass@k/pass^k one-line definitions are this article's own paraphrase, not direct quotes. The compounding-accuracy table (e.g. 0.99^100 ≈ 36.6%) is plain arithmetic illustrating independent-step compounding, not measured data.

Verification note (2026-07-16): Re-checked all named tools/frameworks and cited figures against current sources. One material drift was found and fixed: the MCP-Atlas paragraph previously claimed "20 frontier models," "an 11-category diagnostic taxonomy," and "63.3% of diagnosed failures are cognitive rather than tool-call related." The paper (arXiv 2602.00933, confirmed via its arXiv HTML full text) actually evaluates 13 frontier and open models, uses a four-category failure taxonomy (Tool Usage, Task Understanding, Response Quality, Logical Errors defined in Section 4.2), and finds the opposite of what was claimed: "Tool Use Errors Dominate," averaging 56.7% of failures (47.5–68.5% range) versus 30.3% for Task Understanding, 8.5% for Response Quality, and 4.5% for Logical Errors (Section 5.2, Table 4). The 36-server/220-tool/1,000-task scope figures were independently correct and unchanged. All other quotes (Anthropic's evals guide, τ-bench/τ²-bench abstracts, Sierra's blog, Phil Schmid's context-engineering post) were re-verified verbatim against their live pages with no drift found.

Sources

• Anthropic — Demystifying evals for AI agents
• τ-bench: A Benchmark for Tool-Agent-User Interaction (arXiv 2406.12045)
• τ²-bench: dual-control tool-agent-user environments (arXiv 2506.07982)
• Sierra — Benchmarking AI agents (pass^k reliability)
• Phil Schmid — The New Skill in AI is Context Engineering
• Confident AI — LLM Agent Evaluation: The Complete Guide
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (arXiv 2602.00933)