How to Evaluate AI Agents

Evaluate agent performance using representative tasks, controlled environments, observable execution traces, and repeated trials.

Today’s AI agents extend language models with tools, instructions, and an execution loop for completing multi-step tasks. They can search the web, query customer records, edit code, invoke APIs, and modify external systems. As a result, an agent produces not only an output but a sequence of actions that can alter system state.

As agents shift from producing answers to taking action, evaluation has to evolve with them. A correct outcome tells us that an agent reached its destination, but not what happened along the way. Did it use the right tools? Follow the required steps? Recover when something went wrong? Answering these questions starts with understanding the core components of a well-designed agent evaluation.

Start with the right tasks

The foundation of an agent evaluation is the task set. Much like a test set in conventional model evaluation, it defines the distribution against which performance will be measured. For agents, however, task coverage extends beyond input and expected output.

The table below outlines the types of tasks a well-designed evaluation set should include. Within each area, tasks should vary in complexity, workflow length, tool dependencies, required resources, ambiguity, state, constraints, failure conditions, and expected difficulty.

Evaluation Task Set

Task	Description	What it tests
Core workflows	Representative tasks from the primary workflows the agent is expected to perform.	Whether the agent reliably completes expected work.
High-impact edge cases	Less common scenarios where failure has meaningful consequences, including ambiguous inputs, conflicting constraints, tool failures, and cases requiring escalation.	Whether the agent handles difficult or unusual conditions safely and appropriately.
Capability boundaries	Tasks the agent cannot or should not complete because of missing capabilities, tools, permissions, information, or policy constraints.	Whether the agent recognizes its limits and responds appropriately rather than fabricating results or taking unsupported actions.

Real-world tasks

Real-world tasks are valuable because they capture the complexity of using a product as it actually exists: unexpected requests, incomplete information, hidden dependencies, and constraints that are difficult to anticipate in a designed test. But production data tells an uneven story. Common workflows appear everywhere, while rare but consequential failures may barely appear at all. Abandoned tasks can leave less evidence than successful ones, and some cases require expert judgment to establish what should have happened.

Fortunately, much of the material for these tasks already exists. Useful sources include:

• Support cases
• Operational logs
• Incident reviews
• Audit trails
• Shadow traces
• Workflow documentation
• Interviews with domain experts

Production failures are often the best source of new evaluation tasks. When an agent struggles with a real workflow, that failure can be created into a task and added to the evaluation suite. Anthropic’s agent-evaluation guidance recommends combining these observed failures with tasks derived from product requirements, then moving established capability tests into the regression suite over time. If the original case contains sensitive data, remove or replace that information carefully. The goal is to protect the underlying records without simplifying away the conditions that caused the agent to struggle.

Synthetic tasks

Synthetic tasks allow evaluators to create targeted scenarios, vary one condition at a time, and expand coverage without waiting for those cases to appear naturally. They are especially useful when production data is limited or when important conditions are too rare, costly, or risky to reproduce in a live system. But the control that makes synthetic tasks useful can also make them less realistic. Generated cases may use repetitive language, combine conditions in unlikely ways, or simply fail to capture the distribution of work an agent will encounter in the real world.

To reduce this gap, build synthetic tasks from real evidence. Ground them in:

• Validated workflows
• Product requirements
• Company policies
• Failures observed in real traces

Then systematically vary inputs, permissions, system state, tool responses, and task complexity. LLMs can generate variations efficiently, but domain experts should verify that each scenario is realistic, its expected outcome is correct, and the task does not leak the solution.

Environment and evaluation harness

A task specifies the goal, but it is only one part of the evaluation. The agent needs an environment in which to act, and the evaluation harness manages the experiment as the agent interacts with that environment. The table below describes the role of each and their components.

Evaluation Environment and Harness

Component	Role	Evaluation requirement
Environment	Provides the state, data, tools, permissions, constraints, external systems, and tool behavior the agent encounters while completing the task.	Reproduce the conditions that could materially change agent behavior or task outcomes, including relevant failures and edge conditions.
Harness	Initializes the environment, presents the task, mediates agent and tool interactions, enforces execution limits, records the trace and resulting state, determines when the run ends, and passes evidence to graders.	Run each trial from a controlled, isolated state and preserve the evidence needed for reproducibility and grading. Validate the harness with known-pass and known-fail cases to catch reset, tool, instrumentation, and grading errors.

The evaluation also needs to be reproducible. Version the task, environment, tool schemas, policies, grader, and agent configuration so that changes in results can be traced back to changes in the system. When production reveals a new failure, reduce it to the smallest reproducible task and add it to the evaluation task set. Over time, these tasks can become regression tests, and changes to a model, prompt, tool, or policy can be measured easily.

Example

Imagine evaluating a support agent asked to refund a damaged order. Its environment contains the customer account, order history, refund policy, permissions, and payment tools, including real constraints such as approval limits and API failures. Without those conditions, the agent may succeed in the evaluation using a workflow that would fail in production.

The harness establishes the starting state, presents the request, gives the agent access to the appropriate tools, records what happens, and determines when the run ends. It then preserves the evidence for grading and resets the environment before the next trial. Grading can examine more than the final state. Apple’s ToolSandbox benchmark, for example, evaluates both intermediate milestones and the final outcome, making it possible to verify not only whether the agent reached the goal but whether important steps occurred along the way.

Choose the right metrics

Task success is necessary, but it is not sufficient to characterize agent performance. Evaluation should separately capture whether the agent reaches the intended outcome, respects required constraints, recovers from failures, and uses time and resources efficiently. This multi-metric approach follows the broader principle demonstrated by Stanford’s HELM framework, which evaluates language models across multiple dimensions rather than treating accuracy as a complete measure of performance. For agent systems, a useful metric stack includes:

• Outcome metrics: Measure whether the agent reaches the required, verifiable final state. Award partial credit only when partial completion has meaningful value.
• Constraint metrics: Measure violations of required policies, permissions, privacy rules, and security boundaries. Keep these separate from task success.
• Trajectory metrics: Measure how the agent executes the task, including invalid tool calls, retries, recovery, escalation, and unnecessary steps. OpenAI’s trace-grading guidance shows how examining an agent’s decisions and tool calls can reveal behaviors that are not visible from the final outcome alone.
• Efficiency metrics: Measure latency, cost, token usage, tool calls, and human intervention. Compare efficiency only across runs that meet the required outcome and constraints.

It's tempting to reduce an evaluation to one number, particularly when comparing agents or deciding whether to ship a new version. But a composite score can hide the failures an evaluation is supposed to expose. An agent, for example, might reduce latency and cost while introducing a critical security vulnerability. Averaging those changes together creates a better score without necessarily creating a better agent.

Test for Consistent Performance

Agent behavior is inherently variable. Even with the same task, repeated executions can produce different trajectories and outcomes because of model stochasticity, tool behavior, interaction state, and environmental conditions. A successful trial therefore establishes capability but does not, by itself, establish reliability.

For critical tasks, evaluate repeated trials under controlled conditions and report success across the resulting distribution. pass@k estimates whether at least one of k attempts succeeds and is appropriate when repeated attempts are part of the workflow. Tau-bench introduced pass^k to measure the probability that all k attempts succeed, placing a much stricter requirement on consistency. It's particularly useful for workflows where reliable behavior across repeated executions matters and where a failed run cannot simply be discarded and retried.

Reliability often declines as tasks become longer and more complex. METR’s time-horizon research captures this by estimating the length of human-completable tasks that agents can complete at a given success rate. The same principle applies to product evaluations. Results should be sliced by factors such as workflow length, tool dependencies, ambiguity, and opportunities for recovery. Otherwise, improvements on shorter or simpler tasks can raise aggregate success rates while longer, stateful workflows remain unreliable.

Conclusion

The goal of agent evaluation is not just to produce a score, but to understand how an agent behaves throughout a task. Representative tasks show whether the right capabilities are being tested. Controlled environments make those tests reproducible. Execution traces reveal what happens along the way, and repeated trials show whether that behavior is dependable. Combined, this evidence helps distinguish occasional success from reliable performance when conditions change, tools fail, and tasks become more difficult.