Updated October 29, 2026. The article now retains useful output metrics while adding process and tool checks. It removes the claim that traditional evaluation stops working for agents.
An agent’s final result and its path through the task answer different engineering questions. Evaluate both when the path affects cost, permissions, latency or reliability.
Quick answer: Keep suitable output metrics, then add checks for captured tool operations and task boundaries. A correct final answer does not prove that an action was authorized or that execution was efficient. An orderly sequence of calls does not prove that the final answer is correct.
Retain the outcome check
For classification, accuracy can still answer a defined question about predicted labels. For a structured extraction task, field-level correctness can still matter. Adding tools or multiple steps does not make these measures meaningless.
Define the task before selecting a metric. An exact-match check may fit a normalized identifier but be too restrictive for a natural-language explanation. A judge-based rubric may help evaluate the explanation, but it needs validation against reviewed examples.
Add process checks where the task needs them
Consider an illustrative travel-information agent that looks up a route and summarizes travel time. A correct summary does not settle whether it used an approved tool, sent only appropriate data or repeated a paid request unnecessarily.
| Layer | Example check |
|---|---|
| Input | Was the user’s requested route interpreted correctly? |
| Tool use | Did the request use the intended origin and destination? |
| Returned evidence | Did the result concern that route and time context? |
| Final answer | Did the summary accurately reflect the result? |
| Execution | Were there avoidable duplicate calls or failed retries? |
These checks have different input requirements. Do not combine their results into a universal score without explaining what the combination means.
Evaluate what is available
Use captured messages, tool definitions, inputs, outputs and relevant timings. Instrumentation determines which operations are recorded. A trace does not promise access to all private reasoning inside a model. OpenTelemetry’s trace documentation explains the underlying recorded-operation model.
If a required field is missing, preserve that condition as a coverage gap. Do not turn an unavailable check into a passing result.
Review a candidate change on both layers
Suppose a change reduces duplicate route lookups. Check that it actually reduces captured calls on comparable cases, that the final result remains correct and that failed lookups still receive appropriate handling.
Keep representative cases and previously observed failures. Compare the candidate and baseline with recorded application versions and evaluation settings. Use the evidence to decide whether the change improves the application.
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. My monitoring guide connects execution evidence with evaluations; the eval-driven guide brings both into release review.
Follow my work on LinkedIn. This refresh was drafted with AI assistance.
Top comments (0)