Updated October 29, 2026. The unsupported percentage about missed failures has been removed. Output metrics remain useful alongside captured process evidence.
A model-level evaluation and an agent-level evaluation can overlap, but they do not always have the same unit of analysis. A model call produces an output from supplied inputs. An application using that model may retrieve information, call tools, enforce permissions and assemble a final result.
Quick answer: Evaluate the model call for the property you need, then evaluate the application’s task outcome and relevant captured actions. Keep accuracy, tool correctness, cost and latency as distinct questions. The presence of multiple steps does not make output correctness irrelevant.
Use the task to choose the unit
For an entity extraction call, a reviewed set of expected fields may be appropriate. For a support agent that retrieves a policy and drafts an answer, inspect both the response and whether the relevant policy reached the model.
For an agent that performs an action, add evidence about authorization and the actual operation. A correct explanation is not proof that the requested action happened, and an executed operation is not proof that it was the right one.
Connect the layers
| Unit | Example question |
|---|---|
| Model response | Is the extracted field correct for the supplied text? |
| Retrieval | Does the selected source concern the requested product? |
| Tool operation | Do the inputs and result match the intended action? |
| Whole task | Did the user’s requested outcome occur? |
| Execution | What usage and latency accompanied that outcome? |
Use the evidence actually captured. Do not assume that a trace exposes every internal thought or every uninstrumented operation. Trace and span concepts are a useful foundation for choosing instrumentation boundaries.
Turn a discrepancy into a regression case
Suppose the model summarizes supplied text correctly, but retrieval supplied the wrong policy. Keep the model-level result and record the application-level failure. Test a corrected retrieval rule on the original failure and related ordinary cases.
A system prompt defines task instructions, not independent factual truth. Preserve appropriate source context or reviewed references when the check needs them.
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. See the monitoring guide and evaluation workflow for the development loop around these checks.
Follow my work on LinkedIn. This refresh was drafted with AI assistance.
Top comments (0)