An agent acceptance test should establish whether the workflow performed the intended task. A tool invocation count is only part of that evidence.
Quick answer: Define the user's goal, controlled tool responses, allowed operations and observable outcome. Check submitted arguments and the final explanation against that contract. Include a meaningful failure branch and a case where the agent should stop or clarify rather than invoke another tool.
Build a small fixture
For an illustrative inventory assistant, the user requests an item at a named branch. The controlled result contains stock for that branch and a similarly named item elsewhere.
The expected answer must use the requested item and branch. If reservation is outside the authorized task, the agent should not create one simply because a reservation tool is available.
Check the whole path
| Check | Evidence |
|---|---|
| Input interpretation | Correct item and branch arguments |
| Tool result use | Answer follows the supplied inventory |
| Action scope | No unrequested reservation |
| Failure handling | Empty or unavailable results explained |
| Final response | Accurate scope and useful next step |
These are proposed test conditions, not a report of a benchmark run.
Include a counterexample
Return a valid empty result for the requested branch. The agent should not use another branch's stock as though it answered the original question.
If broader search is allowed, record the expanded scope and describe it honestly. The acceptance condition should distinguish that helpful extension from a silent substitution.
Preserve the evidence
Save the fixture version, actual arguments, relevant result and decision. If the test fails, another engineer should be able to inspect the mismatch without reconstructing it from a score.
I'm Shashank Agarwal, creator of API.market and founder and CEO of Noveum AI. Start from API.market setup, then use acceptance criteria and the regression-test guide to make the result repeatable.
Follow my work on LinkedIn. Drafted with AI assistance.
Top comments (0)