Understand how coding agents discover, select, and use tools, and how those behaviors affect successful use of your product.
A developer asks their coding agent to integrate your product with their application. The agent finds your docs, installs the SDK, and starts the integration work. Then, it hits an error it can't resolve. After a few attempts, it switches to another product, completes the integration, and moves on. The overall task passed, but the agent failed to use your product successfully.
Looking at tool use helps explain how the agent arrived at the final outcome. You can see which tools it discovered, how it used them, and ultimately where it changed course. These signals help you go deeper than simple task success, to see how successfully agents are using your products.
Is the agent struggling, or is the product?
One failed tool call may say little, but patterns across agent evaluation runs can reveal where they consistently struggle to use the product. When agents repeatedly choose the wrong API, miss the same required parameter, or misinterpret the same response, those patterns may point to parts of the product that are difficult for agents to discover or use. Looking across evaluation runs helps distinguish isolated agent behavior from friction that appears consistently.
These evaluation results give product teams concrete changes to test. Update a tool description, error message, documentation page, or SKILL.md, then rerun the same tasks under comparable conditions to measure the effect. The goal is not to make every agent follow the same path, but to determine whether the change helps agents reach the correct outcome more reliably and with less unnecessary work.
Did the agent find and choose the right tool?
Before an agent can use a tool correctly, it has to find and select the right one. Comparing the tools available at each decision point with the tool the agent chose helps distinguish discovery problems from selection problems. Across evaluation runs, these patterns can reveal tools that agents consistently overlook, confuse, or choose at the wrong time, pointing to where discovery, descriptions, or guidance may need improvement.
The decision path below shows how to distinguish discovery issues from selection issues during an agent workflow.
Tracing Agent Tool Discovery and Selection
Did the agent use the tool correctly?
A successful tool call does not necessarily mean the tool was used correctly. In multi-step workflows, each call must use the right inputs, carry forward relevant state, and remain consistent with the task. Evaluating calls in sequence can reveal errors that individual tool responses may not show.
A few things are especially useful to check:
- Arguments: Did the agent provide the correct values for the task?
- State: Did it preserve and use relevant information from earlier steps?
- Defaults: Did omitted values change the tool's behavior or outcome?
- Scope: Did the call stay within the required resources and permissions?
What do repeated tool calls tell us?
Repeated tool calls can reveal where an agent is having difficulty using a product, but repetition alone is not a failure. Retries are often part of agent recovery, especially when the underlying state has changed or the agent is testing a new approach. But when the same call repeats under essentially the same conditions without advancing the task, it can indicate that the agent is stuck. Execution traces help distinguish productive recovery from repetition that adds work without progress.
Did the agent know what to do next?
A successful tool call, or even several, doesn't always mean the task, or even that step of the task, is complete. The agent still has to interpret what the response actually tells it and decide what should happen next. A request that was accepted but is still processing, for example, should lead to another check rather than a claim that the work is finished.
For product builders, this makes tool and response design part of how well agents can use a product. A useful response should give the agent enough information to understand what happened and decide what to do next. That might include the current status, an identifier needed for a later step, or an error that explains how to recover. The clearer that information is, the easier it is for the agent to continue without guessing and/or wasting time and resources.
The final outcome should still be verified against what actually happened in the environment. An agent can interpret a response incorrectly and report success even when the task is incomplete. Evaluating the end-to-end execution trace helps connect the agent's interpretation to the actual outcome and identify where the workflow broke down.
Connecting Agent Behavior to Product Outcomes
Tool-use metrics show how an agent worked through a task, but no single metric tells you whether it used the product effectively. More calls may indicate unnecessary work or careful verification; repeated calls may reflect a loop or successful recovery; and errors may block progress or be quickly resolved. Interpreting these signals requires connecting them to the execution trace and the final task outcome.
Below are common tool-use measurements, what they can help you investigate, and what requires additional context to interpret.
| Measurement | What to investigate | What it can't tell you |
|---|---|---|
| Tool calls | How much tool interaction the task required | Whether fewer calls would be better |
| Unique tools | Which tools the agent used | Whether it chose the right tools |
| Repeated calls | Where the agent retried, recovered, polled, or looped | Whether repetition was necessary |
| Tool errors | Where tool interactions failed | Whether they prevented progress |
| Tool duration | Where the agent spent time using tools | Whether that time contributed to progress |
Rather than optimizing for fewer calls, fewer errors, or shorter tool time in isolation, evaluate whether a product change helps agents complete tasks more reliably and with less unnecessary work. This connects changes to docs, tools, APIs, or agent instructions to the resulting agent behavior and task outcomes.
Top comments (2)
The piece about response design being as important as tool descriptions hits the hardest operational failure mode. If an API returns an HTTP 200 with an empty JSON object or a generic status string on an async job, an agent usually assumes the mutation is complete and moves straight to verification or the next dependent step. Returning explicit follow-up handles or clear transition states right in the return payload cuts out half the guesswork before the model has to infer whether it should poll or proceed.
The table framing (what a measurement tells you vs what it can't) is the most useful part of this for me. One metric I'd add that isn't fully captured by "tool errors" or "repeated calls": tool-call precision at the decision point, i.e. of the tools the agent actually had available in its manifest at that step, what fraction of calls were even plausible candidates for the task vs obviously wrong. We've seen agents "succeed" on a task by brute-forcing through 4-5 wrong tool calls before landing on the right one - task-success metrics say pass, repeated-calls says nothing failed, but the product's tool descriptions were clearly ambiguous enough to cost real latency and tokens. Tracking precision-at-first-call specifically (not just eventual success) surfaced discoverability problems for us that error rates alone missed, since a wrong-but-successful call doesn't throw an error at all.