A tool call can pass schema validation and still select the wrong product, skip a prerequisite, or execute before the user has provided enough information.
That makes tool calling a more demanding problem than generating structured output.
For today’s reading, I looked at three benchmarks: BFCL, ToolSandbox, and τ-bench. They examine different parts of this problem, from selecting functions to completing stateful conversations reliably.
1. Function-call accuracy and workflow success measure different capabilities
The Berkeley Function Calling Leaderboard (BFCL) evaluates function calling across serial and parallel calls, abstention, and stateful multi-step settings. Its AST-based evaluation examines call structure beyond whether an output is parseable JSON.
The paper reports that strong single-turn performance does not resolve challenges involving memory, dynamic decisions, and long-horizon reasoning.
For a backend developer, this raises a useful distinction:
Can the model construct the expected call?
Can the agent complete the intended workflow?
A model might correctly invoke an exchange endpoint while choosing a replacement that violates the customer’s preferences.
The function exists. The arguments have valid types. The decision remains incorrect.
2. Wrong arguments can come from failed reasoning over real data
The original τ-bench paper documents a particularly relevant failure: an agent chooses the right kind of tool but supplies incorrect arguments after reasoning over product inventory.
Its example involves selecting a replacement lamp under brightness and power-source preferences. This is a harder problem than avoiding invented IDs: an existing item can still be the wrong item.
That distinction changes how I would classify failures:
| Failure | What went wrong? |
|---|---|
| Invalid identifier | The referenced record does not exist |
| Incorrect selection | The record exists but fails the user’s constraints |
| Incorrect operation | The tool performs the wrong action |
| Premature execution | Necessary information or prerequisites are missing |
Checking that an ID exists addresses only the first row.
Argument grounding also requires checking why that record was selected.
3. Tool correctness depends on environment state
ToolSandbox evaluates dependencies between tools and the state of their environment.
One example involves sending a message when cellular service is disabled. Completing the task requires handling the relevant settings before retrying the operation. The benchmark also tests canonicalization—turning natural-language expressions into API-ready values—and cases where necessary information is unavailable.
Consider this illustrative backend case:
{
"tool": "refund_payment",
"arguments": {
"payment_id": "pay_742",
"amount": 1200
}
}
Even if the schema accepts this call, execution may depend on:
- Whether the payment was captured.
- Whether 1,200 is within the remaining refundable amount.
- Which currency and unit the amount represents.
- Whether another refund already changed the balance.
My engineering conclusion: a tool schema describes accepted inputs; the backend must enforce the conditions under which those inputs are actionable.
4. A correct final state can hide an incorrect process
τ-bench evaluates outcomes by comparing the final database state with an annotated goal.
However, the authors explicitly acknowledge a limitation: reaching that state may be insufficient. An agent could perform the expected update while skipping required user confirmation.
ToolSandbox approaches trajectory evaluation through milestones and minefields: events that should occur and events that must not occur.
This suggests two complementary checks for an application:
Outcome check: Did the intended change happen?
Trajectory check: Were required steps followed, and forbidden actions avoided?
For example, a refund test should verify both the resulting balance and whether the workflow’s authorization requirements were satisfied.
5. Reliability requires repeated evaluation
τ-bench distinguishes two metrics:
- pass@k: at least one of k trials succeeds.
- pass^k: all k trials succeed.
In its original 2024 experiments, the GPT-4o function-calling agent achieved 61.2% single-trial success on retail tasks, while pass^8 fell below 25%. These are historical results under that paper’s setup, rather than estimates of current models.
The distinction matters because finding one successful run and delivering consistent service are different objectives.
What I would evaluate in a backend integration
Based on these papers, I would build tests that separately measure:
- Call correctness: appropriate tool and grounded arguments.
- State handling: prerequisites, changed records, and recovery after errors.
- Clarification: whether the agent asks when essential information is missing.
- Workflow compliance: required steps and prohibited actions.
- Outcome correctness: expected database changes without unintended changes.
- Consistency: repeated success from the same initial conditions.
These are proposed application checks, not results from an experiment I have run.
The main lesson from this reading: tool calling needs evaluation at the level of decisions, execution traces, and state transitions.
Valid JSON is one useful check along that path. It cannot establish that the agent did the right thing.
Top comments (2)
The bad call is the expensive one. Schema validation passes, the wrong tool runs, and the model re-plans from the full transcript. A prerequisite check in code, before the call, is cheaper than the repair.
Research papers