When a model returns several tool calls in one turn and one of them is broken, the framework has to decide what goes into the next request. agentic-arena asks two questions of that request: did the successful call's result reach the model, and was the broken call reported at all?
What was measured
Three kinds of broken call were tested, each sent in a batch next to a valid call: an unknown tool name, malformed arguments, and a missing required argument.
| unknown tool | malformed args | missing required arg | |
|---|---|---|---|
vanilla, pydantic_ai, microsoft_af
|
both | both | both |
langgraph |
both | good only | both |
smolagents |
error only | good only | error only |
openai_agents |
raises | both | both |
google_adk |
raises | raises | both |
"both" means the good result and the error both reached the model.
Three ways to get it wrong
The results show three failure modes, and they do different amounts of harm:
- Silent partial. The broken call vanishes with no message of any kind, and the run answers from partial evidence without the model ever being told.
- Successful sibling discarded. smolagents does report the error, but as a rewritten task rather than a turn in the transcript. The good result is dropped along with the history, so the model sends the identical batch again.
- Raises. This one is loud. The repo prefers it of the three because nothing is silently wrong.
Batching changes an earlier result
On its own, a langgraph call with malformed arguments produces no tool message and the graph halts, so the item fails where you can see it. Batched with a call that succeeds, the graph carries on and the drop becomes invisible. The repo records the visible failure as the better outcome. So "batching changes nothing" turned out to be false. The rule the repo checks instead is that batching may rescue a fault but must never make a framework worse than it is when calls run one at a time.
The probe had its own bug
The first version of this measurement counted the word "observation" in the messages a framework returned. One corpus entry describes Tokyo Tower as a "communications and observation tower", so the probe read 3 for a batch of 2. The overall direction of the results held, but one claim did not: it had said the model is never told, and smolagents does tell the model. The probe now matches each outcome's own text, and a test checks that none of those markers appear in the corpus.
Caveats
All of this runs in mock mode. The model's turns are scripted and byte-identical for every framework, so any difference comes from the framework's own error handling. These are behaviour measurements, not answer-quality rankings. No finding here says any framework produces a better answer; that needs a live run.
Reproduce
python -m pytest tests/test_parallel_tool_calls.py -q
The full write-up is in docs/findings.md, section 2b, and in the field guide.
Top comments (0)