If your agent parses the model's tool call and forwards the arguments to the handler without normalising them first, 240 of 300 labelled malformed calls still return something the declared schema rejects. The adapter itself raises on 51 of them. After a normalising shim runs first, that number is 0.
This is not a survey of production traffic. It is TOOLCALL-300 — 300 synthesised tool calls in 12 categories (25 cases each), each paired with ground truth against a fixed declared schema, plus a scorer that grades any adapter you plug in.
The control everyone should run first
Pass the model output straight through. Repair nothing.
TOOLCALL-300 adapter: naive
cases 300
exact match 10 / 300 (3.3%)
refused correctly 10 / 50
invented calls 40
false refusals 50
schema-invalid returns 240
Every point the control scores comes from output so broken it did not parse. On the 250 cases where there is a correct call to recover it scores 0. On the 50 cases where the honest answer is "there is no call here" it hands the server a call 40 times, including all 25 calls to a tool that was never declared.
Category by category
Two adapters were run over all 300: pass-through (naive) vs a normalising shim. Scoring is exact match on the resulting call, not "did it run".
| category | pass-through | normalised |
|---|---|---|
| unrecoverable | 10/25 | 24/25 |
| truncated | 0/25 | 19/25 |
| args as string | 0/25 | 25/25 |
| array vs scalar | 0/25 | 25/25 |
| enum violation | 0/25 | 25/25 |
| extra undeclared arg | 0/25 | 25/25 |
| hallucinated tool | 0/25 | 25/25 |
| missing required arg | 0/25 | 25/25 |
| multiple calls | 0/25 | 25/25 |
| nested flattened | 0/25 | 25/25 |
| type coercion | 0/25 | 25/25 |
| wrong tool name | 0/25 | 25/25 |
The three shapes that cost the most in production:
-
args as string — arguments arrive as a JSON string inside the
argumentsfield, so every typed field is a string and every numeric comparison downstream is wrong. -
type coercion —
"3"where the schema says integer,"true"where it says boolean. Validators reject it; the ones that do not are worse. - array vs scalar — a single value where the schema declares a list, or the reverse. This half-works until a caller iterates a string character by character.
Full breakdown with the grading spec: Why LLM tool-call arguments fail schema validation.
Try it on your own adapter first, for nothing
The scorer, reference normaliser, and 12 open cases are public domain (CC0). All 300 run in the free kit with answers sealed.
python3 score.py --corpus open12.jsonl --adapter yourmodule:normalise
python3 score.py --corpus open12.jsonl --adapter naive # the control
The full 300
The remaining 270 cases with label rationale for each one — why that ground truth and not another — are the paid product. €29 single developer, €99 team/CI licence. Instant download after payment.
The scorer, generator, toolshim.py, and 30 free cases stay CC0 forever whether you buy or not.
Top comments (0)