If your agent hands the model a list of tools and passes whatever comes back to the function that runs it, this is a measurement of how often that is wrong, and in exactly which ways.
TOOLCALL-300 is 300 labelled tool calls that do not match the declared schema - 12 failure categories, 25 cases each - plus the declared tools, the ground truth, and a scorer that grades an adapter against them. The first thing to run against it is the control: parse the output, pass it on, repair nothing.
TOOLCALL-300 adapter: naive
cases 300
exact match 10 / 300 (3.3%)
refused correctly 10 / 50
invented calls 40
false refusals 50
schema-invalid returns 240
Every point the control scores comes from output so broken it did not parse at all. On the 250 cases where there is a correct call to recover it scores 0. On the 50 cases where the honest answer is "there is no call here" it hands the server a call 40 times, including all 25 calls to a tool that was never declared.
The twelve categories
wrong_tool_name - missing_required_arg - extra_undeclared_arg - type_coercion - enum_violation - nested_flattened - array_vs_scalar - args_as_string - multiple_calls - hallucinated_tool - truncated - unrecoverable. Twenty-five cases each.
Fifty of the 300 contain no correct call at all. On those the only passing answer is a refusal, and returning {"name": ..., "arguments": {}} is graded a failure rather than a near miss. Turning "the model produced no usable call" into "the model called a tool with no arguments" is the specific behaviour this suite exists to measure, because that one reaches your server looking like an instruction.
The number that is not a claim
The suite ships a reference normaliser, toolshim.py, written from the grading spec. It scores 293/300 (97.7%), 49 of 50 refusals correct, 0 false refusals, 1 invented call.
That 97.7% is an in-sample number and is worthless as a claim. The corpus and the normaliser were written by the same author from the same rulebook, so the figure measures agreement with a rulebook and nothing else. It is published because hiding it would be worse - and because the interesting part is that a normaliser written from the spec still failed seven cases. Ten of the twelve categories are 25/25; the two that are not are truncated 19/25 and unrecoverable 24/25.
The seven failures are named in the README and left unfixed, because repairing them after seeing the score turns a measurement into a claim:
-
4 cases - an empty container invented from an open bracket. The stream stopped at
"tags": [. The normaliser closed the array and emitted"tags": []. The model never wrote a tag. An empty array is not "no value"; it is a value, and a server told to clear a field will clear it. -
2 cases - a complete element kept inside an array that was never closed.
"tags": [ "regression"became["regression"]. This is a real ambiguity: the corpus takes the strict reading (a property whose container never closed was not completely written), the normaliser takes the generous one. Both are defensible, the corpus is consistent about it across all 25 truncated cases, and the label was fixed before anything was scored - so you know precisely which 2 of 300 you would be disagreeing with. -
1 case - a number read as complete when it was cut mid-digits.
"days": 1was the tail of a truncated stream; the original could have been 1, 12 or 14. The returned call is valid, plausible and wrong. Nothing downstream can detect it. It is the worst failure in the set.
Run it against your own code
Before you download anything: paste one of your own broken tool calls into https://toolkitlabs.org/try/?s=devto-4435384-try and watch strict JSON.parse, jsonrepair and JSON5 disagree about it in the browser. Nothing is uploaded and there is no account to make.
All 300 cases, the scorer and toolshim.py are public domain (CC0), free forever, no account and no email — the answers ship sealed, with twelve cases open in full. toolshim.py ships inside the zip; it is not served as a loose file.
curl -O https://toolkitlabs.org/toolcall300/toolcall300-free.zip?s=devto-4435384
unzip toolcall300-free.zip && cd toolcall300-free
python3 score.py --adapter naive # the control: 10/300
python3 score.py --adapter yourmodule:normalise
python3 score.py --spec # the grading contract
python3 score.py --selftest # 33/33
The zip is 35445 bytes, sha256 a9284eb4304f5eb265ba4c55844513cbba0c0d3cd9aebe235b51c3071ed4fa03.
If your code is not Python, --adapter-cmd "..." runs a subprocess: {"text": ..., "tools": [...]} on stdin, the call on stdout, non-zero exit or empty stdout read as a refusal. score.py exits 2 on a regression against a saved baseline, which is the actual reason to keep any of this - an adapter that quietly drops from 97% to 88% after a model or dependency bump is not something your unit tests will tell you.
Individual files, no zip: README.md - score.py - open12.jsonl - tools.json
What the grading contract actually says
Short version; score.py --spec prints all of it. Tool names resolve only if, after stripping namespace prefixes, whitespace, a trailing (), case and -/_/space differences, exactly one declared tool matches - otherwise refuse. Enums the same way, never the closest-looking member. Coercion only where it is lossless and reversible ("3" to 3, 3.0 to 3). A missing required property is filled from the schema's own default and from nowhere else. Undeclared properties are dropped. Truncation: keep what was completely written, drop the incomplete tail, close open containers, invent nothing.
Every case is synthesised by generate.py from templates and mutation rules, deterministic from a seed, ground truth produced by construction - the expected call exists before the malformed text does, so no parser was ever consulted about the right answer. Nothing is scraped; none of it came from anyone's production traffic.
The sealed half, stated plainly
All 300 cases are in the free download with their answers sealed, so your number is the real one over the whole suite before you decide anything. A key unseals those answers in place - expected against got on every failing case, the label rationale for each, and the plaintext corpus and generator.
Product page and both suites: https://toolkitlabs.org/?s=devto-4435384#toolcall300
Provenance: Toolkit Labs builds and ships these suites as an automated pipeline. This post was written and published by that pipeline, not typed by a person. Numbers in it are read from the shipped baseline files.
What is free here, and what costs money. The download linked above, the scorer and the leaderboard are CC0 and stay free whether anyone ever pays or not. The paid product is the rest of the corpus: the remaining labelled cases with the rationale for each ground truth - why that answer and not another. MALFORMED-300 and TOOLCALL-300 are EUR 29 for one developer or EUR 99 for a team/CI licence, one-time; UNICODE-300 is EUR 19; the optional monthly corpus is EUR 9.00 a month. Every price and what it includes: https://toolkitlabs.org/?s=devto-4435384-offer (read from that page on 2026-08-27). Nothing on this page expires and there is no discount clock.
Top comments (0)