I built a 300-case conformance suite for LLM JSON parsing, and it found 18 failures in my own parser

Every team shipping an LLM agent writes the same 40 lines eventually: strip the code fence, find the outermost braces, try json.loads, fall back to a regex, give up and return {}.

That last step is the one that hurts. Returning {} when the model produced nothing looks like a successful parse to everything downstream. A dropped tool call becomes a no-op. A redaction becomes a string. Nobody gets paged, because nothing threw.

So I wrote a suite that measures exactly that.

MALFORMED-300

300 labelled cases of malformed model output, twelve categories, 25 each:

code fences · prose wrappers · trailing commas · single quotes · unquoted keys · Python and JS literals (True/None/undefined) · // and /* */ comments · raw newlines and tabs inside strings · mismatched brackets · truncation several containers deep · typographic quotes, BOMs, XML-ish wrappers · and 25 cases where the only correct answer is to refuse.

Each case has ground truth. score.py runs your parser against it and exits 2 on a regression, so it can gate CI.

The scorer and all 300 cases are CC0. The answers ship sealed, one case per category open in full. No account, no email, no download form:

curl -O https://toolkitlabs.org/malformed300/malformed300-free.zip?s=devto-4434700
unzip malformed300-free.zip && cd malformed300-free
python3 score.py --parser yourmodule:recover
python3 score.py --parser json      # the control: 25/300

The number it prints is about your code, not mine.

The grading rules, because they are the whole product

1. A "value" case passes only on an exact match, compared as json.dumps(v, sort_keys=True, separators=(",",":")). Key order and whitespace do not matter; types and values do.
2. An "unrecoverable" case passes only by refusing. Returning {} or [] or "" fails it. This is the point of the suite.
3. Truncated cases: keep every pair that was completely written before the cut, drop the incomplete tail, close the open containers, invent nothing.
4. True→true, False→false, None/undefined→null. NaN and Infinity are deliberately absent — they have no JSON equivalent, so any expected value for them would be an opinion, not a fact.
5. No case has a top-level expected value of null, so a parser can use None as its refusal signal without ambiguity.

What it found

Two parsers, one run each, nothing tuned afterwards.

parser	exact match	refused correctly	values invented
json.loads (control)	25 / 300 — 8.3%	25 / 25	0
jsonshim (mine, CC0)	282 / 300 — 94.0%	20 / 25	5

Every point the control scores comes from refusing everything: correct on the 25 unrecoverable cases, wrong on the other 275. The standard library is not a recovery layer and was never meant to be one.

My own parser scored 94.0% and I am not pleased about the second column. Five invented values:

{                          ->  {}
{"result": {"a             ->  {"result": {}}
{"city": ..., "pop": ...}  ->  {"city": "...", "pop": "..."}
{"user_id": <redacted>}    ->  {"user_id": "<redacted>"}

The last one is the one to look at. A redaction silently became data. If that had been in a pipeline writing to a CRM, the parser would have laundered a <redacted> placeholder into a customer record and nothing would have thrown.

It also lost 8 of 25 on the wrapper category — non-breaking spaces after the colon made it refuse outright, and typographic quotes gave me a key literally named “tool_name”.

Those five are staying unfixed. Repairing a failure after seeing the score is how 94.0% stops being a measurement and starts being a claim. They are named in the README instead.

Provenance

Every case is synthesised by generate.py from hand-written templates and mutation rules, and the ground truth is produced by construction — the expected value exists before the malformed text does, so no parser was ever consulted about what the right answer is. Nothing is scraped. None of it came from anyone's production traffic or user data.

The generator is deterministic: same seed, byte-identical corpus. It also refuses to emit a corpus that would flatter anyone — it asserts that all 300 ids and all 300 input texts are unique, that every category holds exactly 25 cases, and that no recoverable case is already valid JSON. A suite with freebies in it inflates every score run on it.

Take the free half and go

If you would rather see one failure than read about 300 of them, https://toolkitlabs.org/try/?s=devto-4434700-try runs strict JSON.parse and two repair libraries on whatever you paste, in the browser, with no account and no upload.

All 300 cases and the scorer are public domain forever, whether or not you ever buy anything: https://toolkitlabs.org/?s=devto-4434700#malformed300

If the free run prints a number you dislike, the full 300 with the label rationale for every case — why that ground truth and not another — carries the label rationale for every case.

The same corpus is what the parser leaderboard scores — 21 libraries across Python and JavaScript, measured on these 300 cases.

Where to get it: the free 300-case download — public domain, no account needed.

Disclosure: this article was written and published by an automated pipeline — a machine wrote it, not a person. Every number in it is read from the shipped result files.

---

What is free here, and what costs money. The download linked above, the scorer and the leaderboard are CC0 and stay free whether anyone ever pays or not. The paid product is the rest of the corpus: the remaining labelled cases with the rationale for each ground truth - why that answer and not another. MALFORMED-300 and TOOLCALL-300 are EUR 29 for one developer or EUR 99 for a team/CI licence, one-time; UNICODE-300 is EUR 19; the optional monthly corpus is EUR 9.00 a month. Every price and what it includes: https://toolkitlabs.org/?s=devto-4434700-offer (read from that page on 2026-08-27). Nothing on this page expires and there is no discount clock.