I benchmarked 7 Python JSON parsers on 300 malformed LLM outputs — mine lost the column I cared about

I maintain a small stdlib tool called jsonshim that pulls JSON out of a language model's reply. Last week I built a 300-case conformance suite for that job, MALFORMED-300, and published what it found in my own parser: 18 failures, five of them values invented where the model had produced nothing.

The obvious next question was one I had been avoiding: how does it do against the libraries people actually reach for?

So I ran all of them. Once each. Here is what came back.

The table

Seven parsers, 300 labelled cases, Python 3.12.3, one run each, nothing tuned afterwards.

parser	exact match, all 300	exact match, 275 recoverable	refused correctly	values invented
jsonshim	282 / 300 — 94.0%	262 / 275 — 95.3%	20 / 25	5
json-repair 0.63.2	265 / 300 — 88.3%	264 / 275 — 96.0%	1 / 25	24
dirtyjson 1.0.8	123 / 300 — 41.0%	98 / 275 — 35.6%	25 / 25	0
json5 0.15.0	118 / 300 — 39.3%	93 / 275 — 33.8%	25 / 25	0
pyjson5 2.0.1	118 / 300 — 39.3%	93 / 275 — 33.8%	25 / 25	0
demjson3 3.0.5*	110 / 300 — 36.7%	85 / 275 — 30.9%	25 / 25	0
json.loads (control)	25 / 300 — 8.3%	0 / 275 — 0.0%	25 / 25	0

* demjson3 reports 3.0.5 as its module __version__; the pip distribution that installs it reports 3.0.6. Versions in the table are read from the installed module.

Two columns because these libraries do not share a design goal, and one column would quietly punish some of them for a decision their authors made on purpose.

All 300 grades by the suite's contract: 25 of the cases are unrecoverable — the model emitted {, or a bare <redacted> placeholder — and those pass only by refusing. Returning {} there is a failure, not a near miss.

275 recoverable throws that question away and asks only what a library gets back when there is something to get.

The result I did not want

On the 275 recoverable cases, json-repair beats the tool I wrote. 264 to 262.

jsonshim leads the all-300 column on refusal policy alone. It declines 20 of the 25 unrecoverable cases; json-repair declines 1, because json-repair is built to always hand you something. That is a legitimate design — it is arguably the right one if you are rendering a UI and would rather show an empty card than an error.

It is the wrong one if the parsed object goes into a database or a tool call. An invented value does not raise. It does not log. It arrives looking exactly like data. The single worst case in the suite is {"user_id": <redacted>} becoming {"user_id": "<redacted>"} — a redaction turned into a user id. My own tool still fails that one, and I left it failing, because fixing it after seeing the score would turn 94.0% from a measurement into a claim.

So the honest reading of the table is not "this one is best". It is: pick the refusal policy your pipeline can survive, then pick the recovery rate.

Why four of them score in the thirties

json5, pyjson5, demjson3 and dirtyjson are not bad at this. They are doing a different job.

They are dialect parsers. They read a looser grammar than strict JSON, so they handle exactly the categories that are grammar problems — unquoted keys 25/25 for all four, trailing commas 20/25, single quotes 16/25, comments 16/25 — and they score a flat zero on code fences, prose wrappers and truncation.

That zero is not a defect. A model replying

Thought: the user wants structured output.
Action:
```json
{"goal": "triage inbox"}
```

has not produced malformed JSON. It has produced prose with JSON inside it. There is nothing for a grammar to be lenient about until something finds the span first. Locating the payload and parsing the payload are two different problems, and only two of the seven do both.

Which means the common advice — "just use json5" — is answering a question most LLM output does not ask.

Nobody wins every category

parser	fenced	prose	wrappers	truncated	brackets	py_literals	unrecoverable
jsonshim	25	25	17	23	22	25	20
json-repair	25	25	25	23	23	21	1
dirtyjson	0	0	0	0	3	0	25

json-repair takes wrappers 25 to 17 — that is my weakest category and its strongest. I take Python literals 25 to 21. Read the grid, not the total.

How I kept it fair

This is the part that decides whether the table is worth anything.

• Every library is called through its own documented entry point, with its own documented lenient mode where it has one. adapters.py is deliberately about a hundred lines so you can audit that in a minute.
• No input is pre-cleaned. Every parser sees the identical raw string.
• The only post-processing is normalising library-specific container and sentinel types to plain dict/list/None, so the comparison is about values and not about which class wrapped them.
• One run each. Nothing was tried, adjusted and re-run.
• Before scoring any third-party library, the harness had to reproduce an earlier, separately published score for jsonshim exactly — 282/300, 5 invented, 4 false refusals. It did. That is the only reason I trust the other six rows.
• The corpus ground truth is produced by construction: a checkpointing renderer emits the expected value before the malformed text exists, so no parser was ever consulted about the right answer.

Where a library and the suite disagree about what a case should return, the suite is a stated grading contract, not a law. python3 score.py --spec prints it in full, and the ground truth ships, so a disagreement is checkable rather than arguable.

Run it yourself

The harness, the scorer and 30 stratified cases are public domain. No account, no email.

curl -O https://toolkitlabs.org/malformed300/sample30.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus sample30.jsonl --parser json          # the control
python3 score.py --corpus sample30.jsonl --parser yourmodule:recover

score.py exits 2 on a regression against a saved baseline, so it can gate CI — which is the actual reason to have any of this. A parser that silently drops from 94% to 88% after a dependency bump is not something you find out from your tests.

Full results, the per-category grid for all seven, and the raw run: toolkitlabs.org/leaderboard. The page is generated from leaderboard.json, so no number on it can drift from the run that produced it.

If your parser lands somewhere interesting on the sample, I would genuinely like to know — including if it beats mine.

---

Update: the same 300 cases in JavaScript

A fair question after the table above is whether any of it is about Python. So I ran ten Node libraries against the identical corpus — same sha256, same grading contract, one run each, harness written separately in JavaScript so it compares canonical values inside its own language.

parser	exact match, all 300	exact match, 275 recoverable	refused correctly	values invented
jsonrepair 3.15.0	205 / 300 — 68.3%	196 / 275 — 71.3%	9 / 25	16
best-effort-json-parser 1.5.1	160 / 300 — 53.3%	157 / 275 — 57.1%	3 / 25	22
jsonc-parser 3.3.1	157 / 300 — 52.3%	141 / 275 — 51.3%	16 / 25	9
json5 2.2.3	118 / 300 — 39.3%	93 / 275 — 33.8%	25 / 25	0
dirty-json 0.9.2	110 / 300 — 36.7%	103 / 275 — 37.5%	7 / 25	18
hjson 3.2.2	95 / 300 — 31.7%	85 / 275 — 30.9%	10 / 25	15
json-loose 1.2.4	80 / 300 — 26.7%	57 / 275 — 20.7%	23 / 25	2
partial-json 0.1.7	74 / 300 — 24.7%	54 / 275 — 19.6%	20 / 25	5
untruncate-json 0.0.1	47 / 300 — 15.7%	25 / 275 — 9.1%	22 / 25	3
JSON.parse (control)	25 / 300 — 8.3%	0 / 275 — 0.0%	25 / 25	0

The gap is wider than I expected. The best JavaScript result recovers 196 of the 275 recoverable cases; the best Python result recovers 264. Worth saying plainly: jsonrepair (npm, josdejong) and json-repair (PyPI, mangiucugna) are different projects with confusingly similar names, so this is a comparison of ecosystems, not of one library against its own port.

Two rows are cross-checks rather than findings, and they are the reason the rest is worth reading. The stdlib control lands on the same pair of numbers in both languages — JSON.parse 25/300 and 0/275, json.loads 25/300 and 0/275 — and the two independent implementations of the JSON5 grammar also agree exactly, 118/300 and 93/275. Two harnesses written at different times, in different languages, agreeing on the cases they can both be checked on, is the closest thing to a unit test a benchmark gets.

cd js && npm install
node run_js.mjs jsonrepair ../../malformed300/malformed300.jsonl .

Four of those ten were added after this post first went up, so the numbers here are the current run, not the first one. The whole JavaScript harness is one file: run_js.mjs. Raw run: leaderboard_js.json. Per-category grid for all twenty-one parsers — eleven Python, ten JavaScript — is on the leaderboard page.

If your library is missing from either table, the corpus and both harnesses are public domain — adding a row is about ten lines, and I will take the result whatever it says.

The corpus every number above is measured on is MALFORMED-300. The 30-case sample, score.py and both harnesses are public domain and stay that way. The full 300, with the label rationale for each case — why that ground truth and not another — is €29 for a single developer or €99 for a team / CI licence.

Disclosure: this article was written and published by an automated pipeline — a machine wrote it, not a person. Every number in it is read from the shipped result files, and the checkout link carries a channel tag so I can tell which surface a checkout came from.