When a model is asked for JSON and returns prose, fences, or a half-cut object, most repair libraries pick something and hand it back. On the 25 cases in MALFORMED-300 where the honest answer is "no value exists", jsonshim-mcp invents 0 of them. It also refuses 55 recoverable documents it could have salvaged. Read those two numbers together.
This is not production traffic. It is 300 synthesised malformed JSON strings in 12 categories (25 each), each with ground truth, scored by the same score.py used on the parser leaderboard.
What it is
jsonshim-mcp is a one-file MCP stdio server (stdlib only, CC0) exposing two tools to an agent:
-
recover_json— pull JSON out of a model reply, report every repair, return an error with no value when recovery is impossible -
classify_json_failure— name the failure mode without guessing a value
Point your MCP client at it:
{"mcpServers":{"jsonshim":{"command":"python3","args":["/absolute/path/jsonshim_mcp.py"]}}}
The numbers
MALFORMED-300 parser: jsonshim-mcp
cases 300
exact match 218 / 300 (72.7%)
recoverable recovered 193 / 275
unrecoverable invented 0 / 25
false refusals 55
For comparison on the same corpus: json.loads scores 8.3% (refuses everything). json-repair recovers 264 of 275 recoverable cases but invents a value on 5 of the 25 unrecoverable ones. The standalone jsonshim library scores 282/300 exact but shares that same 5-invention blind spot.
jsonshim-mcp is not the same code as jsonshim. It trades recovery rate for a harder refusal contract.
Category by category
| category | recovered |
|---|---|
| comments | 25/25 |
| fenced | 25/25 |
| prose_wrapped | 25/25 |
| single_quotes | 25/25 |
| trailing_comma | 25/25 |
| unquoted_keys | 25/25 |
| unrecoverable | 25/25 (refused, never invented) |
| py_literals | 21/25 |
| wrappers | 17/25 |
| brackets | 5/25 |
| truncated | 0/25 |
| raw_control | 0/25 |
Two flat zeros, not hidden: truncated output and raw control characters inside strings both come back refused every time. If your pipeline depends on salvaging half-written streaming JSON, this is the wrong tool.
Free files vs the labelled corpus
The server and scorer are free (CC0). 12 open sample cases ship in the free zip; the other 288 answers are sealed.
The full 300 labelled cases with rationale per label — the file you need to regression-test your own parser against — are €29 single developer / €99 team-CI: buy link on the product page. Instant unlisted download after Stripe payment; no account, no email gate.
Numbers are the raw scorer output on the corpus as shipped. In-sample where noted. Machine-readable catalog.
Top comments (0)