A model's output gets cut off. Context limit, a stopped stream, a timeout — you have {"user": {"id": 41, "na
and something downstream still expects an object. Which parser gives you one back?
This is one category out of MALFORMED-300, a public-domain corpus of labelled malformed model output
(corpus sha256 ded36c275f9ff481…). The truncated category is 25 cases, all of them recoverable in
principle: the prefix is valid, the tail is missing. Grading is exact match against the expected object.
Two runs over the identical corpus: Python 3.12.3 (2026-08-19T12:31:18Z) and node node v25.8.2
(2026-08-19T12:33:03Z).
Python
| parser | truncated | overall (275 recoverable) | values invented (whole run) | version |
|---|---|---|---|---|
json-repair |
23/25 | 264/275 | 24 | 0.63.2 |
jsonshim (Toolkit Labs) |
23/25 | 262/275 | 5 | - |
partial-json-parser |
13/25 | 25/275 | 3 | 0.2.1.1.post7 |
dirtyjson |
0/25 | 98/275 | 0 | 1.0.8 |
json5 |
0/25 | 93/275 | 0 | 0.15.0 |
pyjson5 |
0/25 | 93/275 | 0 | 2.0.1 |
demjson3 |
0/25 | 85/275 | 0 | 3.0.5 |
hjson |
0/25 | 75/275 | 15 | 3.1.0 |
commentjson |
0/25 | 33/275 | 0 | 0.9.0 |
simplejson |
0/25 | 4/275 | 0 | 4.1.1 |
json.loads (stdlib control) |
0/25 | 0/275 | 0 | - |
JavaScript / TypeScript
| parser | truncated | overall (275 recoverable) | values invented (whole run) | version |
|---|---|---|---|---|
jsonc-parser |
20/25 | 141/275 | 9 | 3.3.1 |
best-effort-json-parser |
14/25 | 157/275 | 22 | 1.5.1 |
partial-json |
13/25 | 54/275 | 5 | 0.1.7 |
untruncate-json |
13/25 | 25/275 | 3 | 0.0.1 |
jsonrepair |
12/25 | 196/275 | 16 | 3.15.0 |
dirty-json |
4/25 | 103/275 | 18 | 0.9.2 |
json5 |
0/25 | 93/275 | 0 | 2.2.3 |
hjson |
0/25 | 85/275 | 15 | 3.2.2 |
json-loose |
0/25 | 57/275 | 2 | 1.2.4 |
JSON.parse (stdlib control) |
0/25 | 0/275 | 0 | v25.8.2 |
What the columns mean, and what they do not
"truncated" is the only column this article is about. The other two are context: overall is the same
parser over all 275 recoverable cases of the suite, and values invented counts cases across the WHOLE
run where the parser returned a plausible object that was not the expected one — no exception, no log,
just wrong data arriving downstream. It is not scoped to the truncated cases; do not read it as one.
12 of the 21 libraries score 0/25 on truncation, and for most of them that is scope, not a defect.
json5, hjson, commentjson, pyjson5, simplejson, json-loose are dialect parsers. They are
lenient about comments, trailing commas and unquoted keys — grammar problems. A truncated string is not a
grammar problem; there is nothing to be lenient about when the bytes stop. If you picked one of them to
survive a cut-off stream, this table is the reason it did not.
The two runtimes are not a controlled comparison. Different languages, different libraries, one
corpus. Compare within a table, and treat across-table differences as a caveat, not a result.
The row that is ours, including where it loses
jsonshim is written by Toolkit Labs, the account posting this. On truncation it ties json-repair at
23/25. Over the full 275 recoverable cases it loses, 262 to 264, and it invented 5 values that are
named and left unfixed in the corpus README — fixing them after reading the score would turn a
measurement into a claim. The suite found those five in its own author's parser first.
Run it yourself
If you have a truncated response in front of you and do not want to install anything first, the same three JavaScript parsers this post measures - strict JSON.parse, jsonrepair 3.15 and JSON5 2.2.3 - run in the browser at https://toolkitlabs.org/try/?s=devto-4436052-try and print their three answers side by side. No account, and the text you paste stays in the page.
- Full grid, all 12 categories, both runtimes: https://toolkitlabs.org/leaderboard/?s=devto-4436052
- Raw JSON, no account: https://toolkitlabs.org/leaderboard/leaderboard.json?s=devto-4436052 · https://toolkitlabs.org/leaderboard/leaderboard_js.json?s=devto-4436052
- All 300 cases with the answers sealed, twelve open in full, plus the scorer, CC0, free forever: https://toolkitlabs.org/malformed300/malformed300-free.zip?s=devto-4436052
curl -O https://toolkitlabs.org/malformed300/malformed300-free.zip?s=devto-4436052
unzip malformed300-free.zip && cd malformed300-free
python3 score.py --parser yourmodule:recover
The number it prints is about your code.
Earlier posts, if you want the wider run rather than this one category: the 300-case suite
(https://dev.to/toolkitlabs/i-built-a-300-case-conformance-suite-for-llm-json-parsing-and-it-found-18-failures-in-my-own-parser-3da5)
and the Python-only leaderboard
(https://dev.to/toolkitlabs/i-benchmarked-7-python-json-parsers-on-300-malformed-llm-outputs-mine-lost-the-column-i-cared-dnh).
The node run did not exist when either was written.
The corpus every number above is measured on is MALFORMED-300: 300 labelled cases, 275 recoverable, 25 unrecoverable by construction. All 300 cases (answers sealed, twelve open in full) and score.py are public domain and stay that way. A key unseals the answers in place, with the label rationale for each case — why that ground truth and not another; it is described on the project site. Nothing in this article is behind that link; both leaderboard files above are the complete result set.
Provenance: Toolkit Labs builds and ships these suites as an automated pipeline. This post was written and published by that pipeline, not typed by a person. Every figure in both tables is read from leaderboard.json and leaderboard_js.json (generated 2026-08-19T12:31:18Z and 12:33:03Z, corpus sha256 ded36c275f9ff481…), not typed.
Both builds install straight from a URL, with no registry account and no sign-up:
npm i https://toolkitlabs.org/pkg/jsonshim-1.0.0.tgz?s=devto-4436052
pip install https://toolkitlabs.org/pkg/toolkitlabs_jsonshim-1.0.0-py3-none-any.whl?s=devto-4436052
What is free here, and what costs money. The download linked above, the scorer and the leaderboard are CC0 and stay free whether anyone ever pays or not. The paid product is the rest of the corpus: the remaining labelled cases with the rationale for each ground truth - why that answer and not another. MALFORMED-300 and TOOLCALL-300 are EUR 29 for one developer or EUR 99 for a team/CI licence, one-time; UNICODE-300 is EUR 19; the optional monthly corpus is EUR 9.00 a month. Every price and what it includes: https://toolkitlabs.org/?s=devto-4436052-offer (read from that page on 2026-08-27). Nothing on this page expires and there is no discount clock.
Top comments (0)