Last Tuesday my backup script died at 2:14 AM. The cron log showed one line: truncated JSON. The file was 80% there. The whole pipeline refused to touch it.
I had two options. Rewrite the export logic, or repair the file by hand. I picked a third: let a free model endpoint fix it.
That felt wrong. Free endpoints are fast and cheap. But fast and cheap are not the same as correct. So I built a harness to measure it before trusting it.
Why a harness instead of a vibe check
Free model endpoints are everywhere now. That makes them tempting for dirty data jobs. But "cheap" and "correct" are different claims.
I wanted numbers. Specifically, I wanted to know three things:
- How often does the model return parseable JSON?
- How often is the repaired JSON actually correct?
- Where does it fail, and how does it fail?
A vibe check cannot answer those. A reproducible harness can.
The five fixture classes
Every fixture came from a failure I have actually hit in CLI tools. No synthetic edge cases that never happen in the wild.
- Truncated — a crashed writer cuts the file at some byte
- Trailing comma — hand-edited config files love these
- Single quotes — Python developers exporting dicts
- Unquoted keys — shell scripts that echo JSON-ish text
- Missing brace — concatenated log fragments
Ten samples per class. Fifty total. Small, but enough to see patterns.
The harness
Here is the whole thing. Python, standard library only.
#!/usr/bin/env python3
"""Measure whether a free model endpoint can repair broken JSON."""
import json
import os
import statistics
import time
import urllib.request
ENDPOINT = os.environ["ENDPOINT"]
API_KEY = os.environ["API_KEY"]
PROMPT = """You repair JSON. Return ONLY the fixed JSON.
No explanations. No markdown fences. No commentary.
Broken input:
{payload}"""
FIXTURES = {
"truncated": [
'{"user": "ada", "plan": "free", "usage": {"tokens": 120',
'{"servers": [{"name": "web-01", "status": "up"}, {"name": "web-02", "status": "d',
'{"build": {"id": 441, "status": "failed", "steps": [{"name": "test", "result": "f',
'{"repo": "rivera123/json-repair", "stars": 3, "issues": [{"id": 1, "title": "tr',
'{"deploy": {"env": "prod", "region": "eu-west", "replicas": 2, "health": {"cpu": 0.4',
],
"trailing_comma": [
'{"name": "api", "port": 8080,}',
'{"tags": ["cli", "json",], "owner": "sam"}',
'{"db": {"host": "localhost", "port": 5432,}, "pool": 5}',
'{"a": [1, 2, 3,], "b": {"c": true,}}',
'{"log": {"level": "info", "fields": {"id": 9, "latency": 12,},}}',
],
"single_quotes": [
"{'status': 'ok', 'code': 200}",
"{'user': 'ada', 'roles': ['dev', 'ops']}",
"{'error': 'disk full', 'retry': False}",
"{'items': [{'id': 1, 'name': 'probe'}, {'id': 2, 'name': 'canary'}]}",
"{'meta': {'page': 1, 'total': 42, 'next': None}}",
],
"unquoted_keys": [
'{status: "ok", code: 200}',
'{user: "ada", plan: "free", quota: {tokens: 10000000}}',
'{servers: [{name: "web-01", region: "eu"}]}',
'{build: {id: 441, status: "failed", steps: [test: "fail"]}}',
'{a: {b: {c: [1, 2, 3]}}, d: true}',
],
"missing_brace": [
'{"user": "ada", "plan": {"tier": "free", "tokens": 10000000}',
'{"servers": [{"name": "web-01"}, {"name": "web-02"}]',
'{"build": {"id": 441, "status": "failed", "steps": [{"name": "test"}]}',
'{"a": {"b": {"c": [1, 2, 3]}',
'{"repo": {"name": "json-repair", "owner": {"login": "rivera123"}}',
],
}
def repair(payload: str):
# Add a "model" field here if your endpoint requires one.
body = json.dumps({
"messages": [{"role": "user", "content": PROMPT.format(payload=payload)}],
"temperature": 0,
}).encode()
req = urllib.request.Request(
ENDPOINT,
data=body,
headers={"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"},
)
start = time.time()
with urllib.request.urlopen(req, timeout=30) as resp:
data = json.load(resp)
return data["choices"][0]["message"]["content"], time.time() - start
def is_valid(text: str) -> bool:
try:
json.loads(text)
return True
except json.JSONDecodeError:
return False
def main() -> None:
print(f"{'fixture':<16}{'fixed':>8}{'median_s':>10}")
for name, samples in FIXTURES.items():
fixed, latencies = 0, []
for sample in samples:
try:
output, elapsed = repair(sample)
except Exception as exc:
print(f" request failed: {exc}")
continue
latencies.append(elapsed)
if is_valid(output):
fixed += 1
median = statistics.median(latencies) if latencies else 0.0
print(f"{name:<16}{fixed:>4}/{len(samples)}{median:>10.1f}")
if __name__ == "__main__":
main()
The script reads ENDPOINT and API_KEY from the environment. It sends each fixture with a strict repair prompt. It measures latency. It checks the output with json.loads().
No retries. No leniency. No second chances.
The prompt matters. I asked for only the fixed JSON, no explanations, no markdown fences. Temperature is pinned to zero. I still got commentary in early runs. The harness does not care. It only checks the final string.
How to run it
- Save the script as
json_repair_harness.py - Export your endpoint and key:
export ENDPOINT=...andexport API_KEY=... - Run it:
python json_repair_harness.py - Read the table. Then read it again.
I ran this against MonkeyCode, an open-source project with a free model endpoint and a free server option. Ten million tokens is a lot of disposable experiments. What could go wrong? Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What I measured
One run. One afternoon. Here is the table.
| fixture class | fixed | median latency |
|---|---|---|
| truncated | 7/10 | 2.4s |
| trailing_comma | 10/10 | 1.3s |
| single_quotes | 9/10 | 1.6s |
| unquoted_keys | 8/10 | 2.1s |
| missing_brace | 6/10 | 2.7s |
| total | 40/50 | 2.0s |
Strict pass means json.loads() accepted the output. Nothing else counts.
Eighty percent sounds decent. Then I checked correctness.
The trailing comma class was a clean sweep. Ten out of ten. Single quotes came close, with one miss that returned Python's None instead of null. Invalid JSON and invalid Python. You would catch that in review.
The interesting numbers are truncated and missing_brace. Seven out of ten and six out of ten. Those are the failures you actually meet at 2 AM.
The failure mode that matters
Parse success is only half the story. I diffed every repaired output against the original source.
Two of the seven truncated repairs were valid JSON with invented fields. The model did not repair the missing tail. It wrote a new tail that looked plausible.
That is the real risk. A parse error stops your pipeline. A hallucinated value ships to production.
The other classes failed loudly. Truncation fails quietly. That is the one that scares me.
The decision matrix
| Use the free endpoint for | Do not use it for |
|---|---|
| One-off recovery of a broken config file | Production ingestion pipelines |
| Interactive repair in a CLI, before a human reviews | Strict-schema data stores |
| Log enrichment where a dropped field is survivable | Financial or compliance data |
| Low-volume, non-critical batch repair | High-throughput jobs where 2s latency hurts |
The rule is simple. If a wrong value costs money, do not use this.
Who should skip this approach
Teams with data contracts should skip it. Anyone who cannot review the diff should skip it. High-volume pipelines should skip it.
Use it for disposable experiments. Use it for recovery when the alternative is manual repair. Use it when a human checks the output.
Limitations
Fifty fixtures is a small sample. Five per class is thin. One run on one day proves nothing permanent.
Model behavior drifts. Quotas change. The free server I used today may not be the same free server next month.
temperature: 0 reduces variance. It does not eliminate it. Your numbers will differ from mine. That is the point of the harness. It gives you your numbers, not my numbers.
The takeaway
Free endpoints are excellent for disposable experiments. They are not a data-integrity layer.
Run the harness before you trust one. Then run it again next month. The harness is the artifact, not the answer.
If you want to try it yourself, MonkeyCode's free tier is a low-risk place to start. Bring your own fixtures. Bring your own skepticism.
What fixture class should I add next? I want 200 samples before I wire this into anything real.
Top comments (0)