The Model's JSON Was Almost Valid. I Made It Grade Its Own Homework for 48 Hours.

Every extraction pipeline I have ever pointed at a language model shares the same dirty secret: the JSON comes back almost valid. Almost is where the bugs live, because almost passes your eyes and then fails your schema at midnight. So I built a loop where the model grades its own homework, then let it run for 48 hours on a free server to see what breaks.

The experiment

The idea was simple: take plain-text payloads that look like webhook bodies, extract five fields against a small schema, and give the model exactly one chance to fix its own mistakes. I wrote the rules down before writing any code, because rules written after a failure are just excuses.

1. Pass one asks the model to return the fields as JSON.
2. A validator checks the result against the schema.
3. If validation fails, pass two sends the original payload, the bad JSON, and the exact validation errors back to the model.
4. Every attempt, raw text included, lands in a JSONL log.

I ran that loop for 48 hours on MonkeyCode's free server option, using its free model access for both passes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Here is the loop, trimmed to the parts that mattered.

import hashlib
import json
import time
from datetime import datetime, timezone

import jsonschema
import requests

SCHEMA = {
    "type": "object",
    "required": ["event", "customer_id", "amount", "currency"],
    "properties": {
        "event": {"type": "string", "enum": ["charge.succeeded", "charge.failed"]},
        "customer_id": {"type": "string", "pattern": "^cus_"},
        "amount": {"type": "integer", "minimum": 0},
        "currency": {"type": "string", "minLength": 3, "maxLength": 3},
    },
}

SEEN: set[str] = set()

def now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()

def call_model(prompt: str) -> str:
    # Point this at the free model endpoint you are testing.
    resp = requests.post(
        "https://your-endpoint.example/v1/chat/completions",
        json={
            "model": "your-free-model",
            "messages": [{"role": "user", "content": prompt}],
        },
        timeout=90,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def first_object(text: str):
    text = text.strip()
    if text.startswith("```

"):
        text = text.split("\n", 1)[1].rsplit("

```", 1)[0]
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end == -1:
        return None
    try:
        return json.loads(text[start : end + 1])
    except json.JSONDecodeError:
        return None

def validate(obj):
    return sorted(
        jsonschema.Draft7Validator(SCHEMA).iter_errors(obj),
        key=lambda err: str(err.path),
    )

def append_log(entry: dict) -> None:
    with open("attempts.jsonl", "a") as fh:
        fh.write(json.dumps(entry) + "\n")

def process(text: str) -> dict:
    digest = hashlib.sha256(text.encode()).hexdigest()
    if digest in SEEN:
        return None  # already handled after a redelivery
    SEEN.add(digest)

    first = call_model(
        f"Return one JSON object with fields {list(SCHEMA['properties'])}. Payload: {text}"
    )
    obj = first_object(first)
    errors = validate(obj) if obj else ["no JSON object found"]
    append_log({"ts": now_iso(), "digest": digest, "pass": 1, "ok": not errors, "raw": first})

    if not errors:
        return obj

    second = call_model(
        f"Your JSON failed validation with: {[str(e) for e in errors]}. "
        f"Return only corrected JSON. Original payload: {text} Previous JSON: {first}"
    )
    obj2 = first_object(second)
    append_log({"ts": now_iso(), "digest": digest, "pass": 2, "ok": obj2 is not None, "raw": second})
    if obj2 is not None:
        return obj2
    raise ValueError(f"both passes failed for {digest}")

def run(queue):
    consecutive_failures = 0
    for text in queue:
        try:
            process(text)
            consecutive_failures = 0
        except requests.HTTPError as exc:
            consecutive_failures += 1
            append_log({"ts": now_iso(), "http_error": str(exc)})
            if consecutive_failures >= 3:
                print(f"breaker open at {now_iso()}, sleeping 60s")
                time.sleep(60)
                consecutive_failures = 0

The run() function takes any iterable of raw strings; mine was a file of webhook-shaped payloads. I started it with python two_pass_extractor.py and watched the failures with tail -f attempts.jsonl | jq -c 'select(.ok == false)'.

The failure log

The log is the real artifact, because the failures only became visible once I could read every raw response in order. What broke, you ask? Five patterns stood out.

1. Markdown fences appeared far more often than I expected. The model wrapped its answer in fenced json code blocks often enough that my first naive json.loads() rejected a painful number of responses. The fix was the small normalizer you see above, but the real lesson was simpler: log the raw text, or you will never see the pattern at all.

2. The second pass echoed the first pass's mistakes. For the first few hours my correction prompt just said "fix it", and the model happily returned the same invalid JSON. Once I started pasting the actual validator errors into the prompt, the second pass stopped repeating itself. A generic instruction is not feedback; the schema errors are the feedback.

3. Redeliveries looked like duplicates. The server hiccuped once during the window, the queue redelivered a batch, and my content-hash dedupe quietly turned a potential double-processing bug into a no-op. Idempotency keys are not optional once anything can restart, and on a free server anything can restart.

4. Timestamps disagreed. The server's clock and the API's timestamps drifted apart enough that ordering by response time was misleading, so I logged my own UTC timestamp at the moment of each attempt. When you are reading a 48-hour log at 2 AM, you want one clock, not two.

5. The circuit breaker worked, but it was noisy. Three consecutive HTTP errors opened a 60-second cooldown, which was the right call, yet I logged every raw exception and the log became hard to read. Next time I will log the breaker opening and closing instead of every individual failure.

What I'd keep

If I had to restart the same experiment tomorrow, what would survive the cut? Four decisions.

• Schema-first validation. The validator is the source of truth; the model is a suggestion engine.
• One correction pass, never more. A second retry rarely fixed what the first, better-informed pass could not, and every extra pass multiplied cost and noise. This is not a retry loop; it is a second pass with better information.
• The JSONL log with raw text. Without it, the fence pattern and the echoed mistakes would have stayed invisible.
• The content-hash dedupe. It cost almost nothing and absorbed the redelivery storm.

Where this falls apart

The loop is useful, but it is not a universal hammer, and I would be lying if I said it fits every job.

• Large or deeply nested schemas. I tested a deliberately small schema, and a bigger one stresses the correction pass differently, so run your own soak test before trusting it.
• Latency-sensitive callers. A two-pass loop on a free server is a correctness tool, not a throughput tool; I measured neither latency nor concurrency.
• Zero-tolerance workloads. If you cannot accept the occasional item where both passes fail, route those items to a human review queue instead of raising an exception like my script does.
• Anything permanent. MonkeyCode's free model access and free server worked for this experiment, but availability and behavior can change, so keep a fallback path and check the current docs before building on top of them.

The most useful thing I built this week was not the extractor; it was the log that told me what the extractor was doing wrong. If you keep field notes like this, I would genuinely like to hear what your 48 hours taught you.