DEV Community

Tamer Kalla
Tamer Kalla

Posted on

Make LLM JSON Output Boring: Schema Validation and Repair in Python

The problem

Every LLM feature I have shipped that returns JSON has failed in the same unglamorous ways. The model wraps the object in markdown fences even though the prompt said not to. It leaves a trailing comma after the last field. It writes strings with single quotes, Python-style. It invents an enum value ("urgent", when the allowed set is low, medium, high) or returns a confidence of 1.4 on a zero-to-one scale. Then json.loads throws, or worse, nothing throws and a bad value flows downstream.

None of this is exotic, and none of it is really a model problem. It is a trust problem. Raw model output is the least trusted input in your system: produced by a stochastic process, influenced by whatever the user typed, and liable to change shape the day your provider swaps a model version. Treating it like trusted input, with a bare json.loads and a hopeful except that retries the identical prompt, is how you get paged.

The fix is not a framework. It is a small layer with an explicit policy: what gets repaired, what gets sent back to the model, and what gets rejected. This article builds one in dependency-free Python.

The approach

Three layers, applied in order.

First, parse defensively: try the raw text exactly as returned. Second, repair syntax, in a bounded and logged way: a short, ordered list of purely syntactic fixes, each one recorded. Third, validate semantics against a hand-rolled schema: required keys, types, enums, numeric ranges. If anything still fails, re-ask the model once, with the specific errors in the follow-up prompt. If that fails too, return a clean failure object. The layer never raises on model output, and a failed outcome never carries partial data.

One rule holds the design together: repairs fix how the JSON is written, never what it means. A repair may strip a fence. It may not turn "urgent" into "high", or clamp 1.4 down to 1.0. Semantic problems go back to the model, or they get rejected. They never get quietly regexed into plausibility.

Step 1: Defensive parsing and bounded repair

_REPAIRS = [
    ("stripped markdown code fence", _strip_fences),
    ("removed trailing comma(s)", _remove_trailing_commas),
    ("converted single-quoted strings to double-quoted", _fix_single_quotes),
]


def parse_json(raw: str) -> tuple[dict | None, list[str], str | None]:
    """Returns (data, repairs_applied, error). Exactly one of
    data / error is set."""
    text = raw.strip()
    try:
        data = json.loads(text)
        if isinstance(data, dict):
            return data, [], None
        return None, [], f"expected a JSON object, got {type(data).__name__}"
    except json.JSONDecodeError as exc:
        last_error = str(exc)

    repairs: list[str] = []
    for label, fix in _REPAIRS:
        fixed = fix(text)
        if fixed == text:
            continue
        text = fixed
        repairs.append(label)
        try:
            data = json.loads(text)
        except json.JSONDecodeError as exc:
            last_error = str(exc)
            continue
        if isinstance(data, dict):
            return data, repairs, None
        return None, repairs, f"expected a JSON object, got {type(data).__name__}"
    return None, repairs, last_error
Enter fullscreen mode Exit fullscreen mode

Repairs are cumulative, a parse is attempted after each change, and the order runs safest first: fence stripping changes nothing inside the object, trailing-comma removal touches structure, single-quote conversion is the bluntest tool and goes last. The repair log travels with the result. A parse that needed two repairs is accepted, but it is not silent, and that distinction matters (more below).

Step 2: A hand-rolled schema validator

Why hand-rolled: the schema format is twenty lines of convention, it adds no dependency, and the error strings are written for two audiences at once, your logs and the model that will read them during the re-ask.

def validate(data: dict, schema: dict) -> list[str]:
    """Schema shape: {field: {"type": ..., "required": bool,
    "enum": [...], "min": number, "max": number}}.
    Returns human-readable errors, empty when valid."""
    errors: list[str] = []
    for name, spec in schema.items():
        if name not in data:
            if spec.get("required", False):
                errors.append(f"{name}: missing required key")
            continue
        value = data[name]
        expected = spec.get("type")
        if expected and not _TYPE_CHECKS[expected](value):
            errors.append(f"{name}: expected {expected}, got {type(value).__name__}")
            continue
        if "enum" in spec and value not in spec["enum"]:
            errors.append(f"{name}: {value!r} not in {spec['enum']}")
        if isinstance(value, (int, float)) and not isinstance(value, bool):
            if "min" in spec and value < spec["min"]:
                errors.append(f"{name}: {value} below min {spec['min']}")
            if "max" in spec and value > spec["max"]:
                errors.append(f"{name}: {value} above max {spec['max']}")
    return errors
Enter fullscreen mode Exit fullscreen mode

Step 3: The re-ask loop

def extract(prompt: str, model_fn, schema: dict, max_reasks: int = 1) -> Outcome:
    outcome = parse_and_validate(model_fn(prompt), schema)
    attempts = 1
    while not outcome.ok and attempts <= max_reasks:
        retry_prompt = (
            f"{prompt}\n\n"
            "Your previous answer was not valid JSON for the required schema. "
            "Problems found:\n"
            + "\n".join(f"- {e}" for e in outcome.errors)
            + "\nReply with the corrected JSON object only. No prose, no code fences."
        )
        outcome = parse_and_validate(model_fn(retry_prompt), schema)
        attempts += 1
    outcome.attempts = attempts
    return outcome
Enter fullscreen mode Exit fullscreen mode

One re-ask, not a loop until success. A re-ask costs a call and doubles your latency on the failure path, and a model that still fails with the errors spelled out is telling you something a third attempt rarely fixes. Note that the second answer is validated with exactly the same strictness as the first. A re-ask is a fresh generation, not an edit, and it can fix the enum while breaking a field that was fine.

Step 4: Run it

The demo (run_json_guard.py, full code with json_guard.py) extracts a support ticket against this schema, using stub models so it runs with no API key:

TICKET_SCHEMA = {
    "summary": {"type": "str", "required": True},
    "priority": {"type": "str", "required": True, "enum": ["low", "medium", "high"]},
    "confidence": {"type": "float", "required": True, "min": 0.0, "max": 1.0},
    "tags": {"type": "list", "required": False},
}
Enter fullscreen mode Exit fullscreen mode

Four scenarios: a clean answer, a fenced and single-quoted answer with a trailing comma, a stub whose first answer breaks the schema twice and corrects itself on the re-ask, and a stub that answers in prose no matter what. Actual output:

--- Scenario 1: clean parse ---
status: OK
attempts: 1
repairs: none
data: {"confidence": 0.9, "priority": "high", "summary": "Export fails on large files", "tags": ["export"]}

--- Scenario 2: fenced, single-quoted, trailing comma ---
status: OK
attempts: 1
repairs: ['stripped markdown code fence', 'removed trailing comma(s)', 'converted single-quoted strings to double-quoted']
data: {"confidence": 0.92, "priority": "high", "summary": "Export fails on large files", "tags": ["export"]}

--- Scenario 3: schema failure, then re-ask ---
first attempt errors:
  - priority: 'urgent' not in ['low', 'medium', 'high']
  - confidence: 1.4 above max 1.0
after re-ask: OK (attempts: 2)
data: {"confidence": 0.88, "priority": "high", "summary": "Export fails on large files", "tags": ["export"]}

--- Scenario 4: unrecoverable, prose on both attempts ---
status: FAILED
attempts: 2
repairs: none
errors:
  - unparseable JSON: Expecting value: line 1 column 1 (char 0)

scenario 4 note: failure returned as data, no exception raised
scenarios behaved as designed: 4/4
Enter fullscreen mode Exit fullscreen mode

Read scenario 2 closely, because it is the whole argument. The data was accepted, and the repair log says exactly what had to be forgiven to accept it. Scenario 3 shows the boundary working: nothing was syntactically wrong, so no repair applied, and the semantic errors went back to the only component that can fix semantics, the model. Scenario 4 is a failure the caller can route, log, and alert on, instead of a traceback.

The tradeoffs: repair, re-ask, or fail

Repair when the flaw is purely presentational and the field's meaning cannot change: fences, whitespace, a trailing comma in a display summary. Re-ask when the value is semantic but the action it drives is reversible and a second call is affordable: classifications, summaries, draft extractions. Fail, cleanly and loudly, when the field drives something irreversible or expensive: a payment amount, a permission change, anything a human should see before it happens. In that third case a schema-valid answer can still be factually wrong, and a dead-letter queue is cheaper than a confident wrong write.

Silent repair deserves its own warning, because it is the default in most quick implementations. These repairs operate on raw text, not on parsed structure, so they cannot tell a trailing comma that is syntax from one inside a string value, and the single-quote fixer cannot tell a quote that delimits a string from an apostrophe inside one. On a summary field, the worst case is a mangled sentence. On an identifier, an email address, or a dosage, a "fixed" value can be a different value, and nothing downstream will ever know. That is why every repair here is logged, why repairs stop at syntax, and why, in stricter setups, I mark sensitive fields no-repair in the schema: if they arrive malformed, the outcome fails and a human looks. The log is the audit trail that makes any of this defensible.

Real gotchas

bool is an int in Python. isinstance(True, int) is true, and True compares equal to 1, so a naive type check waves a boolean through an int field and even through a max range check. The type table in json_guard.py excludes bool explicitly for the int and float checks. This one bites in the wild as a model answering "confidence": true.

A rising repair rate is a signal, not a nuisance. If repairs jump after a prompt edit or a model version change, output format is degrading before it breaks. Track the repair log as a metric. The week you stop looking at it is the week a repair starts hiding a real regression.

Never hand back partial data on failure. The moment callers can read the object off a failed outcome, someone will, usually under deadline pressure. Failed outcomes here carry errors and the repair log only, so using bad data requires deliberately bypassing the layer.

Retries without the errors are just expensive hope. Re-sending the same prompt that produced invalid JSON mostly produces it again. The re-ask works because it names the exact problems. Vague follow-ups ("please fix the JSON") test the model's mood; specific ones test its reading.

Wrap-up

Boring is the goal: raw text in, a validated object or a named failure out, every forgiveness logged, one bounded second chance, and no exceptions thrown at your callers. Graduate to a full schema library when your objects nest deeply or schemas become shared contracts across teams, and consider provider structured-output modes where available, while keeping this validation layer on top: a guaranteed shape still says nothing about whether a confidence of 1.4 makes sense in your domain. Contracts first, forgiveness second, silence never.

Top comments (0)