Your LLM returns JSON that crashes json.loads about 1 time in 20. Here is how to handle it

If you call an LLM and ask for JSON, it works in the demo and then fails in production at the worst moment. The usual culprits are a markdown code fence around the JSON, a sentence before it ("Sure! Here is the JSON:"), or a trailing comma.

Here is a small, step-by-step way to deal with all three in plain Python.

Step 1: Strip the usual wrapping

import json
import re

def extract_json(text: str):
    text = text.strip()
    # 1. remove ```
{% endraw %}
json ...
{% raw %}
 ``` fences
    fence = re.search(r"```

(?:json)?\s*(.*?)

```", text, re.DOTALL)
    if fence:
        text = fence.group(1).strip()
    # 2. keep only the outermost {...} or [...]
    starts = [i for i in (text.find("{"), text.find("[")) if i != -1]
    if not starts:
        raise ValueError("no JSON found")
    start = min(starts)
    end = max(text.rfind("}"), text.rfind("]"))
    return text[start : end + 1]

Step 2: Parse, and repair the one common error

def parse_llm_json(text: str):
    raw = extract_json(text)
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        # remove trailing commas before } or ]
        fixed = re.sub(r",\s*([}\]])", r"\1", raw)
        return json.loads(fixed)  # let it raise if still broken

Step 3: Validate the shape, not just the syntax

Valid JSON can still have the wrong keys. Check the fields you rely on:

def require(data: dict, schema: dict):
    for key, typ in schema.items():
        if key not in data:
            raise KeyError(f"missing key: {key}")
        if not isinstance(data[key], typ):
            raise TypeError(f"{key} should be {typ.__name__}")
    return data

result = require(parse_llm_json(reply), {"title": str, "score": int})

Step 4: Retry once with the error message

Feed the exact error back to the model. One retry fixes most failures, and more than two retries usually just burns tokens:

def ask_json(call_llm, prompt, schema, retries=1):
    reply = call_llm(prompt)
    for attempt in range(retries + 1):
        try:
            return require(parse_llm_json(reply), schema)
        except (ValueError, KeyError, TypeError) as e:
            if attempt == retries:
                raise
            reply = call_llm(
                f"{prompt}\n\nYour previous answer was invalid: {e}. "
                "Return only the corrected JSON, no extra text."
            )

Better: ask the API to enforce it

Parsing defensively is the fallback. Where the provider supports it, use structured outputs so the model is constrained to your schema in the first place. See OpenAI's Structured Outputs guide and Python's json module docs for the error types you can catch (json.JSONDecodeError carries the line and column).

Recap

1. Strip fences and surrounding prose.
2. Parse, and remove trailing commas.
3. Validate keys and types.
4. Retry once, quoting the error.

What is the weirdest broken-JSON output you have gotten from a model? I'd like to add it to the test cases.