DEV Community

Syed Anzar
Syed Anzar

Posted on

Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models

#llm #python #ollama #json

Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models

You asked a local model for JSON. You got JSON. You json.loads() it and — JSONDecodeError: Expecting value. Because buried in the "JSON" was a code fence, three sentences of "Here is your result:", and a trailing comma no parser will forgive.

If you've wired a local LLM into an agent, an ETL job, or a backend endpoint, you've hit this. The naive fix is a regex that strips code fences. That regex works until it doesn't, and "until it doesn't" always lands in production at 2 a.m.

This article gives you the real fix: a pipeline that combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries. By the end you'll have a copy-pasteable structured_extract() you can drop into any local-LLM project.

Why format="json" is not enough

Ollama's format parameter accepts two very different things:

  • format="json" means JSON mode. The model is steered to emit a valid JSON object. That's it. It does NOT enforce your field names, types, or required keys. You can still get {"result":"..."} when you wanted {"severity":"high","summary":"..."}.
  • format set to a JSON Schema object means structured output. Ollama applies constrained decoding: at every generation step it sets the probability of any token that would violate your schema to zero. The model physically cannot emit a markdown fence, surrounding prose, or a structurally invalid object.

So the first rule: pass a real JSON Schema, not the string "json". The Python Ollama client makes this trivial with Pydantic:

from ollama import chat
from pydantic import BaseModel

class Country(BaseModel):
    name: str
    capital: str
    languages: list[str]

response = chat(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Tell me about Canada."}],
    format=Country.model_json_schema(),
)

country = Country.model_validate_json(response.message.content)
Enter fullscreen mode Exit fullscreen mode

This is the official recommended pattern and it works on Ollama 0.3.0 or newer. For most straightforward schemas on a 7B-plus model, this alone kills the parse failures.

But "most straightforward schemas" hides the real edge cases. Three things still bite you:

  1. You aren't always on Ollama. A different local server, an older Ollama, or any endpoint that only offers loose JSON mode won't constrain decoding, and you'll be back to fences and partial objects.
  2. Constrained decoding constrains structure, not truth. It guarantees the shape; it does not guarantee the values are correct. If the text says "salary is competitive" and your schema demands an integer, the model WILL hallucinate a number to fill it.
  3. Small models still misbehave on nested or optional fields (more below).

So the robust design is: constrain when you can, defend when you can't, validate always, retry with feedback.

The resilient parser (for when constraints aren't there)

If you can't rely on constrained decoding, you need a parser that survives hostile output. json_repair (PyPI json-repair) is the drop-in upgrade for json.loads() — it fixes missing quotes, trailing commas, truncated values, and strips stray prose:

import json_repair

bad = 'Extracting now: {"users":[{"name":"Ada","role":"admin",}],"ok":true'
obj = json_repair.loads(bad)
# -> {'users': [{'name': 'Ada', 'role': 'admin'}], 'ok': True}
Enter fullscreen mode Exit fullscreen mode

Two gotchas from the library docs:

  • By default it tries stdlib json.loads first and only falls back to the repair parser on failure, so feeding it valid JSON is safe.
  • If you already know the input is broken, pass skip_json_loads=True to skip the fast path. Do NOT use that flag on input you expect to be valid; the repair parser can reshape valid JSON.

json_repair also supports schema/pydantic-guided repair and a strict=True mode that raises instead of repairing. We'll use the gentle default.

The validation plus retry layer

Never trust the parsed object. Validate it against your contract, and when validation fails, retry with the error fed back to the model — not a blind re-roll. One to three attempts is the right ceiling; beyond that, fail loudly and keep the raw output so you can debug.

You can get this for free with instructor:

import instructor
from pydantic import BaseModel

client = instructor.from_provider("ollama/qwen2.5:7b")

result = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Classify this support ticket: ..."}],
    response_model=Ticket,
    max_retries=2,
    timeout=30.0,   # TOTAL across retries, important for slow local models
)
Enter fullscreen mode Exit fullscreen mode

But instructor leans on the OpenAI-compatible endpoint, which still depends on the backend honoring the schema. The fully self-contained version below works directly against the Ollama chat API and shows exactly what's happening.

The complete drop-in pipeline 

from ollama import chat
from pydantic import BaseModel, field_validator, ValidationError
import json_repair
import json

BACKTICK = chr(96)
FENCE = BACKTICK * 3

def _defensive_parse(content):
    cleaned = content.strip()
    if cleaned.startswith(FENCE):
        cleaned = cleaned.split(chr(10), 1)[1]   # drop the opening fence line
        if cleaned.rstrip().endswith(FENCE):
            cleaned = cleaned.rstrip()[:-3]       # drop the closing fence line
    start = cleaned.find("{")
    end = cleaned.rfind("}")
    if start != -1 and end != -1:
        cleaned = cleaned[start:end + 1]         # keep only the {...} body
    try:
        return json.loads(cleaned)
    except json.JSONDecodeError:
        return json_repair.loads(cleaned)        # last resort

def structured_extract(
    model_cls: type[BaseModel],
    prompt: str,
    model: str = "qwen2.5:7b",
    max_retries: int = 3,
):
    schema = model_cls.model_json_schema()
    last_error = None
    for attempt in range(1, max_retries + 1):
        messages = [
            {"role": "system",
             "content": "Return ONLY JSON matching the schema. No prose, no code fences."},
            {"role": "user", "content": prompt},
        ]
        if attempt > 1 and last_error:
            messages.append(
                {"role": "user",
                 "content": f"Your previous output failed validation: {last_error}\n"
                            f"Fix it to match the schema exactly."}
            )
        resp = chat(
            model=model,
            messages=messages,
            format=schema,                 # constrained decoding (Ollama >= 0.3.0)
            options={"temperature": 0},
        )
        try:
            return model_cls.model_validate(_defensive_parse(resp.message.content))
        except (ValidationError, ValueError) as e:
            last_error = str(e)
    raise RuntimeError(
        f"Structured extraction failed after {max_retries} attempts. "
        f"Last error: {last_error}\nRaw: {resp.message.content!r}"
    )
Enter fullscreen mode Exit fullscreen mode

model_validate (not model_validate_json) takes the already-parsed object, so a JSON-mode endpoint that slips a fence through still lands in the defensive parser. With constrained decoding on, the fence rarely appears, but defense-in-depth is the whole point.

The 9 mistakes that make structured output silently wrong

# Mistake Symptom Fix
1 Required fields the text doesn't contain Fabricated values ("competitive" becomes 50000) Use Optional[X] = None so absence is valid
2 Deeply nested schemas (List[Dict[str, List[Model]]]) Empty intermediate arrays on sub-12B models Keep nested arrays flat; use a bigger model
3 Optional[str] comes back as empty string not None None checks silently fail field_validator normalize empty -> None
4 One giant 20-field schema Lower per-field accuracy Split into 2-3 sequential 6-7 field calls
5 Trusting model output blindly Valid-but-wrong-shape JSON ships Always validate against the contract
6 format="json" instead of a schema Right shape, wrong keys/types Pass a full JSON Schema object
7 No retry or infinite retry Lost data or hangs Retry with error feedback, max 3, then fail loud
8 Schema description as prompt Model ignores field meaning Restate semantics in the prompt prose
9 Rigid schema for generative tasks Stilted, constrained output Use a system prompt for generation, schema for extraction

Mistake 3 is the one almost everyone ships. Add the normalizer:

from pydantic import BaseModel, field_validator
from typing import Optional

class Review(BaseModel):
    summary: str
    sentiment: Optional[str] = None

    @field_validator("sentiment", mode="before")
    @classmethod
    def empty_to_none(cls, v):
        if isinstance(v, str) and v.strip() == "":
            return None
        return v
Enter fullscreen mode Exit fullscreen mode

Mistake 1 is subtler and more dangerous than a parse error: json.loads succeeds, validation succeeds, and you store a confidently wrong number. Optional plus None is the only honest signal a field was absent.

Trade-offs you should accept

  • Constrained decoding is Ollama/server-specific. If your backend doesn't support schema format, you fall back to the parser plus validator layer, which is strictly weaker (it can't prevent bad structure, only recover from it). Know which one you're on.
  • Schema size costs context. A large Pydantic model's JSON schema can be hundreds of tokens, shrinking the prompt budget. Focused schemas win.
  • temperature=0 is non-negotiable for extraction. Higher temperatures make the model invent enum values and field contents. Deterministic is what you want here.
  • Use structured output for extraction, not generation. If you're pulling facts that exist in the source text, schema-constrain it. If you're writing new content that should follow a shape, a well-crafted system prompt usually produces better results than a rigid schema.

Practical takeaways

  1. Pass a JSON Schema object, not "json" — get constrained decoding.
  2. Define schemas with Pydantic, use Optional plus Field(description=...), and temperature=0.
  3. Always run a defensive parse plus model_validate even when constrained — hostile output is rare but real.
  4. Retry with the validation error, cap at 3, then fail loudly with raw output logged.
  5. Normalize empty string to None for Optional fields so downstream None checks work.
  6. For non-Ollama or JSON-mode-only endpoints, lean on json_repair as your parse safety net.

Structured output stops being a coin flip the moment you stop trusting the model and start enforcing a contract. Constrain what you can, defend what you can't, validate everything, and your agent loop stops dying on malformed JSON.

References

Top comments (0)