Structured outputs: getting JSON you can trust from LLMs

Originally published at AI Frontier Post.

---

Malformed JSON from LLMs doesn't have to be a fact of life. Here's the full reliability stack — strict schemas, provider enforcement, constrained decoding, and validation-and-repair loops — that ends the bug for good.

Every LLM app builder has the same scar. You wire a model into a pipeline, ask it to return JSON, and it works in the demo. Then production arrives: a trailing comma here, a missing quote there, an extra explanatory paragraph wrapped around your payload, and your json.loads() explodes at 2 a.m.

This is a solved problem. Getting JSON you can trust from an LLM is a stack of four layers — schema design, provider enforcement, constrained decoding, and validation loops — and you need all four. Here's how they fit together.

Why "just ask for JSON" fails

Asking a model to "respond in valid JSON" works most of the time, which is the worst kind of reliability. The failure modes are varied and annoying:

• Malformed syntax — trailing commas, unescaped quotes, markdown fences around the payload.
• Wrong shape — valid JSON, but a field is missing, renamed, or the wrong type. JSON mode (OpenAI's older json_object, Mistral's json_object) only guarantees syntax, not adherence to your schema.
• Truncation — a long response hits the token limit mid-object and you get half a document.
• Refusals — a safety refusal doesn't follow your schema at all, so parsing it blindly gives you garbage.

No single fix covers all of these. That's why the answer is a layered stack, not one trick.

Layer 1: Write a schema the model can actually satisfy

Everything starts with the schema itself. A sloppy schema enforced perfectly is still sloppy. Design rules that pay off:

• Define the schema once, in code. Write a Pydantic model (Python) or Zod schema (JavaScript) and derive the JSON Schema from it. Your request constraint and your response validator then can never drift apart — they share one source of truth.
• Be strict and explicit. On OpenAI, set strict: true, list every key in required, and set additionalProperties: false. This is the difference between "hopefully the model cooperates" and server-side enforcement.
• Use enums for closed choices. "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]} beats hoping the model doesn't invent "mostly fine".
• Describe every field. Field descriptions become part of the model's instructions — they're prompts, not decoration.
• Keep schemas modest. Deep nesting, recursion, and exotic constraints are where even good providers wobble. Flatten what you can; the flatter the schema, the higher the hit rate.

A small but honest example:

from pydantic import BaseModel, Field
from typing import Literal

class SupportSummary(BaseModel):
    issues: list[str] = Field(description="Top issues mentioned, verbatim short phrases")
    sentiment: Literal["positive", "neutral", "negative"]
    confidence: float = Field(ge=0.0, le=1.0)

Layer 2: Let the provider enforce it

The big labs all offer server-side structured output now, though the knob differs per provider:

Provider	Mechanism	Schema enforcement	Notes
OpenAI	text.format with type: "json_schema", strict: true (Responses API) or response_format (Chat Completions)	Full	SDK helpers: responses.parse() takes a Pydantic model and returns typed output
Google Gemini	response_schema + response_mime_type: "application/json"	Full	Returns strictly validated JSON when a schema is set
Anthropic (Claude)	Tool Use: define a tool with input_schema, force it with tool_choice	Full (via tool args)	No generic JSON-mode switch; read the tool's arguments as your object. Newer SDKs increasingly add parse helpers on top
Mistral	response_format: {"type": "json_object"}	JSON only, not your schema	Valid JSON guaranteed; validate against your schema client-side
AWS Bedrock	Converse API toolConfig with JSON schema, forced via toolChoice	Full	Model-agnostic across hosted models

The OpenAI pattern is the cleanest illustration of where the industry has landed:

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]

response = client.responses.parse(
    model="gpt-5",
    input=[
        {"role": "developer", "content": "Extract the event information."},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
    ],
    text_format=CalendarEvent,
)
event = response.output_parsed  # a CalendarEvent, not a string

Note what this buys you: no parsing code at all. The typed object comes out of the SDK.

Two caveats the official docs are honest about. First, the first request with a new schema has extra latency while the API processes the schema — subsequent identical-schema calls don't. Second, enforcement isn't absolute: a refusal or a response cut off by the token limit can still come back schema-invalid, which is why Layer 4 exists.

Layer 3: When you self-host, constrain the decoder

If you run your own models — Llama, Qwen, Mistral weights on your own GPUs — there's no provider API to lean on. The equivalent machinery is constrained decoding (also called structured generation), and it's arguably the most reliable approach of all.

Instead of validating output after generation, constrained decoding enforces structure during token generation. At every step, the sampler masks out tokens that would violate your schema: if the next valid token must be a digit, string-starting tokens are blocked outright. The model can only ever sample schema-valid continuations, so the final output matches by construction — no retries, no post-processing.

The ecosystem is mature:

• Libraries: Outlines, Microsoft Guidance, XGrammar
• Inference servers: vLLM and SGLang have structured generation built in (vLLM accepts structured_outputs: {"json": schema} or {"regex": "..."} in the request)

The regex option deserves a mention: for fixed-format strings — order codes, dates, IDs — a regex constraint is lighter than a full schema and just as airtight.

Layer 4: Validate everything, then run the repair loop

Here is the non-negotiable rule, even with native provider enforcement: never trust a raw parse. Always validate every response against your schema client-side, because constraints can still be interrupted by length caps and refusals.

The loop is generate → validate → repair:

from pydantic import BaseModel, ValidationError
import json

def get_structured(call_llm, user_text, schema_cls, max_attempts=3):
    messages = [{"role": "user", "content": user_text}]
    for _ in range(max_attempts):
        raw = call_llm(messages)  # JSON-mode / schema-enforced call
        try:
            return schema_cls.model_validate(json.loads(raw))
        except (ValueError, ValidationError) as e:
            # Feed the validator's error back so the model self-corrects
            messages.append({"role": "assistant", "content": raw})
            messages.append({"role": "user", "content":
                f"That was invalid: {e}\nReturn corrected JSON only."})
    raise RuntimeError("No valid output after retries")

Why this works so well: the validator's error message is specific ("field 'confidence' missing", "expected float, got string"), and models are excellent at fixing precisely-described mistakes. One repair attempt resolves the vast majority of transient failures.

Three discipline points make this production-safe:

• Cap the attempts. Two to four tries, then stop. An unbounded correction loop is a cost and latency hazard — a model that can't produce your schema in three tries has a schema problem, not a luck problem.
• Fail typed, not loud. When retries are exhausted, return a structured error object (or a safe default), not an unhandled exception that kills the request.
• Fix the schema before retrying forever. Repeated failures on the same field mean the schema is ambiguous or too complex. Flatten it, add an example, or split the extraction into smaller calls.

Anti-patterns to retire

Anti-pattern	Why it breaks	Replace with
json.loads(resp) right after a completion	No schema check; throws on any malformation	Native schema output + validated parse
Regex-stripping markdown fences	Regex can't reliably parse nested/escaped JSON	Ask for raw JSON; stop requesting fenced output
Bare JSON mode with no schema	Valid JSON, wrong shape	Schema-constrained output, or JSON mode + validate + retry
data["field"] straight off the parse	KeyError when a field is missing or renamed	Validate into a typed model, then access attributes

The takeaway

Think of it as a reliability ladder. Good schemas give the model a target it can hit. Provider enforcement (OpenAI structured outputs, Gemini response schemas, Claude tool-use schemas) makes the target a guarantee in most cases. Constrained decoding brings the same guarantee to self-hosted models. And the validate-and-repair loop catches everything the other layers miss — truncations, refusals, and the odd provider hiccup.

Most teams can stop climbing at Layer 2 plus the repair loop: that's already a two-nines improvement over prompt-and-pray. But if you implement only one thing from this article, make it the loop. Define your schema once in code, validate every response against it, and feed the error back on failure. Malformed-JSON bugs don't survive that discipline.

Comments

[Brian · AI News]

The layered ladder is the useful part. Schema constraints catch shape errors while validation and capped repair handle the messy edge cases.