Anyone who shipped an LLM feature before 2026 knows the ritual. Write "return valid JSON only" in the prompt. Wrap the parse in a try block. Add a retry for the runs where the model opened with "Sure! Here's your JSON:" and wrapped everything in a markdown fence. Maybe strip fences with a regex. That whole layer is now dead weight.
Anthropic shipped Structured Outputs to GA on February 4, 2026, which made it the last major provider to support this natively. OpenAI has offered it since August 2024, Gemini does it through response_schema, and the open-source serving stack (vLLM, Ollama, SGLang) has shipped grammar-constrained decoding for well over a year. Parsing and praying is now a decision you are making, not a constraint you are living under.
I wrote a longer version of this on DevToolLab, LLM Structured Outputs: Get JSON That Always Parses, with more provider detail. This is the compressed take: the three things people mean by "structured output," the token-level mechanism behind the guarantee, the per-provider API, and the schema rules that quietly break your requests.
Three Different Things Share the Name
Conflating these is where most confusion begins.
Prompt-only JSON means you describe the shape in the prompt and hope for the best. Nothing is guaranteed, and the failure rate rises with schema complexity and falls with model size.
JSON mode is a flag, OpenAI's json_object being the canonical example, that forces the output to be syntactically parseable. That is all it forces. Required fields can go missing, types can be wrong, extra keys can appear. OpenAI labels it legacy now.
Schema-constrained generation is the real thing: you supply a JSON Schema and the response is guaranteed to conform, every required key present, every type right, every enum value drawn from your list. When OpenAI, Anthropic and Google say "Structured Outputs," this is what they mean.
| Approach | Parses as JSON | Conforms to your schema | Needs retry logic |
|---|---|---|---|
| Prompt-only | No | No | Yes |
JSON mode (json_object) |
Yes | No | Sometimes |
Schema-constrained (json_schema, strict) |
Yes | Yes | No |
Only the last one lets you delete code rather than accumulate it.
The Math That Kills Agent Loops
A single call failing to parse 1% of the time reads as acceptable. Put it in an agent that makes ten tool calls and the run has roughly a 10% chance of dying somewhere in the middle. Chain two or three agents together and you have built something that fails often enough to be unshippable.
The failure modes are dull and repetitive: a trailing comma, an unexpected code fence, an enum value the model invented, a number returned as a string, a required field that just is not there. Calling these model errors misses the point. Unconstrained sampling draws from the entire vocabulary at every step, and no part of that process knows your structure exists.
What Constrained Decoding Actually Does
The guarantee comes from constrained decoding, sometimes called guided or grammar-constrained generation, and the concept is straightforward even though the implementation is not.
Generation is token by token. At each step the model produces a probability distribution across a vocabulary that often exceeds 100,000 tokens, then samples one. Constrained decoding inserts itself between those two steps: your JSON Schema gets compiled into a formal grammar, the decoder tracks the current position within that grammar, and any token that would produce invalid output has its probability driven to zero. Broken JSON is not merely unlikely, it is unreachable. Anthropic's framing is that the schema becomes a formal grammar constraining generation token by token.
Open source got there first. Outlines made grammar-based generation practical for open weights, and XGrammar made it fast enough to be free, using adaptive token-mask caching. That is why XGrammar is the default backend in vLLM from v0.7.0 on, and why SGLang, TensorRT-LLM and MLC-LLM use it too. Overhead on JSON generation is effectively nil. The mechanism is identical everywhere; only the API surface and the accepted slice of JSON Schema differ.
Provider by Provider
OpenAI supports it from gpt-4o-2024-08-06 forward, including the GPT-5 models. You pass a JSON Schema with strict: true, via response_format in Chat Completions or text.format in the Responses API. The Python SDK wraps all of it in a parse helper that accepts a Pydantic model and hands back a typed object:
# Reference: requires OPENAI_API_KEY
from openai import OpenAI
from pydantic import BaseModel
class Ticket(BaseModel):
title: str
priority: str
estimate_hours: float
resp = OpenAI().responses.parse(
model="gpt-5",
input="Open a ticket: login redirect broken, high priority, ~3.5h",
text_format=Ticket,
)
print(resp.output_parsed) # Ticket(title=..., priority='high', ...)
Watch two things. Legacy json_object guarantees syntax only, never your schema. And because the output is grammar-constrained, a safety refusal cannot arrive as ordinary content, so OpenAI returns it in a separate refusal field that you must check before touching the payload.
Anthropic has been GA since February 2026 across the current lineup, Opus 4.8, Sonnet 5 and Haiku 4.5, with the beta header no longer required. The pattern matches:
# Reference: requires ANTHROPIC_API_KEY
from anthropic import Anthropic
from pydantic import BaseModel
class Contact(BaseModel):
name: str
email: str
plan_interest: str
resp = Anthropic().messages.parse(
model="claude-opus-4-8",
max_tokens=1024,
messages=[{"role": "user",
"content": "John Smith (john@example.com) wants the Enterprise plan."}],
output_format=Contact,
)
print(resp.parsed_output) # Contact(name='John Smith', ...)
At the raw HTTP level the schema rides in an output_config.format object. Claude covers two cases: JSON against a schema, shown above, and tool use, where a tool call's arguments are automatically constrained to that tool's input schema. The second is what stops agent function calls from arriving malformed.
Gemini takes a config object instead. Set response_mime_type to application/json and supply response_schema, which accepts a Pydantic model, a dict, or an enum:
# Reference: requires a Gemini API key
from google import genai
from pydantic import BaseModel
class Recipe(BaseModel):
name: str
ingredients: list[str]
resp = genai.Client().models.generate_content(
model="gemini-3.5-flash",
contents="Give me a simple pancake recipe.",
config={"response_mime_type": "application/json", "response_schema": Recipe},
)
print(resp.parsed)
Gemini takes a subset of JSON Schema and will reject schemas that are very large or deeply nested, so keep the structure flat.
Local models need no hosted API at all. Ollama has accepted a JSON Schema in format since 0.3.0, constraining the local model's decoding and dropping fences and preamble on its own:
# Reference: requires a running Ollama server + a pulled model
from ollama import chat
from pydantic import BaseModel
class Country(BaseModel):
name: str
capital: str
population: int
resp = chat(model="llama3.1",
messages=[{"role": "user", "content": "Tell me about Japan."}],
format=Country.model_json_schema())
print(Country.model_validate_json(resp.message.content))
On self-hosted vLLM or SGLang the parameter is guided_json, with XGrammar underneath. Identical guarantee, running on hardware you control.
Wiring It Into an Application
Four steps, and they are the same regardless of provider.
Start by defining the shape exactly once, in Pydantic for Python or Zod for TypeScript. That single definition serves as both the schema you transmit and the type you validate against.
from pydantic import BaseModel, Field
from enum import Enum
import json
class Priority(str, Enum):
low = "low"; medium = "medium"; high = "high"
class Ticket(BaseModel):
title: str
priority: Priority
tags: list[str] = Field(default_factory=list)
estimate_hours: float
print(json.dumps(Ticket.model_json_schema(), indent=2))
Look closely at what that emits. The required array contains title, priority and estimate_hours, but not tags, because tags carries a default. Hold that thought, it causes real pain shortly.
Second, send it, either as a raw schema or as a Pydantic model handed to a parse helper, and receive a conforming typed object.
Third, validate locally anyway. It costs microseconds and it protects you from SDK quirks, proxy rewrites, and whichever endpoint in your stack lacks native support:
from jsonschema import validate, ValidationError
schema = Ticket.model_json_schema()
good = '{"title":"Fix login redirect","priority":"high","tags":["auth"],"estimate_hours":3.5}'
validate(instance=json.loads(good), schema=schema) # raises if wrong
print(Ticket.model_validate_json(good)) # typed object
try:
validate(instance={"title":"x","priority":"urgent","estimate_hours":2}, schema=schema)
except ValidationError as e:
print("rejected:", e.message) # 'urgent' is not one of ['low', 'medium', 'high']
Fourth, deal with refusals and gaps in support. Branch on OpenAI's refusal field before reading data, and for any endpoint without native support keep a repair-then-retry fallback rather than counting braces by hand. DevToolLab's JSON Repair tool is the browser version of that fallback when you are debugging by hand instead of in code.
The Schema Rules That Cost You an Afternoon
This is where the 400 responses come from, because every provider accepts a different subset of JSON Schema.
OpenAI's strict mode requires every field. You cannot simply omit a property from required. Every declared property must appear there, and every object needs additionalProperties: false. Making something optional means keeping it in required while giving it a nullable type such as "type": ["string", "null"]. Combine that with the earlier detail, Pydantic excluding defaulted fields from required, and any model with defaults fails strict mode untouched. This helper repairs any schema in place:
def make_strict(schema: dict) -> dict:
"""additionalProperties:false + every key required, at every level."""
if isinstance(schema, dict):
if schema.get("type") == "object" and "properties" in schema:
schema["additionalProperties"] = False
schema["required"] = list(schema["properties"].keys())
for value in schema.values():
make_strict(value)
elif isinstance(schema, list):
for item in schema:
make_strict(item)
return schema
Certain keywords get dropped or refused. OpenAI strict mode does not enforce minLength, maxLength, pattern or format, and it rejects default outright. Anything depending on a regex pattern has to be checked after the fact. Claude and Gemini each define their own subset, so portability across providers is not automatic. Depth and size limits apply on all three hosted providers, so flatten aggressively and split oversized schemas into separate calls. Checking a response against your schema in the JSON Schema Validator is faster than guessing which keyword the provider silently ignored. The original post goes through each provider's subset in more detail.
TypeScript teams get the same rules through Zod, defining once and validating identically:
import { z } from "zod"
const Ticket = z.object({
title: z.string(),
priority: z.enum(["low", "medium", "high"]),
tags: z.array(z.string()),
estimateHours: z.number(),
})
const parsed = Ticket.parse(JSON.parse(rawModelOutput)) // throws on mismatch
What This Does Not Fix
Constrained decoding enforces shape, never truth. Nothing prevents "estimate_hours": 400 on a two-line fix, or a confidently extracted email address belonging to the wrong person. Schema conformance is the floor of output quality, not the ceiling, and evals remain the only way to measure the rest.
There is a modest cost too. Forcing the grammar can steer the model down a slightly less natural path, and compiling an unfamiliar complex schema adds a one-time warm-up that gets cached afterward. For extraction, classification and tool calling that trade is obvious. For long-form creative writing it is usually the wrong instrument.
Conclusion
Structured outputs convert "get JSON out of an LLM" from a reliability problem into a schema-design problem, which is a far better problem to own. Define the shape once in Pydantic or Zod, ship it as a JSON Schema, let constrained decoding enforce it, and keep local validation as a cheap backstop. Fences, trailing commas, absent fields and illegal enums all stop at the source.
Treat the schema as a contract. Get additionalProperties, required and nullability right, keep the structure flat, and stay honest that a valid shape is not a correct answer. Then find your single most parse-prone call, the one wrapped in the ugliest error handling you own, and convert it this week. It is usually about ten lines of change that removes a hundred lines of defense.
Provider availability, model names and API parameters here reflect July 2026. This area moves fast, so confirm field names and the supported JSON Schema subset in each provider's current docs before shipping.
Top comments (0)