Structured Outputs in 2026: How to Make AI Agents Return Valid JSON, Every Time

Your agent worked in the demo. Then, in production, it returned this:

{"status": "success", "total": 1999, "items": ["keyboard", "hub"],}

A trailing comma. Your json.loads() threw, the pipeline crashed, and the retry prompt you added — "return valid JSON, seriously" — made things worse, because now the model occasionally wraps the JSON in a markdown code block and adds a smug apology.

Every developer building agents in 2026 hits this wall. Agents are pipelines of LLM calls: extract, decide, call a tool, format a reply. One call that returns slightly-off text breaks everything downstream. This article shows you the fix — structured outputs — with working Python you can run today, plus the validation-and-retry pattern that keeps multi-step agents alive when models misbehave.

Why this became the #1 reliability problem

Three shifts in the last year made text-parsing the bottleneck:

1. Agents make dozens of LLM calls per task. A single research agent might call the model 20 times. If each call has a 5% chance of malformed output, the whole run has a ~64% chance of failing somewhere. One parse error kills the job.
2. Reasoning models changed the failure mode. They don't just fail — they fail creatively: JSON inside a code block, extra prose before and after, "total": "1999" as a string, or an invented field the schema never asked for.
3. Downstream systems are typed. Your database column is INTEGER, your API expects amount_cents, your n8n workflow maps json.total into a field. The model's output must match a contract, not approximate it.

Prompt-only JSON ("return JSON") gets you 85–95% reliability on a good day. Structured outputs push that past 99% by making the decoder enforce the schema — the model physically cannot emit tokens that violate it.

The reliability ladder

Approach	What guarantees it	Failure rate	Cost
Free text + regex	Nothing	High	0
"Return JSON" prompt	The model's mood	~5–15%	0
JSON mode (response_format)	Valid JSON syntax	~1–3%	0
Function calling	Argument schema at decode time	<1%	0
Structured outputs	Full JSON Schema, constrained decoding	<0.5%	~0

Structured outputs are the top rung: you give the API a schema (usually a Pydantic model), and the provider's constrained decoding guarantees the response conforms. No parsing, no retries, no prayers.

Implementation 1: structured outputs with the Responses API

Install the SDK:

pip install openai pydantic

Define your contract as a Pydantic model, then call responses.parse — the SDK validates the output and hands you a typed object:

import os
from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class OrderExtraction(BaseModel):
    order_id: str = Field(description="Order ID, e.g. ORD-8821")
    items: list[str] = Field(description="Product names, exactly as written")
    total_cents: int = Field(description="Total in cents (integer, never a float)")
    currency: str = "USD"

resp = client.responses.parse(
    model="gpt-4.1-mini",
    input=(
        "Extract the order details from: "
        "'Your order ORD-8821: 2x Mechanical Keyboard, 1x USB-C Hub. Total $89.98.'"
    ),
    text_format=OrderExtraction,
)

order = resp.output_parsed
print(order.order_id, order.items, order.total_cents)  # ORD-8821 ['Mechanical Keyboard', 'USB-C Hub'] 8998

Note the details that matter in production:

• total_cents: int, not total: float. Money as integer cents eliminates float rounding bugs. The schema says integer, the decoder enforces it.
• Defaults encode business rules. currency: str = "USD" means the model only overrides it when the text says otherwise.
• Field(description=...) is not optional. Descriptions are how the model knows what to put in each field. Sparse schemas produce confident garbage.

The SDK raises a ValidationError if the response doesn't match, and resp.output_parsed is already a validated OrderExtraction — no manual json.loads, no .get() chains.

Implementation 2: the provider-agnostic retry loop

Structured outputs are not available on every model or provider. When you're behind a gateway, a local model, or an older API, the production pattern is: JSON mode + validation + self-correcting retry:

import json
from pydantic import ValidationError


def parse_with_retry(client, model, prompt, schema_model, attempts=3):
    """JSON-mode parsing with a validation-error feedback loop."""
    messages = [{"role": "user", "content": prompt}]
    for _ in range(attempts):
        resp = client.chat.completions.create(
            model=model,
            messages=messages,
            response_format={"type": "json_object"},
        )
        raw = resp.choices[0].message.content
        try:
            data = json.loads(raw)
            return schema_model.model_validate(data)
        except (json.JSONDecodeError, ValidationError) as exc:
            # Feed the exact error back — models are surprisingly good at fixing their own JSON
            messages.append({"role": "assistant", "content": raw})
            messages.append({
                "role": "user",
                "content": (
                    f"Your previous output was invalid: {exc}. "
                    f"Return ONLY valid JSON matching this schema: "
                    f"{schema_model.model_json_schema()}"
                ),
            })
    raise RuntimeError(f"Could not get valid output after {attempts} attempts")

This works because the error message is specific: "total: Input should be a valid integer" tells the model exactly what to fix. Generic retries ("please try again") fail ~90% of the time; error-aware retries fix the output on the second attempt in most cases.

Rule of thumb: use structured outputs (Implementation 1) where the provider supports it; use the retry loop (Implementation 2) everywhere else — and note that most failures surface in validation, which is exactly why you keep the Pydantic model as the single source of truth for the contract.

Structured outputs vs. function calling: when to use what

Function calling and structured outputs both enforce schemas, but they answer different questions:

You want the agent to...	Use	Why
Extract or classify data	Structured outputs	The whole response is the data
Decide whether to call something	Function calling	The model chooses from your tools
Call a tool with correct args	Function calling	Arguments validated against the tool schema
Return a full report/object	Structured outputs	Bounded schema, typed result

Mixed pattern that works: the agent picks a tool via function calling, and the tool's response is parsed with structured outputs. Each mechanism handles the failure mode it's best at.

MCP tools: strict schemas are free

If you expose tools to agents via MCP, the schema is your contract — and FastMCP derives it from your type hints and docstring automatically:

from mcp.server.fastmcp import FastMCP
from pydantic import Field

mcp = FastMCP("orders")


@mcp.tool()
def charge_order(
    order_id: str = Field(..., pattern=r"^ORD-\d{4}$", description="Order ID, format ORD-XXXX"),
    amount_cents: int = Field(..., gt=0, description="Amount to charge in cents, must be positive"),
) -> str:
    """Charge a customer's order. Returns the payment reference."""
    # ... payment call ...
    return f"charged {order_id} for {amount_cents} cents"


mcp.run()  # transport=stdio by default

Now every agent that connects to this server gets a strict JSON Schema for charge_order — and a model that violates it gets an error from the tool-calling layer, not from your business logic. Same principle as structured outputs, applied to the tool boundary.

Pitfalls that still bite in 2026

• Refusals are not JSON. If the model refuses (safety, policy), some providers return a refusal string instead of structured output. Check for it explicitly before parsing — an empty output_parsed is a signal, not a bug.
• Shallow schemas beat deep ones. 5+ levels of nesting measurably increases failure. Flatten: extract a summary object, then do a second call for details.
• Constraints have a reliability gradient. enum and const are rock-solid; pattern is good; complex anyOf/oneOf unions are where constrained decoders get confused. Keep unions small and flat.
• Money is integers. Floats in schemas invite 89.9800000001. Use cents (or the smallest unit) everywhere.
• Optional fields vanish. If a field isn't required, the model may omit it when "not applicable." Make required everything your downstream code reads without .get().
• Streaming changes nothing — almost. Structured outputs work with streaming, but you must accumulate the full stream before validating. Don't parse chunks.
• Test with adversarial input. Unicode, emojis, huge numbers, empty strings, and 10,000-word inputs all stress the schema differently. Add these to your evals — this is exactly the kind of case that breaks agents in production.

Production checklist

1. Every LLM call that feeds a machine boundary returns a Pydantic-validated object — never raw text.
2. Money is integer cents; dates are ISO-8601 strings; IDs carry a regex.
3. Structured outputs where supported; JSON mode + error-feedback retry elsewhere.
4. Refusal/empty outputs are handled before parsing.
5. Tool schemas (MCP included) are strict: descriptions on every field, gt/pattern/enum where they apply.
6. Adversarial samples (emoji, unicode, huge numbers) are in your eval suite.
7. json.loads appears in exactly one place in your codebase — inside a wrapper that validates and retries.

Structured outputs won't make agents perfect, but they remove the dumbest, most frequent failure class: the output was almost right, and the pipeline died anyway. Fix that, and the remaining failures are interesting ones — the ones worth debugging.

---

This article is adapted from the AI Agents Playbook — 11 chapters on building agents that ship: agent memory, evals, MCP, and production patterns, with 10 copy-paste agent blueprints. Get it with code **LAUNCH11* for 55% off: https://aibuildershub.gumroad.com/l/ai-agents-playbook — and grab the free tutorials at https://hirara-hermes.github.io/*