AI in Practice, No Fluff — Day 5/10
Every morning I get a briefing. An AI agent gathers data from my calendar, my notes, my project list, and a handful of other sources, then returns it all as structured JSON. A template takes that JSON and renders it into a readable dashboard. The specifics of how that pipeline works are not important for this post. What is important: if the JSON comes back wrong, nothing renders.
Not "renders badly." Nothing. A missing field, an inconsistent key name, a stray sentence of commentary mixed into the data block: any of these breaks the template, and I start my morning staring at an error instead of a dashboard.
This is the case for structured output in two sentences: when a system reads your AI's response instead of a human, "close enough" stops working. A paragraph that answers the question is fine if you are reading it. It is useless if a program needs to parse it.
The first time I set up this pipeline, I told the AI to "return the results as JSON." What I got back was mostly JSON, with a conversational preamble and a closing note that the AI thought I might find helpful. Technically generous. Practically broken. The fix took three iterations: show the exact schema, give two examples of correctly formatted entries, and explicitly say "return only the JSON block, no commentary." Once I did that, the output was clean every time.
That progression is the whole article. Telling an AI what format you want is not the same as telling it what structure you want. Format is "give me JSON." Structure is "give me an array of objects with these specific fields, in this order, with these types." The gap between the two is where most structured output problems live.
Where this fits
In the first series, we covered what makes a good prompt: context, task, format, examples. Earlier in this series, we covered few-shot prompting and why showing examples beats describing what you want. Both of those principles apply here. This post focuses them on a specific problem: getting AI to return data you can actually use programmatically, not just read.
The previous post covered debugging a prompt when the output keeps missing. One of the four failure modes was "bad format specification." Structured output is the deeper dive into that category: what specifically goes wrong with format, why, and what to do about it.
Why "return JSON" is not enough
When you type "return this as JSON," the AI is generating tokens one at a time, left to right, trying to maintain valid JSON syntax while simultaneously producing useful content. It has no schema enforcement. It is doing two jobs at once: being helpful and being syntactically correct.
This works fine for simple requests. "Give me a JSON object with name and email" will usually come back clean. The problems start when the structure gets more complex: nested objects, arrays of items that all need consistent fields, specific data types, fields that should be present even when the value is empty.
The model is not being lazy or difficult. It is generating text in a format it was not specifically optimized for, and every additional structural constraint is one more thing it has to hold in working memory while producing the next token. Inconsistent field names across array items happen because the model does not have a checklist; it is reconstructing the pattern from context on each new object.
How to get reliable structure from a chat
If you are working in the chat window (ChatGPT, Claude.ai, Gemini), you do not have access to API-level schema enforcement. But you can get close with three techniques that stack:
1. Show the schema, do not describe it. This is the few-shot principle applied to structure. Instead of "return a JSON object with fields for name, rating, and summary," write out the actual object:
{
"product": "Example Widget",
"rating": 4,
"sentiment": "positive",
"summary": "One sentence here."
}
The model will mirror that structure far more reliably than it will interpret a prose description of it.
2. Give two rows, not one. A single example shows the shape. Two examples show the pattern. When the model sees two objects with identical field names and types, it treats those fields as mandatory rather than suggestive. This is especially important for arrays where consistency across items is the whole point.
3. Name the constraints explicitly. "Every object must have all four fields, even if the value is null." "Use exactly these field names, no variations." "Do not include any text outside the JSON block." These feel redundant after the examples, but they're the ones that catch the edge cases where the model might improvise. Think of them as the guardrails, not the road.
These three techniques together get you 90% of the way to reliable structured output in a chat window. The remaining 10% is where API features come in.
What the APIs actually do
When you move from the chat window to building something with an API, structured output stops being a prompting problem and becomes a feature you can turn on.
Both Claude and OpenAI (and increasingly other providers) now offer structured output modes that work at a fundamentally different level than prompting. Instead of asking the model to please maintain valid JSON, the API compiles your JSON schema into a grammar that constrains which tokens the model is allowed to generate. The model cannot produce invalid JSON or deviate from your schema, because the generation process only considers tokens that would keep the output valid.
In Claude's API, you pass your schema in the request configuration. In OpenAI's API, you set the response format to "json_schema" with strict mode enabled.
The practical difference is significant. With prompt-based approaches, you are asking the model to be disciplined. With structured output features, the infrastructure enforces discipline for you. The model focuses entirely on producing good content; the system handles the structure.
This is one of the clearest examples of a theme we will return to later in this series: the gap between what you can do in a chat window and what you can do with the API. If reliable structured output matters for your use case, this is one of the strongest reasons to move from the UI to building.
Validation is still your job
Even with schema enforcement, validation is not optional. The schema guarantees structure: every field present, correct types, valid JSON. It does not guarantee accuracy. A model can return a perfectly structured object where the "sentiment" field says "positive" for a review that is clearly negative.
Structure and correctness are different problems. Schema enforcement solves the first one mechanically. The second one is still a judgment the model makes, and it can still be wrong.
For critical applications, validate the content after you validate the structure. Does the summary actually match the source material? Are the extracted values valid? Is the sentiment label consistent with the text? These checks are the same whether you got the JSON from a chat window or an API with schema enforcement.
The takeaway
When you need structured output from AI, start by writing the exact structure you want. Literal JSON. Two examples. Explicit field constraints. This is the same show-over-tell principle from earlier in the series, aimed at a specific problem.
If you need that structure to be bulletproof, the prompting approach has a ceiling. API-level schema enforcement removes that ceiling entirely. When reliable structure matters, this is one of the best reasons to explore what happens on the other side of the chat window.
The model already knows how to generate JSON. Your job is to show it exactly which JSON.
Next up: why the same prompt can behave differently in ChatGPT, Claude.ai, and the API, and what the chat window is doing that you cannot see.
Top comments (0)