An LLM that returns a beautiful paragraph is useless to the function that has to consume it. Real systems don't want prose — they want data. Getting reliable, structured data out of a model is the skill that turns a demo into a product.
The gap between "cool LLM demo" and "LLM in production" is almost always this: the demo prints text for a human to read, while the product needs a machine to parse the output and act on it. Free-form text is where that breaks. Here's how to close it.
Why free text fails in a pipeline
Say you ask a model to extract a customer's name, order number, and issue from a support email. It happily replies: "Sure! The customer is Priya, her order is 4471, and she's reporting a damaged item." Lovely for a human. Now write the code that reliably pulls three fields out of that — across thousands of emails where the model phrases it differently every time, sometimes adds a preamble, sometimes reorders things. You can't. Regex-ing an LLM's prose is a trap you'll maintain forever.
The fix isn't a cleverer parser. It's forcing the model to answer in a structured format in the first place.
The ladder of reliability
There's a progression from "hope" to "guarantee":
1. Ask for JSON in the prompt. "Respond with a JSON object with keys name, order_id, issue." Better than nothing, but the model can still wrap it in markdown, add commentary, or produce malformed JSON. Hope-based.
2. JSON mode / structured output modes. Most serious model APIs now offer a mode that guarantees syntactically valid JSON. That removes the "is it even parseable" class of bugs entirely.
3. Schema-constrained generation. You supply a schema — the exact fields, types, and which are required — and the output is constrained to match it. Now you're not just getting valid JSON, you're getting your JSON.
4. Function / tool calling. The same machinery, framed as "call this function with these typed arguments." This is how agents act on the world: the model emits a structured call, your code executes it. It's structured output wearing a different hat.
Validate at the boundary
Even with structured output, treat the model's response as untrusted input crossing into your system. Define the shape you expect with a schema — in Python, a library like Pydantic is the standard — and validate every response against it. If it doesn't conform, you catch it immediately instead of letting a malformed object propagate three layers deep and crash somewhere far from the cause.
The pattern is: model produces structured output → validate against a schema → hand a clean, typed object to the rest of the system. That validation step is the seam between the probabilistic world of the model and the deterministic world of your code, and it's a discipline I hold across everything I build.
Why this matters more than prompt wording
People spend days tuning prompt phrasing and skip the thing that actually makes an LLM integrable: a typed, validated contract for its output. A mediocre prompt with a strict schema beats a beautiful prompt with free text every time, because the first one your system can trust and the second one it can only hope about.
Seeing it in a real system
Typed contracts are foundational, not decorative. In "Saturday MK1: an AI assistant that's a system, not a prompt", the whole service is built on FastAPI and Pydantic v2 — every request and response is a typed, validated model, so the boundaries between the agent's engines are contracts rather than hope. That's what lets independently-built components compose without quietly corrupting each other's data.
Stop asking an LLM to talk to your system. Make it fill out a form your system defined. That single shift is most of what separates a toy from a tool. More of how I do it at www.divyakush.com.
Divyakush Punjabi · Full-Stack & AI Engineer
Portfolio · GitHub · LinkedIn
Top comments (0)