TL;DR — Structured output and function calling guarantee that an LLM's response parses — they say nothing about whether the values inside it are true. Teams that treat schema validation as a success metric end up shipping schema-conformant but semantically wrong tool calls, and grammar-constrained decoding can make this worse by forcing the model to commit to structure before it has finished reasoning. The fix is to stop treating the schema as a safety net and start treating it as an interface contract with its own semantic tests.
Every team shipping LLM agents eventually discovers the same relief: wrap the model's output in a JSON schema or a function-calling spec, and the chaos of free-text generation becomes a clean, parseable object. No more regexing hallucinated prose. No more "the model said yes but meant no." Validation passes, the pipeline moves on, everyone exhales.
That relief is misplaced. Structured output solves a narrower problem than most engineers think it solves, and the gap between what it guarantees and what people assume it guarantees is where production incidents live.
The Compiler Analogy Everyone Gets Backwards
Function calling and typed outputs are, structurally, a type system bolted onto a probabilistic text generator. That analogy is useful, but most teams draw the wrong conclusion from it. A type checker in a normal codebase catches shape errors: wrong argument count, mismatched types, a missing field. It does not catch logic errors. A function that computes a refund as price + tax instead of price - discount will pass mypy or tsc every single time, because the type checker was never asked whether the computation is correct — only whether it's well-formed.
A JSON schema attached to an LLM's output does the same job, and only that job. It verifies that refund_amount is a number, that customer_id is a string, that status is one of three enum values. It cannot verify that the refund amount is the right refund amount, that the customer ID matches the actual customer in the conversation, or that "resolved" was the correct status for a ticket the model never actually resolved. Schema validation is a syntax check wearing a correctness costume.
This matters because structured output changes the failure mode without eliminating it. Before schemas, a broken LLM response looked broken — malformed JSON, a stray sentence before the braces, something a human glancing at logs would flag immediately. After schemas, a broken response looks identical to a correct one. It parses. It passes validation. It gets forwarded to the payments API or the database write with full confidence, because confidence was never the thing being measured.
Constrained Decoding Trades Reasoning for Compliance
The deeper issue shows up with grammar-constrained decoding — the mechanism underneath most "guaranteed JSON" tooling, where the sampler is restricted at every token step to only emit tokens that keep the output on a path toward valid schema conformance. This is a genuinely clever engineering trick, and it does eliminate malformed output as a category of error. But it has a side effect that doesn't get discussed enough: it forces the model to start emitting structure before it has finished reasoning about the content that goes inside that structure.
Chain-of-thought works, when it works, because the model gets to lay out intermediate tokens before committing to an answer. Grammar-constrained decoding for a tool call often removes that runway. The first token has to be {, the second has to be a quote, the key has to match one of a fixed set of field names — there's no room left for "let me think about which tool actually applies here" unless the schema was deliberately designed to include a reasoning field before the decision fields. Most schemas aren't designed that way, because nobody writing a function-calling spec is thinking about the decoding dynamics; they're thinking about what the downstream system needs to consume.
The practical consequence: a model that would pick the correct tool and the correct arguments if allowed to reason in free text first can pick the wrong tool with full schema conformance when forced to commit immediately. You traded a visible failure (malformed output) for an invisible one (confidently wrong output in a perfectly valid shape). That's a bad trade if nobody notices it happened.
The Retry Loop That Teaches Models to Fake It
Most structured-output pipelines in production have a repair loop: validation fails, the error gets fed back to the model, the model tries again. This is good engineering for syntax errors — a missing closing bracket, a field typed as a string instead of a number. It is much worse for semantic errors, because the retry loop has no way to distinguish "the model emitted invalid JSON" from "the model emitted valid JSON that is wrong," and it only fires on the former.
A model that guesses an argument value, gets it schema-approved, and ships it will never trigger a retry. There is no feedback signal telling the pipeline that the guess was wrong, because wrongness isn't checked. Over many requests, teams that monitor "schema validation pass rate" as a reliability metric watch it climb toward near-perfect while the thing they actually care about — did the tool call do the right thing — stays flat or degrades. The metric and the goal have quietly decoupled, and because the metric is cheap to measure and the goal is expensive to measure, the metric wins the dashboard.
Where the Contract Actually Belongs
The fix isn't a stricter schema. You can add regex patterns, min/max bounds, and nested enums until the schema is airtight, and you will still only be constraining shape. The fix is recognizing that a typed interface between an LLM and the rest of your system is exactly like a typed interface between two services owned by different, mutually
Top comments (0)