Every application that puts a language model in front of real users eventually hits the same wall. The model returns text. Your application needs data. Those are not the same thing, and the gap between them is where a surprising amount of production code ends up living.
You ask for JSON. You get something that looks like JSON. Most of the time it parses. Sometimes there is a trailing comma, a field spelled a little differently than your code expects, a number wrapped in quotes, or a friendly sentence sitting in front of the opening brace. In a demo you fix that by hand. In production it pages you.
The Eighty Five Percent Problem
Prompting a model to answer in JSON lands somewhere between 85 and 95 percent of the time, depending on the model and how awkward the schema is. That sounds acceptable until you multiply it by traffic. At ten thousand calls a day, 95 percent means five hundred failures. Each one either crashes a pipeline, burns a retry, or quietly writes bad data that somebody finds a week later.
Teams route around this in three ways, and all three cost something.
The first is prompt engineering. Repeat the schema, add examples, ask nicely. This raises the hit rate and never reaches certainty, so you still need the fallback path.
The second is post processing. Regex extraction, a JSON repair library, a retry loop with a budget. This adds latency and a category of bugs that has nothing to do with your actual product.
The third is routing everything through function calling, even when you do not want a function called, purely because the tool argument path is more reliable than the plain text path.
How Constrained Decoding Actually Works
The real fix happens a layer below the prompt. At every step a model produces a probability distribution over its whole vocabulary, and normally any token in that distribution can be sampled. Constrained decoding puts a grammar in front of that step and masks out every token that could not legally come next given the schema and what has already been emitted.
If the schema says the next thing must be a field name from a fixed set, every token that does not start one of those names gets zeroed out. If a value must be an integer, quote characters are unreachable. The model still chooses, it just cannot choose its way outside the shape.
That is why the guarantee is different in kind from prompting. It is not that the model has been persuaded to behave. It is that the invalid branches of the output space were removed before sampling. Malformed JSON is not unlikely, it is unreachable.
JSON Mode Is Not Schema Enforcement
This distinction trips up a lot of teams, because both features get marketed with the same words.
JSON mode means the output will be syntactically valid JSON. It will parse. It says nothing about whether your fields are present, whether they are named what you expect, or whether the types match. You can absolutely get a clean parse of an object that is useless to your code.
Schema enforcement means the output conforms to the schema you supplied. Fields present, names exact, types correct, enums respected. That is the one worth wiring up, and it is worth checking which one you actually enabled, because the failure mode of assuming the wrong one is data that parses and then breaks something downstream.
Support and the exact shape of the API differ by provider, and nested objects, unions and recursive schemas are where the differences show up fastest. The full breakdown of how the mechanism works, what each provider supports, and the production patterns that come with it is here: https://www.adaptiverecall.com/structured-output/
What Still Belongs In Your Code
Constrained decoding removes a class of failure. It does not remove the need for judgment.
Keep your validation layer. It costs microseconds and it catches the case where the schema itself was wrong, which is now the more likely bug.
Keep semantic checks. A schema can guarantee that confidence is a float between 0 and 1. It cannot guarantee the number means anything.
Watch the schema complexity. Deeply nested or heavily recursive schemas push the model into narrow corridors and can cost quality on the reasoning that fills them. Flatter schemas usually produce better content inside the fields.
Handle refusals separately. A model that declines to answer still has to say so somehow, and if your schema has no room for that, you get a well formed object full of nothing.
The Takeaway
The parsing code most teams carry is not a sign that models are unreliable. It is a sign that the wrong layer is being asked to solve the problem. Move the guarantee down into decoding, and the retry loop, the repair library and the 3 AM page all go away together, leaving validation to do the small job it was always meant to do.
Top comments (0)