JSON Mode Makes Your LLM Dumber: The Constrained Decoding Trap

I shipped a classifier that was more reliable and more wrong at the same time.

Same model. Same prompt. Same temperature. The only diff in the commit was a response_format block. Parse errors went to zero, my retry loop went quiet, and my accuracy quietly fell off a cliff. That was the day I learned that JSON mode makes your LLM dumber in a way that no error log will ever show you.

Nobody pages you for this. The output is valid. The schema validates. The pipeline is green. The answers are just worse.

TL;DR

• JSON mode is not a request, it's a filter. Structured outputs work by masking illegal tokens to negative infinity at every decoding step, then renormalizing what's left.
• Masking removes the model's escape hatches. If your schema opens with {"answer":, the model must commit to a verdict on token one, with zero room to reason first.
• Field order is a prompt. Generation is left to right, so a reasoning field placed before answer recovers most of the loss. Placed after, it's a postmortem the model writes to justify a guess it already made.
• Required fields manufacture hallucinations. A non-nullable price: number forces the model to invent a number when the document doesn't have one. It has no legal token for "I don't know."
• The fix is boring: reason first, make unknowns representable, keep schemas shallow, and for hard tasks, do two passes instead of one.

What does JSON mode actually do under the hood?

It clamps the sampler. That's it. There is no separate "structured brain" in the model.

At every decoding step, a normal LLM produces logits over the whole vocabulary, softmaxes them, and samples. Constrained decoding inserts one step in the middle: a state machine (compiled from your JSON Schema into a grammar) computes the set of tokens that could legally come next, sets every other logit to -inf, and then softmaxes.

That last part matters more than people realize. The probabilities you sample from are renormalized over survivors. If the model wanted to put 92% of its mass on "Let" (as in "Let me check the invoice header first") and your grammar only permits "refund", "billing", and "other", those three tokens might have shared 0.4% of the original distribution. After renormalization, one of them has 71% probability.

The model didn't decide anything. Your grammar picked from the noise floor and the softmax made it look confident.

Worth separating three things that get mushed together:

• Prompt-only JSON ("respond in JSON, no prose"). No enforcement. Model can still think, can still fail to parse.
• JSON mode. Guarantees syntactically valid JSON. Says nothing about your fields.
• Structured outputs / schema-constrained decoding. Guarantees your exact schema. Strongest guarantee, strongest distortion.

Guarantee strength and reasoning damage go up together. That's the trade nobody puts in the docs.

Why does JSON mode make your LLM dumber?

Because a language model reasons by writing. Tokens are the scratchpad. Take away the scratchpad and you've taken away the compute.

Chain of thought isn't a personality quirk, it's the model spending more forward passes on a problem before committing. Every intermediate token is another pass through the stack, another chance to route through the right circuits. When your schema forces the first emitted token to be part of {"category":, you cut the available thinking budget to roughly zero and then act surprised that a hard classification got sloppy.

I ran my own scrappy check on this, maybe 120 support tickets I had already hand-labeled. Nothing publishable, no error bars, one model, one afternoon. Free-text answers with a "explain, then give me the label on the last line" prompt landed around the high 80s. The same prompt with a strict two-field schema, label first, dropped into the mid 70s. Reordering the schema so reasoning came first pulled it back to within a couple points of the free-text run.

One line moved in a JSON file. That was the whole intervention.

Does field order in your schema really matter?

Yes, and it's the highest-leverage thing in this post. Decoding is left to right, and most structured-output implementations emit object keys in the order your schema declares them.

So this schema is a trap:

{
  "type": "object",
  "properties": {
    "is_fraud": { "type": "boolean" },
    "reasoning": { "type": "string" }
  },
  "required": ["is_fraud", "reasoning"]
}

The model emits true or false first, then writes a paragraph explaining a decision it already locked in. That's not reasoning, that's a press release. Worse, it's actively misleading during debugging, because the explanation always sounds coherent and always agrees with the verdict.

Flip it:

{
  "type": "object",
  "properties": {
    "evidence": { "type": "string", "description": "Quote the exact lines that support your call." },
    "reasoning": { "type": "string" },
    "is_fraud": { "type": "boolean" }
  },
  "required": ["evidence", "reasoning", "is_fraud"]
}

Now the boolean is conditioned on tokens the model actually generated. Same schema, same fields, opposite behavior.

Two smaller things in the same family:

Field names leak into the answer. has_security_issue and is_code_safe are the same question with the polarity flipped, and they will not give you mirror-image results. Key names are tokens in context. They're prompt text with extra steps.

Descriptions are free real estate. Most schema formats let you attach a description per field and it lands in the model's context. That's where your edge-case rules go, right next to the field they govern, instead of buried in paragraph four of a system prompt.

Do required fields cause hallucinations?

They do, and it's the most under-discussed bug in structured extraction. A schema is a contract that the model must produce a value. If reality doesn't contain one, the model doesn't get to abstain. There's no legal token for silence.

Set "price": {"type": "number"} as required, hand the model a PDF with no price, and it will emit a number. 0. 99.99. Whatever fits the shape of the surrounding text. Your validator will wave it through because the type is correct and your downstream code will treat a fabricated 0 as a real free product.

Make unknowns representable:

• Use nullable types (["number", "null"]) and mean it.
• Add an explicit "unknown" member to every enum. Yes, every one.
• Add a confidence enum (high / medium / low) rather than a float, because a model's self-reported 0.87 is theater.
• Never require an array with minItems: 1 for "list the issues you found." You just mandated at least one issue. It will find one.

How do you get structured outputs without losing accuracy?

Stop making one call do two jobs. Reasoning and formatting are different tasks, and squeezing them through the same grammar is where the damage happens.

The two-pass pattern, in order of how much I reach for it:

1. Reason-first schema. Free-text reasoning field, declared first, then the constrained fields. Covers most cases and costs one line.
2. Split the calls. Call one: no constraints at all, let the model think in prose. Call two: a small cheap model with a strict schema whose only job is turning that prose into JSON. Extraction from clean text is an easy task and constraining it costs you nothing.
3. Flatten the schema. Deeply nested objects with long enums are more grammar states, more masking, more off-distribution token boundaries. A flat schema with six fields beats a beautiful nested one with three levels.
4. Keep a control run. Whenever you tighten a schema, re-run your eval set unconstrained. If the gap is big, your schema is doing the damage, not the model.

There's also a subtler failure worth knowing about: token boundaries. Models learn a specific tokenization of text, and a grammar can force a split the model has essentially never seen in training, like emitting { and " separately when its training data is full of {" as one unit. Every forced-off-distribution boundary is a small tax on the next prediction. You mostly can't fix this from userland. Just know it's why the same content sometimes comes out worse inside a schema than outside one.

So does JSON mode make your LLM dumber?

Yes, and now you know the mechanism: JSON mode makes your LLM dumber because schema-constrained decoding masks illegal tokens to -inf and renormalizes the survivors, which strips away the intermediate tokens the model uses to think and forces confident answers out of a distribution that had almost no mass on them. The damage is invisible because the output stays perfectly valid. Put a reasoning or evidence field first in your schema, make every unknown expressible with nulls and "unknown" enum members, keep schemas flat, and split hard problems into a free-text reasoning call followed by a cheap constrained extraction call. The guarantee is worth having. Just stop paying for it with accuracy you never measured.