You flip on strict JSON schema mode. Parse errors go to zero. Then your eval score drops two or three points and nobody can explain why. The model got more reliable and less correct at the same time. That is not a paradox — it is exactly what constrained decoding does when you point it at the wrong task.
Constrained decoding is the machinery behind OpenAI's Structured Outputs, outlines, xgrammar, vLLM's guided decoding, and llama.cpp's GBNF grammars. It guarantees the output matches a schema by editing the model's logits at every step. That guarantee is real. The cost is subtle, and most teams pay it without noticing.
TL;DR
- Constrained decoding masks the logits at each decode step so only tokens that keep the output grammar-valid can be sampled. Invalid JSON becomes structurally impossible, not just unlikely.
- The mask is powered by a finite-state machine (FSM) compiled from your schema/regex, with the allowed-token set precomputed per state — so per-step cost is a gather-and-add, not a regex match.
- It can lower accuracy because masking renormalizes probability over a tiny token set, forcing the model onto tokens it assigned low probability and suppressing reasoning tokens entirely.
- Token misalignment (grammar works on characters, the model emits BPE tokens) is a real failure mode; good engines fix it with token healing and jump-forward decoding.
- Use constrained decoding for extraction and API payloads; for reasoning-heavy tasks, let the model think in free text first, then constrain only the final field.
What is constrained decoding, actually?
Constrained decoding restricts a language model's next-token choice to the set of tokens that keep the output valid under a formal grammar. It does not re-ask the model, retry on parse failure, or fine-tune anything. It intervenes inside the sampling loop.
At each step the model produces a logit vector over the whole vocabulary — 100K+ entries for a modern tokenizer. Normally you softmax and sample. Constrained decoding inserts one operation before that: set the logits of every disallowed token to negative infinity. After softmax, those tokens have probability zero. The model literally cannot emit a character that breaks the schema.
That is the entire trick. Everything else is making it fast and making it not lie about the model's intent.
Why does constrained decoding never emit invalid JSON?
Because validity is enforced as a hard mask, not a soft preference. The schema is compiled into an FSM whose states represent "where we are in the grammar" and whose transitions are labeled with allowed tokens.
The naive version re-checks a regex against the whole partial string on every token. That is too slow for production. The insight that made libraries like outlines practical: for each FSM state, the set of vocabulary tokens that keep the string valid is fixed and can be computed once, offline, and stored as an index. At decode time you look up the current state, gather its allowed-token mask, and add it to the logits.
import torch
def make_logit_processor(fsm, vocab_size):
# fsm.allowed[state] is a precomputed LongTensor of token ids
# that keep the output grammar-valid from this state.
def process(input_ids, logits):
state = fsm.advance(input_ids) # O(1) amortized FSM walk
mask = torch.full((vocab_size,), float("-inf"), device=logits.device)
mask[fsm.allowed[state]] = 0.0 # gather the legal tokens
return logits + mask # illegal tokens -> -inf
return process
# After masking, softmax puts 0 probability on everything illegal.
# Sampling can only pick a token that keeps the JSON valid.
Per step this is a memory read plus an add — negligible next to the transformer forward pass. The expensive part (compiling the FSM and building the per-state index) happens once when you register the schema.
There is a second optimization worth knowing: jump-forward decoding (SGLang, xgrammar). When the FSM state has exactly one legal continuation — after {"temperature": the grammar forces a number, and after the last field it forces } — the engine emits those deterministic tokens without a forward pass. Structural boilerplate becomes free. On heavily-templated schemas this is a real latency win, not a rounding error.
Why can constrained decoding lower accuracy?
Because masking changes the distribution the model samples from, and the model was never trained on that clipped distribution.
Three concrete mechanisms:
1. It renormalizes over a tiny set. Say the model's honest next-token distribution puts 60% on a reasoning word, 30% spread across other prose, and 4% on {. If the grammar demands JSON right now, you mask everything except { and its friends. That 4% gets renormalized to near-100%. You are sampling from the tail of the model's belief and treating it as the head.
2. It kills the scratchpad. Chain-of-thought works because the model spends tokens computing before it commits. A schema that starts with {"answer": forbids those tokens. The model has to emit the answer field with zero prior reasoning tokens in context. For extraction that is fine. For math, multi-hop QA, or classification-with-justification, you have amputated the exact mechanism that makes the model accurate.
3. Key order becomes a silent variable. In {"answer": ..., "reasoning": ...} the answer is decoded first, so the reasoning field is post-hoc rationalization the model already can't use. Flip to {"reasoning": ..., "answer": ...} and the reasoning tokens are in context when the answer is generated. Same schema, same model, materially different accuracy. JSON object key order is part of your prompt now, whether you meant it to be or not.
The fix is not to abandon constraints. It is to constrain the shape while leaving the thinking free — put a free-text reasoning field first, or let the model produce unconstrained chain-of-thought and constrain only a final extraction call.
What is token misalignment, and why does it corrupt output?
Token misalignment is the mismatch between grammars, which operate on characters or bytes, and models, which emit BPE tokens that can straddle grammar boundaries. It is the most under-discussed failure mode in constrained decoding.
Concrete case: your grammar has just forced a closing quote ". The model, unconstrained, would have emitted the single token ", (quote-comma is one common BPE merge). But the FSM only allows the " transition here, so ", is masked out. The model is forced onto a different, rarer tokenization of the same string — pushing it off the distribution it learned. You still get valid JSON, but the token path is one the model never saw in training, and quality degrades.
Token healing is the mitigation: instead of appending a forced prefix and masking, the engine backs up over the last token, and re-decodes constrained to continuations consistent with both the model's natural tokenization and the grammar. Good libraries do this transparently. If you hand-roll a logit processor, you will hit this and your outputs will look subtly wrong in ways no schema validator catches.
This is also why "just add -inf to bad tokens" is a demo, not a product. The demo produces valid JSON. The product produces valid JSON that reads like the model wrote it.
Constrained decoding vs prompting vs prefill: which should you use?
Different providers expose different levels of this, and the guarantee is not the same.
-
OpenAI Structured Outputs (
response_formatwith a JSON schema, GPT-5.x) is true constrained decoding — a grammar enforces the schema at the token level. Validity is guaranteed by construction. -
Anthropic Claude (Opus 4.x / Sonnet 4.x) does not expose raw grammar masks. You get structure by forcing a tool call — set
tool_choiceto a specific tool and Claude fills itsinput_schema. The reliability is very high in practice, but the guarantee comes from tool-use training plus the API validating the call, not from a decoder-level FSM. -
Prefilling the assistant turn (start Claude's response with
{) is not constrained decoding at all — it is a soft nudge. Cheap, no schema guarantee, and it composes with everything.
# Claude: structure via forced tool use, not logit masking.
tools = [{
"name": "extract",
"description": "Return the extracted fields.",
"input_schema": {
"type": "object",
"properties": {
"sentiment": {"type": "string", "enum": ["pos", "neg", "neutral"]},
"confidence": {"type": "number"},
},
"required": ["sentiment", "confidence"],
},
}]
# tool_choice = {"type": "tool", "name": "extract"} forces the call.
# Claude decodes into input_schema; the API rejects a malformed call.
Decision rule I actually use: if the task is extraction, routing, or an API payload where the answer is mostly copied out of context, turn on hard constraints and don't think about it. If the task needs the model to compute or reason, keep the reasoning field unconstrained and first, and constrain only the leaf value — or run reasoning and extraction as two separate calls. Never wrap a reasoning task in a schema whose first key is the answer.
Direct answer: does constrained decoding hurt accuracy, and how do you get valid JSON safely?
Constrained decoding forces valid JSON by compiling your schema into a finite-state machine and masking every grammar-illegal token's logit to negative infinity at each decode step, with the allowed-token set precomputed per FSM state so the runtime cost is a gather-and-add. It never emits invalid output. It can lower accuracy because masking renormalizes probability onto tokens the model rated unlikely, suppresses chain-of-thought scratchpad tokens, and makes JSON key order a hidden variable — and token misalignment can force off-distribution tokenizations unless the engine does token healing. Get both reliability and accuracy by constraining structure, not thought: let the model reason in a free-text field (or a separate unconstrained call) placed before any constrained answer field, and reserve hard schema enforcement for extraction and payload tasks where there is nothing to reason about. On Claude, achieve the same with forced tool use and a well-ordered input_schema; on GPT-5.x, use Structured Outputs but keep a reasoning field first.
Top comments (0)