Grammar-Constrained Decoding: Why Valid JSON Gets Wrong Answers

You turned on strict JSON mode. Schema violations went to zero. Your extraction accuracy went down four points, and nobody on the team can explain why, because every response parses cleanly and looks plausible.

That's the failure mode of grammar-constrained decoding: it doesn't fail loudly. It silently changes which token the model emits at the exact moment the answer is being decided, and then it happily commits to that choice for the rest of the generation. Valid syntax is not evidence of a correct answer — it's evidence the mask worked.

TL;DR

• Grammar-constrained decoding sets disallowed logits to -inf and renormalizes over what's left. The model never "chooses" the constrained path — probability mass gets redistributed onto tokens it may have ranked 40th.
• There is no backtracking. A finite-state machine walks forward one token at a time. Once the mask forces an opening quote for a field, the model must invent a string value even if the correct answer was "not present in the document."
• Grammars operate on bytes; models emit BPE tokens. When a grammar bans the natural merged token (": ", "unknown"), the model is pushed onto rare token paths it saw almost never in training, and quality degrades downstream of the split.
• The fix isn't dropping structure — it's making the schema wide enough that the constraint is never the thing making the decision. Add null/unknown escape hatches, avoid regex pattern on free text, and let reasoning happen outside the constrained span.
• Measure it with a paired eval: unconstrained + parse + retry vs. constrained, scoring task accuracy, not validity rate.

What actually happens to the logits when you constrain decoding?

At each decode step, a constrained decoder computes a boolean mask over the entire vocabulary — 128K entries for Llama 3, ~200K for the o200k family — marking which tokens keep the grammar's automaton in a live state. Disallowed logits are set to -inf, then softmax runs over the survivors.

That last part is the part people skip. It is not a filter. It is a renormalization:

import numpy as np

# Model's true distribution over 5 candidate next tokens
logits = np.array([8.1, 7.9, 3.2, 3.0, 2.8])
labels = ["I", "The", '"', "Based", "Un"]

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

print(dict(zip(labels, softmax(logits).round(3))))
# {'I': 0.52, 'The': 0.42, '"': 0.02, 'Based': 0.02, 'Un': 0.01}

# Grammar says: the next token must open a JSON object/string.
mask = np.array([False, False, True, False, False])
constrained = np.where(mask, logits, -np.inf)

print(dict(zip(labels, softmax(constrained).round(3))))
# {'I': 0.0, 'The': 0.0, '"': 1.0, 'Based': 0.0, 'Un': 0.0}

A token the model assigned 2% probability is now emitted with certainty. That's fine when the model wanted to produce JSON and the mask is just enforcing punctuation. It is not fine when the mask is arbitrating a semantic choice — and every schema does that somewhere.

The clean mental model: constrained decoding samples from P(token | context) / Z restricted to the grammar's support. When the grammar's support and the model's high-probability region overlap, you lose nothing. When they diverge, you're sampling from the tail and calling it a structured output.

Why does constrained decoding change the answer, not just the format?

Because the automaton has no backtracking, and the model has no way to signal "wrong branch."

Take an extraction schema with a required "price" field of type number. The document doesn't mention a price. Unconstrained, the model would write "the document does not state a price." Constrained, the mask walks it to "price": and then permits only [0-9-]. There is no token in the allowed set that means absent. So it emits a digit. Every subsequent digit is then conditioned on that first hallucinated digit, and the model's own coherence pressure makes the number look deliberate.

This is the mechanism behind most "strict mode lowered my accuracy" reports. The schema removed the model's ability to abstain, and abstention was the correct answer for some slice of your inputs. The confabulation rate on that slice goes to ~100%, and because the output is well-formed, downstream validation never catches it.

The same trap shows up with enum. A three-value enum forces every input into one of three buckets, including inputs that belong in none of them. You've converted a recognizable failure (unparseable output, or a hedge) into an unrecognizable one (a confident wrong label).

Why do token masks fight the tokenizer?

Grammars are defined over characters or bytes. Models emit BPE tokens, and BPE tokens routinely straddle grammar boundaries. ": is one token. ": " is often one token. {" is one token. "unknown" might be one token in one tokenizer and four in another.

Modern libraries handle this correctly at the state-machine level — Outlines precomputes an FSM-state → allowed-token-IDs index, and XGrammar/llguidance use byte-level tries with adaptive masking so the per-step cost stays in the microsecond range. Correct, meaning the emitted byte stream always satisfies the grammar. But correct is not the same as in-distribution.

When the grammar forbids the natural merged token, the model must express the same bytes through a rarer token path. If your schema uses "pattern": "^[A-Z]{3}-\\d{4}$" on an ID field, the model can no longer emit the ID as the handful of tokens it saw in pretraining; it emits a character-ish path the grammar permits. Those sequences have far lower training frequency, the hidden states after them are less well-calibrated, and the degradation carries into subsequent fields, not just the constrained one.

Whitespace is the cheap version of this. A JSON grammar that permits flexible whitespace lets the model use the pretty-printed token paths it actually saw in training data. A grammar that forbids spaces after : forces the compact path. Both are valid JSON; only one matches the distribution the weights encode.

Why is the first request with a new schema slow?

Because the grammar has to be compiled into a state machine, and the mask index built for it. On deeply nested schemas with many anyOf branches, this can dominate the latency of a short generation. Anthropic's structured outputs make this explicit: new schemas incur a one-time compilation cost, then hit a 24-hour cache — so a service that mints a schema per request pays that cost every time.

The practical rule: schemas are static assets. Define them at module scope, serialize them deterministically, and never interpolate a request ID or timestamp into a description field. A schema that varies per request is a schema that never caches — and on providers where schema bytes participate in the prompt prefix, it also invalidates your prompt cache.

How do you keep valid JSON without the accuracy hit?

Three moves, in order of impact.

1. Give the model somewhere to put "I don't know." Every field that can be absent should be nullable, and every enum should carry an explicit escape member. This is the single highest-leverage change:

SCHEMA = {
    "type": "object",
    "properties": {
        "price":     {"type": ["number", "null"]},
        "currency":  {"type": ["string", "null"]},
        "category":  {"enum": ["hardware", "software", "services", "unknown"]},
        "evidence":  {"type": "string",
                      "description": "Verbatim span supporting the extraction, or '' if none."},
    },
    "required": ["price", "currency", "category", "evidence"],
    "additionalProperties": False,
}

Now the grammar's support contains a token sequence that means absent, and the model's abstention probability has somewhere to land instead of being redistributed onto a fabricated digit.

2. Use the provider's strict mode rather than hand-rolling a grammar. With the Anthropic API, output_config.format constrains the response, and strict: true on a tool constrains that tool's arguments:

import anthropic

client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=4096,
    output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    messages=[{"role": "user", "content": document}],
)

Know the constraints before you design the schema. Structured outputs support the basic types, enum, const, anyOf, allOf, $ref/$defs, and the standard string format values — and require additionalProperties: false. Recursive schemas, numeric bounds (minimum, maximum, multipleOf), and string length bounds (minLength, maxLength) are not supported; the Python and TypeScript SDKs strip unsupported keywords from the wire schema and validate them client-side. Two more sharp edges worth internalizing: a safety refusal (stop_reason: "refusal") is not guaranteed to match your schema, and hitting max_tokens mid-object gives you truncated JSON, not an error. Branch on stop_reason before you call json.loads.

On self-hosted vLLM, the equivalent is a guided-decoding backend:

from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct",
          guided_decoding_backend="xgrammar")

params = SamplingParams(
    temperature=0.0,
    max_tokens=512,
    guided_decoding=GuidedDecodingParams(json=SCHEMA),
)

3. Keep reasoning outside the constrained span. The constraint should apply to the transcription of an answer, not to its derivation. Either use a model with native thinking (adaptive thinking on Claude 4.x runs before the constrained output block), or split into two calls: one unconstrained call that reasons and cites evidence, one cheap constrained call that transcribes that reasoning into the schema. The second call is a near-deterministic formatting task, which is exactly the regime where masking costs nothing.

How do you tell whether grammar-constrained decoding is hurting you?

Run a paired eval on the same inputs, three arms:

1. Constrained decoding with your production schema.
2. Unconstrained generation + json.loads + one retry on parse failure.
3. Constrained decoding with the widened schema (nullables, enum escape hatch).

Score task accuracy, and separately track the abstention rate — how often each arm produces null/unknown. If arm 1's abstention rate is near zero while arm 2's is 8%, you've found your regression: the grammar is manufacturing answers for inputs that don't have them. Also log validity rate for arm 2; if it's already 98%, the entire justification for hard constraints is a 2% retry, and you may be paying for that in accuracy.

The direct answer

Grammar-constrained decoding produces valid JSON with wrong answers because masking is not filtering — it zeroes disallowed logits and renormalizes, so probability mass the model assigned to "no price is stated" gets redistributed onto whatever digit the grammar permits, and a forward-only state machine gives the model no way to take it back. The tokenizer compounds it: when the grammar bans the merged token the model would naturally emit, generation continues along low-frequency token paths whose hidden states are less reliable. Keep the structure, but widen the schema until abstention is representable, keep regex patterns off free-text fields, treat schemas as cached static assets, and let the model reason before the constrained span rather than inside it. Then measure task accuracy, not validity rate — validity was never the thing that was broken.