Constrained Decoding: Your JSON Schema Key Order Is a Prompt

I flipped one support-ticket classifier from free text to a strict JSON Schema and accuracy fell off a cliff. Same model. Same temperature. Same prompt. Same 300 tickets in my eval set. The only thing that changed was that the output now parsed 100% of the time.

That guarantee comes from constrained decoding, and constrained decoding is not a formatter that runs after the model. It runs inside the sampling loop, one token at a time, and it can quietly rewrite what the model is allowed to think.

TL;DR

• Constrained decoding masks logits at every decode step. A grammar compiled from your JSON Schema sets every disallowed token to -inf before sampling. The model never "fixes up" output afterward.
• Key order in properties is generation order in grammar-based implementations. If label comes before reasoning, the model commits to an answer with zero reasoning tokens in its own context.
• Moving the reasoning field first took my 300-ticket eval from 71% to 84% exact-match on a small local model. One key reorder, no prompt change. Your delta will differ; the direction won't.
• Over-constraining costs accuracy even with good key order, because the mask prunes the model's preferred token path and pushes it onto off-distribution continuations (tight whitespace, long enum strings).
• Fix: reasoning fields first, short distinct enum values, whitespace allowed in the grammar, and log how often the mask overrides the model's top token.

What does constrained decoding actually do?

Constrained decoding compiles your schema into a state machine, then at every decode step it asks that machine "which token ids are legal from here?" and kills everything else before sampling.

logits = model(tokens)                      # [vocab_size], e.g. 128k floats
allowed = grammar.allowed_tokens(state)     # sometimes 23 ids. sometimes 1.
logits[~allowed] = float("-inf")
next_id = sample(logits, temperature=0.7)   # sampling happens AFTER the mask
state = grammar.advance(state, next_id)

vLLM guided_json, Outlines, XGrammar, llama.cpp GBNF, and provider-side strict JSON modes all differ in how fast they build that mask. They do not differ in the shape of the loop. Four consequences fall out of it:

1. The model cannot backtrack. A token that got sampled is in the transcript forever.
2. When the mask leaves exactly one legal token, the model's opinion is irrelevant for that step. Punctuation, quotes, and your key names are dictated.
3. Probability mass is redistributed. If the model wanted a token you banned, that mass lands on the best surviving option, which may be a bad one.
4. Everything the model emits, it also reads. Output is autoregressive context.

Point 4 is where key order stops being cosmetic.

Why does JSON Schema key order change your LLM's answer?

Because the model can only condition on tokens it has already emitted, and in a grammar-based implementation the key order in your schema is the order it emits them.

Look at these two schemas. They validate the exact same set of documents.

# Schema A: answer first
{"type": "object",
 "properties": {
   "label":     {"enum": ["billing", "bug", "feature", "spam"]},
   "reasoning": {"type": "string"},
   "confidence":{"type": "number"}},
 "required": ["label", "reasoning", "confidence"],
 "additionalProperties": False}

# Schema B: reasoning first
{"type": "object",
 "properties": {
   "reasoning": {"type": "string"},
   "label":     {"enum": ["billing", "bug", "feature", "spam"]},
   "confidence":{"type": "number"}},
 "required": ["reasoning", "label", "confidence"],
 "additionalProperties": False}

Schema A forces this transcript:

{"label": "billing", "reasoning": "The user is asking about a refund...

The "billing" token was sampled when the model's context contained the ticket, the system prompt, and the four characters {"la. It had no working space. The reasoning field that follows is not reasoning, it is a post-hoc justification generated after the answer is locked. It will rationalize a wrong label as happily as a right one.

Schema B forces the model to spend 40 or 80 tokens on the ticket before the enum position arrives. That is chain-of-thought, just wearing a JSON key. On my 300 tickets, Schema B beat Schema A by 13 points of exact-match label accuracy. One reorder. Same prompt, same seed, single run on a small local model, so treat the number as a direction rather than a benchmark.

One caveat worth thirty seconds of your time: not every implementation pins key order. Some grammars accept keys in any order, and some providers reorder to match the declaration list. Find out which one you have. Generate 20 samples, print the raw string before it hits json.loads, and look at the first key. json.loads destroys the evidence, because Python dicts will happily show you insertion order that came from the wire, not from your schema.

Why does forcing JSON make a model dumber even with good key order?

Because the grammar reasons in characters and the model reasons in tokens, and the two disagree at the boundaries.

Your model saw billions of tokens of pretty-printed JSON during training: newline after the brace, two-space indent, a space after every colon. A tight grammar that emits {"label":"billing"} walks the model through a byte sequence it rarely saw. The token ":" and the token ": " are different ids with different continuation distributions. Once you commit to the compact one, every downstream prediction is slightly off-manifold. This is the same family of problem as token healing in prefix completion: a character-level constraint slices through a merge the tokenizer wanted to make.

So: let the grammar allow optional whitespace, and ask for pretty-printed output. It costs you maybe 15 extra tokens per call and it puts the model back on familiar ground.

The second tax is enum shape. Consider ["NEEDS_HUMAN_ESCALATION", "NEEDS_HUMAN_REVIEW"]. Both options start with the same several tokens. The mask happily emits that shared prefix, and the actual decision gets deferred to the token where the strings finally diverge, by which point the model has already half-committed to a string it may not have chosen at step one. Short labels with distinct first characters keep the decision in a single step, which also means the class probability is readable straight off one logprob.

How do you measure whether your grammar is hurting you?

Log how often the mask overrules the model. That number is the honest measure of how much constraint you are applying, and almost nobody records it.

class MaskPressure:
    """Wrap your grammar's logits processor and count overrides."""
    def __init__(self, inner):
        self.inner, self.steps, self.forced = inner, 0, 0

    def __call__(self, input_ids, scores):
        free_top = scores.argmax(-1)
        masked = self.inner(input_ids, scores)
        self.steps += 1
        self.forced += int((masked.argmax(-1) != free_top).item())
        return masked

    @property
    def pressure(self):
        return self.forced / max(self.steps, 1)

Structural tokens (braces, quotes, key names) will always show as forced, so the absolute number is not the signal. The comparison is. Run the same 50 prompts through two schema variants and look at which one fights the model less. When I tightened a schema and pressure jumped, quality dropped in the same direction every time.

Three more things that reliably help:

• Keep required honest. OpenAI-style strict mode makes every property required and bans additionalProperties. A required error_message field on a successful call forces the model to invent one. Model optionality as a union with null, not as an absent key.
• Leave one escape hatch. A free-text notes field gives the model somewhere to put the thought that does not fit your enums, instead of jamming it into a label.
• Two-pass when it matters. Generate freely, then run a cheap second call that only extracts. You pay another request and you get a model that was never masked while it was thinking. For classification I use one pass with reasoning-first; for anything with real stakes, two.

So does JSON Schema key order really change your LLM's answer?

Yes, and the mechanism is not mysterious. Constrained decoding applies a per-step logit mask built from your schema, so in grammar-based implementations the declaration order of properties becomes the literal generation order of the output. Any field declared before the answer becomes context the model can reason over; any field declared after it is a justification of a decision already made. Put reasoning fields first, keep enum values short and distinct, let the grammar allow the whitespace the model expects, and log how often the mask overrides the model's top token. You keep the parse guarantee and stop paying for it in accuracy.

---

Written by the developer behind Preterview, an interview prep platform.