Why “Return Valid JSON” Is Not a Decoding Constraint

A model can know what valid JSON looks like and still produce a response that your parser rejects.

For example, each of these responses contains a valid-looking JSON object.

Here is the result:

```json
{"answer": "72"}
```

{"answer": "72"}

The calculation is 48 + 24.

{"answer": "72"}
{"answer": "72"}

A person can identify the answer immediately.

A program expecting exactly one JSON value cannot.

This is the distinction I wanted to understand more concretely:

Prompting changes which outputs are likely. Constrained decoding changes which outputs are allowed.

I built a small local lab around Qwen2.5-0.5B-Instruct to inspect that distinction at the tokenizer, generation, validation, and logit-processing levels.

The repository contains the scripts, raw generations, observations, and JSONL evaluation logs:

github.com/Vaibhav701161/constrained-decoding-lab

The experiment

I compared two generation conditions on three simple grade-school math problems.

The free-form prompt asked the model to show brief reasoning and end with a numeric answer.

The structured prompt asked it to return exactly one JSON object:

{
  "reasoning": "your reasoning here",
  "answer": "final numeric answer here"
}

Generation was deterministic, using:

do_sample=False

For every example, the evaluation script recorded:

• the prompt and raw output
• the parsed representation
• JSON validity
• schema validity
• the extracted answer
• exact-match correctness
• prompt and generated token counts
• latency
• parsing or validation errors

The initial smoke-test result was:

Condition	Examples	Valid JSON	Correct
Free form	3	N/A	3/3
Prompted JSON	3	0/3	0/3

This should not be treated as a benchmark.

Three hand-selected examples are far too few, and the extraction logic is intentionally simple.

The failure modes were still useful.

The JSON-prompted outputs included:

• Markdown-fenced JSON
• valid-looking JSON followed by prose
• multiple JSON objects in one response
• continued generation after a complete object

In several cases, the answer was visibly present.

The complete response still failed JSON parsing.

That is the practical limitation of relying only on instructions such as:

Return ONLY valid JSON.
No markdown.
No extra text.

The instruction can make compliant output more probable.

It does not remove non-compliant output from the model’s possible decoding paths.

Validation happens after the mistake

A JSON parser can tell us that the output is invalid after generation.

A schema validator can tell us that the parsed object contains the wrong keys or value types.

Recovery logic may be able to extract the first complete object from a longer response.

These are all useful techniques, but they solve a different problem.

Post-generation validation asks:

Did the model produce something acceptable?

Constrained decoding asks:

Which tokens can the model legally produce next?

The first detects failure.

The second attempts to prevent structural failure.

This does not mean constrained decoding automatically makes an answer correct.

It only means the generated sequence is forced to remain inside a defined language.

A perfectly valid JSON object can still contain the wrong answer.

Structural validity and semantic correctness are separate evaluation dimensions.

Touching the decoding loop

To understand the underlying mechanism, I implemented a small Hugging Face LogitsProcessor.

The processor scans the tokenizer vocabulary and records token IDs whose individually decoded text contains a digit:

for token_id in range(len(tokenizer)):
    piece = tokenizer.decode(
        [token_id],
        skip_special_tokens=False,
    )

    if any(character.isdigit() for character in piece):
        self.banned_token_ids.append(token_id)

At every generation step, it modifies the scores of those tokens:

scores[:, self.banned_token_ids] = -float("inf")

After softmax, those tokens receive zero probability.

The decoder cannot select them.

This is not a grammar-constrained decoder.

It is a static token mask used to observe the basic decoding-time operation directly.

The result was more interesting than simply “the model stopped generating digits.”

It avoided ordinary ASCII digits, but produced digit-like Unicode characters such as circled numbers.

The formatting and output quality also degraded.

That exposed two important problems.

A constraint enforces its implementation, not its intention

“Prevent the model from outputting a number” is an informal intention.

“Set the logits of tokens whose individually decoded strings contain a digit to negative infinity” is an implementation.

Those are not equivalent.

A number can be represented through:

• ASCII digits
• other Unicode digit characters
• number words
• symbols that visually resemble numbers
• token combinations missed by a naive vocabulary scan

The decoder only guarantees the rule that has actually been encoded.

This is especially important when people say that constrained decoding guarantees valid output.

The guarantee is relative to:

• the grammar being enforced
• the tokenizer representation
• the parser state
• byte and Unicode handling
• the definition of an accepting state
• stopping behaviour

A narrow or incorrect formalisation can provide a strong guarantee for the wrong language.

Hard constraints can damage output quality

The original model distribution may place most of its probability mass on a small set of natural continuations.

When those continuations are masked, the decoder must choose from what remains.

The remaining token may be structurally legal but awkward, unlikely, or semantically poor.

This is why evaluating constrained decoding only through a validity percentage is incomplete.

A useful comparison should also measure:

• task accuracy
• latency overhead
• generated token count
• dead ends
• probability mass removed
• output quality
• recovery behaviour
• behaviour across different schemas and prompt styles

A system can produce 100% syntactically valid JSON and still be worse for the actual task.

The real difficulty: grammars operate on bytes, models emit tokens

Setting selected logits to negative infinity is straightforward.

Determining which logits should be masked is the difficult part.

A grammar is usually defined over characters or bytes.

A language model emits tokens.

One token can contain:

• one character
• several characters
• whitespace and punctuation together
• a complete word
• part of a multibyte character
• a fragment whose behaviour depends on surrounding tokens

Suppose the parser currently allows a number beginning with 0.

The valid vocabulary might contain tokens representing:

0
0.
0.4
0.46
0}
0,

Whether each token is legal depends on the grammar and the current parser state.

It is not enough to ask whether a token equals the next valid character.

The decoder has to simulate consuming the token’s entire byte sequence.

Conceptually:

current parser state
        ↓
candidate token bytes
        ↓
simulate every parser transition
        ↓
keep the token or mask it

A token should remain available only when its complete byte sequence can be consumed without making the grammar invalid.

After one token is generated, the parser state changes.

The valid vocabulary set must therefore be computed again.

This makes a real constraint state-dependent.

A static digit ban uses the same mask at every generation step.

A JSON grammar needs a different mask after:

{

than after:

{"answer":

or:

{"answer":"72"

The legal continuation depends on where the parser currently is.

Why checking every token naively is expensive

A modern tokenizer may contain tens or hundreds of thousands of tokens.

Testing every token against the parser at every generation step would add substantial overhead.

This is where techniques such as token tries and cached parser transitions become useful.

A token trie groups vocabulary tokens by shared byte prefixes.

Instead of independently replaying every token through the parser, the decoder can traverse shared prefixes once and prune entire branches as soon as a prefix becomes invalid.

The high-level loop becomes:

1. Read the current parser state.
2. Traverse possible token-byte prefixes.
3. Prune branches that violate the grammar.
4. Collect token IDs ending on viable branches.
5. Mask the rest of the vocabulary.
6. Generate one token.
7. Advance the parser state.
8. Repeat until an accepting state is reached.

The implementation becomes more complicated for recursive structures, strings, escaping, numbers, UTF-8, and ambiguous parser states.

But the central mapping remains:

Parser state → legal token IDs

What this experiment does not prove

There are several limitations in the current setup.

The dataset is tiny

Three examples are enough for a smoke test, not for comparing decoding strategies.

The JSON extractor is deliberately strict and naive

It takes the content between the first opening brace and the final closing brace.

Multiple valid objects therefore become one invalid candidate.

A stronger evaluation should report separate metrics for:

• whole-response validity
• first-object recoverability
• schema validity
• semantic correctness

The free-form extractor is fragile

It selects the final number in the output.

Extra numeric text after the answer can produce an incorrect score.

The model was not prompted through its chat template

The raw prompt was passed directly to an instruction-tuned model.

Comparing this with the tokenizer’s intended chat template is necessary.

The token mask is not byte-accurate

Individually decoding each vocabulary token is useful for inspection.

A production implementation should reason carefully about the underlying token bytes and UTF-8 behaviour.

The constraint is static

The digit processor demonstrates logit masking, not parser-driven constrained decoding.

These limitations are not side notes.

They define what conclusions can and cannot be drawn from the experiment.

The next implementation step

The next useful milestone is a small state-dependent decoder for a restricted output language, for example:

{"answer":"<digits>"}

The implementation should:

• define the language as an automaton or parser
• map tokenizer IDs to byte sequences
• track the current parser state
• simulate candidate token bytes
• allow only tokens that preserve validity
• stop only in an accepting state
• log mask size and latency at every step

Once that works, it becomes possible to compare:

• prompting only
• parsing and repair
• regex or finite-state constraints
• JSON-schema constraints
• different token-filtering strategies

The comparison should keep the model, dataset, prompt content, and semantic scoring as controlled as possible.

The useful distinction

“Return only valid JSON” is still a useful instruction.

It can reduce failures, improve readability, and make the model’s intended output clearer.

But it is not a decoding constraint.

Prompting asks the model to follow a structure.

Validation checks whether it did.

Constrained decoding removes continuations that violate the structure.

These approaches can complement each other, but they do not provide the same guarantee.

The code, raw outputs, and notes for the experiment are available here:

Constrained Decoding Lab