Mayur Agnani

Posted on Oct 24 Originally published at theaicommit.com

Why Forced JSON Output Is Always Valid and Sometimes Wrong

#ai #promptengineering #agents #testing

Engineer's view. This is a required dropdown with no "Not applicable" option. Validation always passes, so the data looks clean. But users who had no right answer picked the first item, and your reports now count them. A JSON schema enforced token by token works the same way.

TL;DR. Forced JSON mode removes the tokens your schema forbids and rescales the rest, so the output always parses. The model still chooses among legal tokens. When it wanted something illegal, it guesses, and the guess looks as sure as a real answer.

Explain Like I'm 5

A teacher hands out a quiz where every question has three boxes. You must tick one. There is no box for "I don't know." So when you don't know, you tick a box anyway. The teacher sees a neat page with every box ticked. The teacher cannot tell which ticks you knew and which you guessed. The fix is easy: add a fourth box that says "I don't know."

The Problem

You have shipped this bug before, in a system with no AI in it.

A signup form asks for a country. The field is required, and the dropdown has no "Other" option. Users from a missing country still have to submit. So they pick the first item in the list. Months later, an analyst asks why so many users live in Afghanistan. The validation never failed once. It only made sure every answer was a legal one.

Structured output has the same bug. You give the model a JSON schema, and the engine guarantees the response matches it. OpenAI, Gemini, vLLM, SGLang and llama.cpp all offer this. Teams switch it on to stop parse errors, and the parse errors stop.

Then quieter failures start. FriendliAI ran Gemma 4 31B against a complex schema. The output filled with newlines until it hit the token limit, and the JSON never closed. In other pipelines, a field limited to three values gets one of the three, even for a document the model cannot read. The value looks as sure as any other.

All of these come from one rule. The engine decides which tokens are legal. The model still chooses among them, and it must choose even when it wanted to say something else.

The fix: give the model a legal way to say what it means, bound what it can repeat, and measure how much it was forced.

The Fix: Give the Model a Legal Way Out

How does the engine force valid JSON?

A model writes one token at a time, a word or a piece of one. At each step it gives every token in its vocabulary a probability. Constrained decoding adds one step. The engine checks the grammar built from your schema, deletes every token that would break it, and rescales the rest to sum to one. Then it samples as usual.

So what happens when the model wanted an illegal token?

Its wish is deleted, and the leftovers are scaled up. Take a field that must be high, medium or low. On a clear invoice, those three hold 95% of the model's probability. On a blurry scan, they hold 15%, because the model wanted to write "unknown." After rescaling, the blurry scan still gets high 40% of the time. Both outputs look the same.

Why does output sometimes fill with whitespace?

JSON allows spaces and newlines between fields, so the grammar never deletes them. A little leftover whitespace cannot do this alone. Even at a 99% share each step, filling 2,048 tokens has odds of about one in a billion. The loop needs the model to be nearly certain at every step. That happens when repetition feeds itself: each newline makes the next more likely.

Does the order of fields matter?

The engine writes fields in the order the schema lists them, left to right. A confidence value written before a reasoning field cannot depend on that reasoning.

What This Means for You

When this matters. You use JSON mode, a response schema or strict tool calls, and your code acts on a field's value. Agents do this constantly. A tool call is a schema the model must fill in. A routing step reads one enum. An extraction job writes rows to a database.

How it affects you. Parse-error rates drop to zero, so dashboards built on them look healthy. The failures move into the values. A forced pick is a wrong row nobody flags. A whitespace loop is a timeout, or a bill for 2,048 tokens of nothing.

This is the same bug as the country dropdown. Validation guarantees that every answer is a legal one. It says nothing about whether the model had an answer.

What to do about it. Today, open the schemas your code acts on. For every enum, ask what the model should say when it does not know. If there is no legal answer for that, add one. Next, put any reasoning field before the fields that depend on it. Whether that improves accuracy depends on the task and the model. It is cheap to test, so measure it on yours rather than assume. Then cap output length and treat a truncated response as a failure. If you host the model, switch off free whitespace and start logging how much probability the grammar removed.

Implementing It

The change.

Four roles touch this: whoever writes the schema, whoever writes the prompt, whoever runs the server, and whoever reads the response.

Role 1, the schema. Give every field your code acts on a legal "I don't know." Put reasoning first. Bound free strings where your provider's schema subset allows maxLength:

from typing import Literal
from pydantic import BaseModel, Field

class Invoice(BaseModel):
    reasoning: str = Field(max_length=600)        # written first, read by the fields below
    vendor: str | None = Field(max_length=120)    # None is the legal "not on the page"
    total: float | None
    confidence: Literal["high", "medium", "low", "unknown"]

Lint every schema in CI with check_schema() from code_example.py. It flags an enum with no way out, an unbounded string, and an answer written before its reasoning.

Role 2, the prompt. Include one filled-in example of the output. FriendliAI's whitespace loop went away when the prompt carried a concrete example. The mask only removes options; the example puts probability on the right ones. Use a real record, not placeholders. An example full of "string" teaches the model to write "string". Show an unknown case too, so the model sees that the way out is allowed.

Role 3, the server, if you host the model. vLLM can forbid whitespace between JSON fields, which removes the loop's open door:

vllm serve <model> \
  --structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}'

It works only with the xgrammar and guidance backends. With transformers, wrap your grammar processor to record how much probability it removed:

import torch
from transformers import LogitsProcessor

class KeptMass(LogitsProcessor):
    def __init__(self, grammar):                  # any processor that sets illegal tokens to -inf
        self.grammar, self.kept = grammar, []
    def __call__(self, input_ids, scores):
        before = torch.softmax(scores.float(), dim=-1)
        out = self.grammar(input_ids, scores.clone())
        self.kept.append((before * torch.isfinite(out)).sum(-1))
        return out

Role 4, the caller. A cut-off response is a failure, not something to repair:

resp = client.chat.completions.create(model=MODEL, messages=msgs,
    response_format={"type": "json_schema", "json_schema": SCHEMA}, max_tokens=800)
choice = resp.choices[0]
if choice.finish_reason == "length":
    raise TruncatedOutput(resp.id)                # retry or route to review; never parse
row = Invoice.model_validate_json(choice.message.content)

Gemini reports the same case as MAX_TOKENS in finishReason. Count truncations per schema, because one bad schema hides inside a healthy average.

How you know it worked. Three numbers per field, on a week of traffic. The truncation rate should be near zero. The unknown rate should be above zero, and roughly match how often a human finds no answer in a sample. If it is exactly zero, the model is still being forced somewhere. If you host, look at kept mass on value tokens only. Structure tokens like { lose mass on every request for harmless reasons. A value position that kept under half the model's probability is a guess worth reviewing.

When Forcing JSON Is the Wrong Tool

When the model needs room to think. A schema that starts on token one leaves no space for free reasoning. For hard tasks, let the model answer in plain text first. Then make a second, cheap call that formats the answer into JSON.

When your schema is huge or changes per request. The engine compiles each new grammar before the first token. A cached schema costs little, but a new one per request adds delay to every first token. Grammar engines cache compiled schemas for this reason.

When you use a small model on a hard schema. Forcing makes the output parse. It cannot make a weak model know the answer. Nested optional fields are where small models fill values with whatever is left.

When the provider's schema subset cannot express your rules. Some keywords are silently unsupported or rejected. Check the response against the full schema in your own code anyway.

Before you adopt it, ask:

What should each field say when the model does not know?
Which fields does my code act on, and is anything written before its reasoning?
How would I notice a value the model was forced into?

Glossary

token — a word or piece of a word; the model writes one at a time.
Structured output — an API mode that guarantees the response matches a JSON schema you supply.
Constrained decoding — deleting the tokens a grammar forbids at each step, then sampling from what is left.
enum — a schema field limited to a fixed list of allowed values.
kept mass — the share of the model's probability that survives the grammar's deletion at one position.

DEV Community