DEV Community

Cover image for Why Forced JSON Output Is Always Valid and Sometimes Wrong
Mayur Agnani
Mayur Agnani

Posted on Edited on Originally published at theaicommit.com

Why Forced JSON Output Is Always Valid and Sometimes Wrong

A model forced into a three-value field picks one of the three even when it cannot read the input. The JSON parses, the value looks sure, and nothing tells you it was a guess.

Engineer's view. This is a required dropdown with no "Not applicable" option. Validation always passes, so the data looks clean. But users who had no right answer picked the first item, and your reports now count them. A JSON schema enforced token by token works the same way.

TL;DR. Forced JSON mode removes the tokens your schema forbids and rescales the rest, so the output always parses. The model still chooses among legal tokens. When it wanted something illegal, it guesses, and the guess looks as sure as a real answer.

Explain Like I'm 5

A teacher hands out a quiz where every question has three boxes. You must tick one. There is no box for "I don't know." So when you don't know, you tick a box anyway. The teacher sees a neat page with every box ticked. The teacher cannot tell which ticks you knew and which you guessed. The fix is easy: add a fourth box that says "I don't know."

The Problem

You have shipped this bug before, in a system with no AI in it.

A signup form asks for a country. The field is required, and the dropdown has no "Other" option. Users from a missing country still have to submit. So they pick the first item in the list. Months later, an analyst asks why so many users live in Afghanistan. The validation never failed once. It only made sure every answer was a legal one.

Structured output has the same bug. You give the model a JSON schema, and the engine guarantees the response matches it. OpenAI, Gemini, vLLM, SGLang and llama.cpp all offer this. Teams switch it on to stop parse errors, and the parse errors stop.

Then quieter failures start. FriendliAI ran Gemma 4 31B against a complex schema. The output filled with newlines until it hit the token limit, and the JSON never closed. In other pipelines, a field limited to three values gets one of the three, even for a document the model cannot read. The value looks as sure as any other.

All of these come from one rule. The engine decides which tokens are legal. The model still chooses among them, and it must choose even when it wanted to say something else.

The fix: give the model a legal way to say what it means, bound what it can repeat, and measure how much it was forced.

Read the full post on The AI Commit

Why Forced JSON Output Is Always Valid and Sometimes Wrong carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.


Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

Top comments (0)