Constraining a model's output with a JSON schema feels like the moment the LLM stops being a text generator and becomes a function. You declare the fields, the server enforces the grammar, and what comes back parses every time. After months of regex-scraping JSON out of prose, it's a genuine relief.
Then you ship it, and you find out what a schema actually promises. It promises a shape. Everything you care about is content, and content is on you.
Here's the list of things that came back schema-valid and wrong, in the order I found them.
The array was the right shape and the wrong length
The model processes several items per call and returns one object per item. The schema says: an array of objects, each with these required fields.
Sometimes it returned one object for three inputs. Perfectly valid — an array of one is an array. The schema constrains what an element looks like; it has no idea how many elements you were expecting, because it never saw your input.
The fix is boring and unavoidable: reconcile the response against the request by id. Send ids in, require ids back, diff the sets, and re-ask individually for whatever went missing. What made this worth catching rather than shrugging at is what silence would have cost — a dropped item doesn't come back around on the next pass, because the next pass selects a fresh top slice of new work. "Probably fine, it'll get picked up later" is only true if something actually picks it up later.
The same reconciliation gives you a free failure mode for the whole batch: if the entire call times out, retry the items one at a time instead of losing all of them to one pathological input.
The field was a string, and it was the wrong language
Each item comes back with a translations object: one key per target language, each holding a title and summary.
A small model mixes those slots up. Under the key for one language, you get fluent text in a different one — sometimes the source language untranslated, sometimes a language nobody asked for. The schema is entirely satisfied: the required key is present, its value is an object, the fields are strings. All true. All useless.
So you write the check the schema can't: does this text use the writing system its language key implies? Count characters by script, compare to what's expected for that key, reject the mismatch.
This works, and it's the easy half.
The hard half: the wrong language glued onto the right one
The check above compares shares — the language's own script should dominate. That catches wholesale substitution and misses the failure that actually reached my users.
The model sometimes finishes a translation correctly and then keeps going, appending a second translation into the same field. What ships looks like a correct headline with a foreign clause welded to its tail. By share, the correct text wins comfortably — the tail is a fraction of the string. The dominance check passed it every time, and a batch of headlines went out with a foreign fragment stapled on the end.
The fix isn't a better ratio. It's noticing that ratios are the wrong tool here:
A threshold that looks reasonable will catch the common case and wave through the dangerous one. Any share-based test assumes the two things being weighed are commensurable. Correct output and corrupt output aren't. Two characters of a script that has no business appearing is not a small problem in proportion to the string — it's the whole failure, at any size.
So the rule became presence, not proportion: scripts that cannot legitimately appear in this language are rejected on sight, at a count of one.
The nuance that makes it survivable is knowing where not to apply zero tolerance. Latin characters stay legal everywhere, because product names, acronyms and companies are legitimate in any language, and banning them would reject far more good translations than it saves. Zero tolerance is only affordable where you can name the reason no legitimate case exists. Otherwise you're just building a stricter bug.
The field was populated, and the contents were filler
The schema language tempts you to push semantics into it. My keyword list looked like a great candidate: require at least three entries, and the model has to produce three.
It does. It pads. Ask a summarizer for a minimum of three keywords per science article and the shortfall gets filled with study, research, science — perfectly valid strings, perfectly useless index terms, and worse than an empty list because now they're indistinguishable from real ones.
I dropped the minimum. The schema's job is to guarantee the field exists and holds strings; how many there should be is a question about meaning, which belongs to the prompt. An upper bound stayed, and only because it matches a truncation the code performs anyway — no point paying to generate items you're about to throw away.
A schema can force a field to exist. It cannot force it to be worth having. Every constraint you add gets satisfied by the cheapest available route, and for a minimum count the cheapest route is filler.
What to do with a value you've rejected
Having caught a bad translation, the instinct is to keep it — something is better than nothing, surely.
It isn't. I drop it and leave the slot empty, which gets you two things. The reader sees an honest fallback to the original language instead of confident nonsense in theirs. And an empty slot is a work item: a later pass sees a missing translation and regenerates it, whereas a filled slot containing garbage looks done forever.
Prefer an absent field to a wrong one. Absence is a state your system can act on. Wrongness looks exactly like success to every piece of code downstream.
Two things about the plumbing
Not every server accepts your schema. Older runtimes reject the schema object outright with a 400. Retrying without it is right, but retry and remember — flip a flag for the process, so you're not paying two round trips for every call from then on. Degrade to prompt-only and log it loudly.
Parse defensively even with a grammar. Whether the wrapper is an array, a sequence of bare objects, or a single object varies with the model and whether the schema survived. Rather than requiring one canonical shape, scan for top-level {...} and decode each one incrementally. It's a few lines, it's shape-agnostic, and it turns "the model wrapped it differently today" from an outage into a non-event.
What I'd take away from it
Structured output moved a whole class of problems — malformed JSON, missing keys, prose wrapped around the payload — from my code into the runtime, and that's a real win I wouldn't give back.
What it didn't do is make the output correct. It relocated the boundary. The grammar handles syntax; everything about meaning is still yours, and the failures that reach production are all on the meaning side, because syntax errors are loud and semantic ones are polite. Wrong language in the right slot, a foreign clause on the tail, three words of filler in a keyword list — every one of those was valid against the schema and would have been caught by five minutes of looking at what actually shipped.
So budget for the validator you write yourself, and expect it to be about as much code as the schema. Mine runs on a science-and-tech digest I maintain, where the model translates into several languages at once and every one of them is a slot it can put the wrong thing into. The schema stopped me from ever parsing a broken response. It never once stopped me from publishing a wrong one.
Top comments (5)
Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.
This is an important distinction between syntactic validity and semantic correctness. JSON Schema gives you structural guarantees, but production reliability requires a second validation boundary.
I would model the pipeline as contract validation followed by semantic validation and reconciliation. Assign every input a stable identifier, enforce cardinality and language constraints, then run script detection, terminology checks, confidence scoring, and domain specific assertions before persistence.
For multilingual systems, I would also maintain language specific tokenization and Unicode script policies rather than relying purely on character ratios. A rejected translation should become an explicit retryable state, not silently resemble successful data.
The deeper principle is that LLM output should be treated as untrusted data. Schema validation makes parsing deterministic, while semantic validation makes behavior observable and recoverable.
Excellent writeup. This is exactly where robust LLM engineering starts becoming distributed systems engineering.
Thanks for reading it that closely — I didn't expect this one to find an audience.
You're right that it's a second validation boundary, and most of what you list is what the system converged on: ids in and out, set diff, per-item re-ask, and a rejected translation that becomes an empty slot rather than a filled-looking one. Two places where practice pushed back on the design.
Script policy over ratios. Agreed on Unicode script properties — and that's where my implementation is weakest rather than strongest. I count codepoints in hand-rolled ranges and bucket Han together with kana, which is fine while the eagerly-generated languages sit far apart, and stops being fine the moment two same-script languages sit next to each other. The honest limit of any script check is that it only sees cross-script errors: English pasted into the Spanish slot is invisible to it, and no Unicode policy fixes that — it needs real language ID. I'd rather ship a check that names what it can't see than one that looks complete.
Confidence scoring is where I stopped. The failures were categorical — wrong script, wrong count — and a deterministic rule catches those at no marginal cost per item. A scoring pass means a second inference per item, i.e. doubling the expensive half of the pipeline to re-detect what a few dozen lines already detect. I'd reach for it when failures stop being categorical and start being gradual: a translation that's subtly bad is exactly the class a rule can't express.
The title probably reads like Chinese is the accident here. It's a resident: both a language I ingest sources in and a supported target language. What varies is when a translation is produced — a couple of languages are generated for everything, the rest on demand: a reader picks the language, and the translation happens. Generating every language for every article up front is mostly work nobody reads. Which makes your retryable-state point cheap to honour: a rejected slot is simply missing, and missing is already the trigger the on-demand path listens for.
Thanks again — "this is where LLM engineering becomes distributed systems engineering" is the right framing.
I am glad to hear that my reply was of some help.
I would like to get to know you better. Would you please contact me? t_g_@kanelim1997
Thanks — glad it was useful. I keep technical conversation on the platform the thread is on: whatever you'd add is worth more to whoever reads the article later than it is to the two of us in private.
One thing worth saying plainly, and it's for your benefit rather than mine: I wouldn't post a messaging handle in a public comment. Threads under active tags get scraped, and what a handle in the open attracts is not the audience you were hoping for — impersonation is the cheap end of it. Contacts belong on your profile, where you decide who sees them and you can take them back.
Thx your advice.
but now I need your help.
now I am finding some junior devs.
but it's difficult.
so i wanna you help me chatting via telegram.