Parsing JSON is the easiest part of working with a generative model, yet most pipelines stop right there and declare victory. For 48 hours I ran a scheduled job that asked a free model to squeeze messy input into a three-key JSON envelope, and the parser passed while the data lied. The guard script I ended up with caught four distinct failure classes, and the last one slipped past every syntax and schema check. How many of your own model outputs are valid, well-shaped, and completely wrong?
The setup
The experiment needed two things: a free model to generate the JSON and a free server that could stay reachable long enough to embarrass me. I used MonkeyCode's free model access and its free server option for the run, mostly because both removed cost as an excuse to stop early. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest of the stack was a boring Python script, a SQLite log, and cron.
Every few minutes the job generated one synthetic event, sent it through the model, and recorded the outcome into a log that I checked like a worried parent. The expected answer was known in advance, so I could classify every miss instead of guessing whether it mattered. That classification is the whole point of the exercise.
What I tried first
The first version of the guard did what most of my previous code did: parse the response and retry on failure. It used json.loads with a one-line strip for the markdown fences that models love to add, and it worked exactly as well as you would expect.
import json
import logging
def parse_response(raw: str):
raw = raw.strip()
if raw.startswith("```
"):
raw = raw.strip("`")
if raw.startswith("json"):
raw = raw[4:].lstrip()
try:
return json.loads(raw)
except json.JSONDecodeError:
logging.warning("parse_failure: %s", raw[:80])
return None
```
The parse bucket caught malformed answers, but it said nothing about whether the content made sense. A response with wrong key names, a string where a number belonged, or a completely invented field sailed through with a green light. That felt productive, right up until I started reading the logs.
## The schema that changed the game
I added a tiny validator with no dependencies, and it immediately began finding the failures the parser had been swallowing. The function below checks that every required key exists, that its type matches, and that no extra keys snuck in. I also made the prompt name the exact keys and include one compact example, which was the cheapest improvement in the whole experiment.
``{% endraw %}{% raw %}`python
def validate_envelope(data, required):
errors = []
for key, expected in required.items():
if key not in data:
errors.append(f"missing_key:{key}")
elif not isinstance(data[key], expected):
errors.append(f"wrong_type:{key}")
extra = set(data) - set(required)
if extra:
errors.append("extra_keys:" + ",".join(sorted(extra)))
return errors
```
The schema bucket quickly became the most informative part of the log. Fenced JSON was a distraction, truncation was loud, but schema violations were the quiet ones that actually changed behavior downstream. Have you ever debugged a bug that passed every single automated check you could think of?
## The failure the schema missed
About halfway through the run I noticed a pattern that neither the parser nor the validator could see. A field called `status` kept repeating its previous value even when the input event had clearly changed, and the JSON was perfectly valid every single time. The cause was my own prompt construction: I was appending the last assistant output to the context, and the model was copying from it instead of reasoning fresh.
That is the stale-value failure, and it is the scariest one because nothing in the response announces it. I built a small detector that compares the current envelope against the previous one for fields that should change when the input changes.
``{% endraw %}{% raw %}`python
def detect_stale(current, previous, should_change):
stale = []
for key in should_change:
if previous and current.get(key) == previous.get(key):
stale.append(f"stale_value:{key}")
return stale
```
The fix was embarrassingly simple: rebuild the prompt from the original instructions and the current event only, and never let a previous answer back into the context. Once I did that, stale values disappeared from the log, and the detector stayed behind as a permanent tripwire. In short, the model was not the liar; I had handed it the lie.
## What broke in 48 hours
Can you guess which bucket produced the most silent damage? Here is the taxonomy that emerged from the log, in the order they annoyed me:
- Truncated answers appeared whenever I asked for a larger envelope, and the fix was compact output plus exactly one retry.
- Markdown fences showed up even when I explicitly said no markdown, and defensive stripping worked but deserved its own log bucket.
- Renamed keys and extra commentary keys were invisible to the parser, and the schema validator caught them while the prompt naming helped but never eliminated them.
- Stale values copied from prior context passed every check, and only the detector caught them while prompt hygiene removed the root cause.
The important detail is that each fix produced a new failure class somewhere else in the file. That is normal, and logging categories is the only way to see the pattern instead of the panic.
## What I'd repeat
- Validate the schema, not just the syntax. Parse success is a low bar, and a green checkmark is not evidence of usefulness.
- Keep every call stateless. Fresh context, one event, one answer, and previous outputs belong in the log, not in the prompt.
- Log failure categories instead of pass or fail. The categories are the report, and the pass rate without them is a confidence trick.
- Keep the requested envelope small. Smaller payloads truncated less frequently, and every truncation was an expensive silent retry.
- Leave a human review step for anything that cannot be retried safely. The guard is a tripwire, not a proofreader.
## Who shouldn't copy this setup
A schema guard assumes somebody can retry or review, so it is the wrong tool for a fully autonomous pipeline that acts on every answer. Anyone sending model output straight into billing, permissions, or regulated decisions should treat this script as a debugging aid rather than a safety guarantee. My run covered one free server, one small envelope, and one test account, so treat the failure mix as a sample, not a benchmark.
The boring lesson is that valid JSON is table stakes, not truth. The interesting lesson is that models will trust whatever you place in context, including their own previous answers, so the context is part of the contract too. If you run something like this on your own free stack for a weekend, I would genuinely like to know which failure bucket wins on your machine. Mine had a way of embarrassing me, and it was not the one I bet on first.
Top comments (0)