DEV Community

Maxim
Maxim

Posted on

How I fixed silent JSON failures in my LLM pipeline (spoiler: json.loads was the trap)

#llm #testing #python #devops

I'm an engineer. I build pipelines where an LLM is not a chat toy — it's a part in a machine.

One evening I was staring at logs. Half the videos went out with broken titles. JSON looked fine. json.loads didn't crash. The data was garbage anyway.

That was the first time we fell into this hole. Not the last.

The trap

The code looks innocent:

data = json.loads(llm_response)
title = data["title"]
Enter fullscreen mode Exit fullscreen mode

Nice and short. And fragile.

An LLM does not return JSON. It returns text that looks like JSON. Like a person you asked to fill a form, and they answered: "Well, you see…" — maybe useful, maybe not.

What we actually got in production:

1. Markdown fences

The model wraps JSON in triple-backtick code fences. json.loads dies immediately. At least that failure is loud.

2. Politeness

Sure! Here is the JSON you asked for:
{"title": "Hello"}
Enter fullscreen mode Exit fullscreen mode

Dies again. The model was helpful. The pipeline wasn't.

3. Trailing commas, smart quotes

{"title": "Hello",}
Enter fullscreen mode Exit fullscreen mode

4. Truncation

{"title": "Hello", "score":
Enter fullscreen mode Exit fullscreen mode

5. The worst one — silent junk

{"ok": true}
Enter fullscreen mode Exit fullscreen mode

instead of {"title": "...", "score": 42}.

json.loads is happy. Valid JSON. Wrong shape. Garbage flows downstream. You notice an hour later. Or never.

Two failure modes, not one

Type What happens When you see it
Loud failure Exception, job dies immediately
Silent failure wrong fields / wrong types too late

try/except handles the first. The second is the one that burns you.

What we tried (and what didn't work)

  • "Just tell the model to always return clean JSON." Sometimes helps. A model is not a compiler.
  • Regex between { and }. Breaks on nested objects and braces inside strings.
  • Another pile of if-statements. Every new response shape = more ifs. Snowball.

We stopped begging the model and started inspecting it.

The fix 

LLM text
   |
   +-- strip markdown fences and "Sure! Here's..."
   +-- normalize quotes / trailing commas
   +-- parse
   +-- validate SCHEMA (types, fields, bounds)
   |
   +-- if fail -> repair loop
   |      re-prompt with the EXACT error
   |      ("score: expected int, got str")
   |
   +-- if API is dying -> circuit breaker
          stop hammering a corpse
Enter fullscreen mode Exit fullscreen mode

Same idea as QC on a factory floor: make it -> check it -> fix or scrap. No hoping.

I didn't start in software. I spent years around real production. A part goes out against a drawing, gets inspected, and scrap doesn't move down the line. An LLM is a machine too. It just lies with more confidence.

Code (no magic) 

from llm_output_guard import Schema, Field, RepairLoop

schema = Schema(
    fields={
        "title": Field(str),
        "score": Field(int, min_value=0, max_value=100),
    }
)

def call_model(err=None):
    # err = exact validation error from last attempt
    return generate(title=..., hint=err)

result = RepairLoop(schema=schema, max_attempts=3).run(call_model)
Enter fullscreen mode Exit fullscreen mode

Repair is not "roll the dice again". It is feed the model its own error. In our runs, attempt 2 fixes most type mistakes.

Separately: circuit breaker — 5 failures -> cooldown -> reset. Otherwise a clever repair loop becomes a hammer that finishes off a dying API.

Numbers from our systems (not from a blog)

  • 49 Instructor injection points in one production pipeline — all through one JSON path
  • 60+ schema contracts
  • 12 behavioral tests just for "dirty" LLM outputs
  • finish_reason=length counts as failure, not success

When you have dozens of calls, "it'll parse somehow" is not a strategy.

Steal this checklist (practical, Monday-morning)

If you only take one thing from this post — take this:

  • [ ] Never json.loads(llm_text) raw. Strip fences first.
  • [ ] Always validate shape after parse (types + required fields).
  • [ ] Treat finish_reason=length as failure, not success.
  • [ ] On schema error: re-prompt with the exact error. Don't roll dice.
  • [ ] Put a circuit breaker around the API. 5 fails -> cooldown.
  • [ ] Log why validation failed. Future-you will need it.
  • [ ] Test with ugly outputs: fences, prose, trailing commas, wrong types.
  • [ ] One shared contract for all call sites. Not 49 special cases.

Quick "before you ship" check

Ask three questions:

  1. What happens if the model returns markdown?
  2. What happens if a field has the wrong type?
  3. What happens if the API dies mid-run?

If the answer is "we crash" or "we ship garbage" — you're not done.

What we do not do

  • No sending secrets to random "JSON fixer" services
  • No treating json.loads(strip(text)) as enough
  • No hiding "almost passed" results

Takeaways

  1. LLM is not a JSON API. It's a text API. JSON is your job.
  2. Silent failures hurt more than crashes. Schema catches what try/except misses.
  3. Repair with the exact error beats regenerate-from-scratch.
  4. Circuit breaker protects you from your own stubbornness.
  5. One contract for every call beats 49 special snowflakes.

We published this as small open-source libraries so nobody rebuilds it again:

If you run LLMs in production, check your json.loads. If it has no schema and no repair — you may already be paying for it. Just haven't seen the invoice.

What's the ugliest LLM JSON you've seen in production? I'm collecting war stories.

Top comments (0)