Three Unrelated Pipelines, One Root Cause: Building a CircleCI Gate That Catches Broken LLM JSON Before It Ships

Yesterday, three completely unrelated pipelines in my system failed the same way within 48 hours.

One generates blog articles. One generates local-business outreach demos. One generates client proposals. They don't share a codebase, a prompt template, or an owner. What they share is a single line buried in each of them: json.loads(response).

All three broke with a variation of the same error: "content generated but JSON parse failed." Not a crash — worse. A silent stall. The model did its job, produced something, and the parser choked on it downstream, after the artifact was already written to disk. By the time anyone noticed, the failure was three layers removed from its cause.

If you're shipping any product where an LLM's output gets parsed as structured data — and at this point, whose isn't — this is worth thirty minutes of your CI pipeline's time, because it will happen to you too, and it will happen more than once.

Why this keeps happening

The instinct is to treat "parse the model's JSON" as a one-line implementation detail. It isn't. It's a contract between two systems that drift independently:

• The model provider ships a checkpoint update and the model gets more polite — it now wraps JSON in a friendly sentence ("Sure, here's the JSON you asked for:") that it didn't add last week.
• Someone edits a prompt template for an unrelated reason and accidentally changes whether the model treats a trailing comma as fine.
• A field that used to always be a string starts arriving as null on some fraction of calls because the model decided ambiguity was better represented that way.

None of these are your bugs, exactly. But they become your outage, because your parser has zero tolerance for any of them, and nothing tells you the contract changed until a human notices missing output days later.

The fix that doesn't scale: patch each pipeline

The first move is obvious and it's what I did initially — wrap the json.loads call, catch the exception, log it, retry once with a stricter re-prompt. Something like:

import json
import re

def parse_llm_json(raw: str) -> dict:
    # Strip markdown code fences if the model added them
    match = re.search(r"```

(?:json)?\s*(\{.*\})\s*

```", raw, re.DOTALL)
    candidate = match.group(1) if match else raw.strip()
    return json.loads(candidate)

def generate_with_repair(prompt: str, call_model, max_attempts: int = 2) -> dict:
    last_error = None
    for attempt in range(max_attempts):
        raw = call_model(prompt if attempt == 0 else
            f"{prompt}\n\nYour previous response failed to parse as JSON "
            f"({last_error}). Return ONLY valid JSON, no prose, no code fences.")
        try:
            return parse_llm_json(raw)
        except json.JSONDecodeError as e:
            last_error = str(e)
    raise ValueError(f"Failed to get valid JSON after {max_attempts} attempts: {last_error}")

This works. It also completely misses the point, because I wrote a version of this three separate times, once per pipeline, and the underlying contract violation still wasn't caught until runtime — after real API spend, after a task was already marked in-progress, after the failure had to be discovered rather than prevented.

The deeper mistake was validating after the artifact was generated and stored, instead of before the task was allowed to complete. Downstream validation finds the bug. It doesn't stop three independent teams (or three independent pipelines, if you're a solo dev) from rediscovering it separately.

The fix that scales: treat it as a contract test, run in CI

The actual fix wasn't a better try/except. It was moving the check to a layer where a bad contract gets caught before it ships, on every commit that touches a prompt template or a parsing schema — not after a customer-facing task fails.

Step one: freeze a fixture set of real model responses, both the ones that parsed fine and the malformed ones that caused the actual incidents. Keep them as JSONL, one response per line, alongside an expected outcome:

{"raw": "{\"title\": \"Post\", \"tags\": [\"a\"]}", "should_parse": true}
{"raw": "Sure, here's the JSON:\n```

json\n{\"title\": \"Post\"}\n

```", "should_parse": true}
{"raw": "{\"title\": \"Post\", \"tags\": [\"a\",]}", "should_parse": false}

Step two: a pytest suite that runs the real parser against every fixture and asserts the outcome matches:

import json
import pytest
from pipeline.parsing import parse_llm_json

def load_fixtures():
    with open("fixtures/llm_responses.jsonl") as f:
        return [json.loads(line) for line in f]

@pytest.mark.parametrize("case", load_fixtures())
def test_parser_contract(case):
    if case["should_parse"]:
        assert parse_llm_json(case["raw"])  # must not raise
    else:
        with pytest.raises(json.JSONDecodeError):
            parse_llm_json(case["raw"])

Step three, the part that actually stopped the repeat: a dedicated CircleCI job gating anything touching prompts or parsing code.

version: 2.1

jobs:
  llm-contract-test:
    docker:
      - image: cimg/python:3.12
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run:
          name: Run LLM output contract tests
          command: pytest tests/test_llm_contract.py -v

workflows:
  version: 2
  build-and-test:
    jobs:
      - llm-contract-test:
          filters:
            branches:
              only: /.*/

Every time a prompt template, a parser, or a schema changes, this job runs the full fixture set — the good responses and the bad ones I've actually seen in production — in about eight seconds. If a change makes the parser reject a response it used to accept, or accept one it used to (correctly) reject, the build fails before merge, not three pipelines and three days later.

What changed operationally

The fixture file is now append-only: every time a pipeline hits a new malformed-response shape in production, it gets added to fixtures/llm_responses.jsonl as a new should_parse: false case before the fix ships. That turns every incident into a permanent regression test instead of a one-off patch. Three months in, the fixture set has caught two prompt-template edits that would have silently broken parsing again — both caught in CI, both zero-impact in production.

The lesson wasn't "add a try/except." It was that an LLM's output shape is an interface with a version history, and the same discipline you'd apply to an external API contract — recorded fixtures, explicit pass/fail cases, a CI gate — applies here too. The parsing bug wasn't three bugs in three pipelines. It was one missing test suite.

Comments

[Heinrich Neb]

Two catches in three months on a suite that runs in eight seconds is the part most posts about CI gates can't produce, so it's worth stating plainly: your gate has been observed to fail. Most haven't, and a gate nobody has watched refuse is indistinguishable from a broken one.

One thing I'd watch in the fixture file, because it's doing two jobs that will eventually pull apart. Right now should_parse: false on the trailing-comma case records what happened and asserts what should happen. Those agree today. The day you decide tolerating a trailing comma is the correct behaviour - a defensible improvement - that fixture goes red, and the obvious fix is to flip the flag. Which silently deletes the record of the incident that put it there. The incident and the expectation want separate fields eventually: seen_in_production: true never changes, should_parse is allowed to.

The cheap question, and you already have the number: what's the ratio of should_parse: false to true in the file, and has it moved? A file that stops gaining malformed shapes means either the provider stabilised or your production detector went quiet - and from inside the repository those look exactly the same. If the count has been flat for a month, that's worth knowing which one it is.