The Model Returned Valid JSON. My Pipeline Trusted It. That Was the Bug.

You check the HTTP status, you parse the body, and you assume the shape is right because the syntax is valid. How often does a model return well-formed JSON that is still structurally wrong? Over 48 hours on a free server with MonkeyCode's free model access, I watched my pipeline accept three responses that parsed perfectly and were still wrong in three different ways. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This is a field note, not a victory lap. I tried a deliberately simple approach, it failed in interesting places, and I ended up with a small validator that I would genuinely use again. Here is what broke, what the fix looked like, and where the fix stops working.

The setup that looked fine

The pipeline was deliberately boring: a queue held jobs, a worker called the model, a parser handed the result to a consumer, and the consumer updated a database. The parser was the part I trusted the most, because it was three lines long.

import json

def parse_model_output(raw: str) -> dict:
    return json.loads(raw)

That function is the whole problem in miniature. json.loads only guarantees that the text is valid JSON, and valid JSON tells you nothing about whether the keys exist, whether the types match, or whether an empty list means "no data" or "the model gave up."

The field log

Three failures happened in 48 hours, and none of them raised a syntax error.

Entry one: the missing key

The first job asked for a title, a summary, and a body. The model returned a title and a body, skipped the summary, and my consumer crashed on result["summary"] with a KeyError. The JSON was valid, the HTTP call was fine, and the model had simply decided that a short input did not deserve a summary.

Entry two: the type drift

The second failure was sneakier because nothing crashed. I asked for tags as a list of strings, and the model returned a single comma-separated string instead. My code iterated over the value, and instead of getting three tags I got individual characters: p, y, t, h, o, n.

Entry three: the silent empty list

The third failure was the worst one, because it produced no error at all. The model returned an empty list where I expected an object with defaults, and the consumer interpreted that as "nothing to do" and skipped the job. The job vanished from the queue, the database never updated, and nobody noticed until I checked the logs.

The validator I wrote

The fix was a small validation layer that runs after parsing and before business logic. It checks every expected field against a declared type, fills in defaults for optional fields, and writes every rejection to a JSONL drift log.

import json
from datetime import datetime, timezone

EXPECTED = {
    "title": {"type": str, "required": True, "default": None},
    "summary": {"type": str, "required": False, "default": ""},
    "tags": {"type": list, "required": False, "default": []},
    "meta": {"type": dict, "required": False, "default": {}},
}

def validate_model_output(raw: str, log_path: str = "drift.jsonl") -> dict:
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        _log_drift(log_path, raw, error=f"invalid_json: {exc}")
        raise

    cleaned = {}
    for field, spec in EXPECTED.items():
        value = data.get(field, spec["default"])
        if value is None and spec["required"]:
            _log_drift(log_path, raw, error=f"missing_required: {field}")
            raise ValueError(f"missing required field: {field}")
        if not isinstance(value, spec["type"]):
            _log_drift(log_path, raw, error=f"type_mismatch: {field}")
            value = spec["default"]
        cleaned[field] = value
    return cleaned

def _log_drift(log_path: str, raw: str, error: str) -> None:
    entry = {
        "ts": datetime.now(timezone.utc).isoformat(),
        "error": error,
        "raw": raw[:500],
    }
    with open(log_path, "a") as fh:
        fh.write(json.dumps(entry) + "\n")

The validator is deliberately boring. It uses only the standard library, it fails loudly for required fields, and it degrades quietly for optional ones. The drift log is the part I would defend in a code review, because it turns a vague feeling that the model changed into a list of concrete failures you can count.

What I'd repeat

If I ran those 48 hours again, I would keep four decisions and drop the rest.

1. Validate before business logic, never after. The consumer never saw a bad shape again once the validator stood in front of it.
2. Log the raw payload on every rejection. Truncated to 500 characters, it was enough to diagnose each failure without filling the disk.
3. Be strict in development and lenient in production. Required fields raised during testing, while optional fields fell back to defaults in production.
4. Count the drift. The number of type mismatches per hour told me the model's behavior had shifted before any user complained.

Where this approach stops working

The validator catches shape errors, and that is the entire scope of its usefulness. It cannot catch semantic errors, and on day three the model returned a perfectly valid summary about the wrong article. No schema checker will ever catch that, because the structure was correct and the meaning was not.

I would also not recommend this approach if your provider supports structured output or JSON Schema mode. If you can constrain the model at generation time, do that, and treat a hand-rolled validator as a safety net rather than a cage. If your team already uses Pydantic, use Pydantic instead of my standard-library version, because the drift log is the only part worth copying.

Who should skip this entirely

You should skip this pattern if your model output feeds directly into a database with strict constraints, because the database should reject bad shapes before your application code ever sees them. You should also skip it if you have a formal API contract, because an OpenAPI schema plus a code generator gives you the same protection with less custom code. And you should definitely skip it if you expect validation to improve answer quality, because it will not.

The honest takeaway

The free model access and the free server were not the source of these failures, and that is exactly the point. The failures came from my assumption that valid JSON meant correct data, and that assumption would have bitten me on any hosting setup. The validator was small enough to review in one sitting, and it caught three silent bugs in the first two days.

If you have watched a model return perfect JSON that was still wrong, I would genuinely like to know what caught it in your pipeline. Drop a comment with your favorite failure, because the shape errors are easy to fix and the semantic ones are the ones worth sharing.