48-Hour Field Notes: A Free Model Swap Broke My Parser, So I Built a Schema Gate

Forty-eight hours ago I pointed a nightly extraction job at a different model and changed nothing else about it. The job exited 0, wrote rows into the table, and nobody noticed until a downstream chart grew an "unknown" bucket the size of a small country. That is the whole story: the model did not crash, my parser did not crash, and my dashboard did not crash — the contract did.

If you have ever swapped a model, a provider, or even a prompt template and trusted an exit code, this post is for you. Below are my field notes: what I tried, what broke, and the one piece of code I would copy into the next job before touching anything else.

The setup: one job, two moving parts, no contract

The pipeline was boring on purpose, and boring pipelines are exactly where silent breakage hides.

• Step 1 — pull a batch of text records and ask a model for a structured summary of each one.
• Step 2 — parse each response and upsert it into a table keyed by record ID.
• Step 3 — write a run summary so a human can see what happened without opening the database.

I ran this against MonkeyCode's free model access, on its free server option, because the whole point of the weekend was to find out whether the workflow was worth building properly. Cheap access is not a benchmark result, and it is not a promise about throughput — it is permission to run the experiment. Treat it that way and you will not be disappointed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Hour 0–6: the swap looked boring, and that was the trap

My parser used defensive defaults everywhere, and I had congratulated myself on that for months. payload.get("summary", "") never raises, str(value) never raises, and int(value or 0) never raises. Three lines of politeness that turned a category error into a plausible-looking row.

The old output returned null for an absent summary; the new one returned "". Both are falsy, both are valid JSON, and only one of them meant "the model had nothing to say." My upsert wrote both into the same column, and every downstream WHERE summary IS NULL check stopped counting.

Is that a model bug? No. It is my missing contract, and it had been missing since the first day.

Hour 9: the first real symptom was a dashboard, not an exception

Nothing in the logs said "error," because nothing was an error in the language my code spoke. The signal came from a chart in a completely different service that grouped rows by a derived category.

• Rows with a real summary landed in named categories.
• Rows with an empty string landed in "unknown."
• Rows with null had been landing in "unknown" too, but rarely, so nobody watched it.

One field-level type change moved a whole batch into a bucket that everyone had learned to ignore. That is how contract drift works: it never announces itself, it just relabels your data.

The artifact: a gate that fails loudly, and a drift report that fails visibly

I wanted two different kinds of failure, because they answer different questions. The gate answers "is this record acceptable right now?" and the drift report answers "did the shape of the output change since last run?"

1. The schema gate

Every record passes through the same validator before it can reach the database. No validator, no write — the two commands are joined with &&, not a warning in a log nobody reads.

# gate.py — validate every record against one shared contract before it touches the DB
from __future__ import annotations
import json
import sys
from pathlib import Path

import jsonschema

SCHEMA = json.loads(Path("schema.json").read_text())
VALIDATOR = jsonschema.Draft202012Validator(SCHEMA)


def validate(record: dict) -> list[str]:
    errors = []
    for err in VALIDATOR.iter_errors(record):
        loc = "/".join(str(p) for p in err.absolute_path) or "<root>"
        errors.append(f"{loc}: {err.message}")
    return errors


def gate(path: str) -> int:
    bad = 0
    lines = Path(path).read_text().splitlines()
    for i, line in enumerate(lines, 1):
        try:
            record = json.loads(line)
        except json.JSONDecodeError as exc:
            bad += 1
            print(f"line {i}: unparseable JSON: {exc}", file=sys.stderr)
            continue
        errs = validate(record)
        if errs:
            bad += 1
            print(f"line {i}: {'; '.join(errs)}", file=sys.stderr)
    print(f"{len(lines) - bad}/{len(lines)} records passed", file=sys.stderr)
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(gate(sys.argv[1]))

The schema is where the empty-string-vs-null decision actually lives, so write it down instead of leaving it to whichever defensive default you typed first.

{
  "type": "object",
  "required": ["id", "summary", "category"],
  "additionalProperties": false,
  "properties": {
    "id": { "type": "string", "minLength": 1 },
    "summary": { "type": "string", "minLength": 1 },
    "category": { "type": "string", "enum": ["billing", "bug", "feature"] }
  }
}

Notice what I gave up: summary may no longer be null. If the model has nothing, it must omit the field or fail, and "fail" is a decision a human can see. additionalProperties: false is the other half — a new field from a new model is a schema change, not a free surprise in your table.

2. The drift report

Schemas catch violations, but they do not catch a field that is still legal and suddenly always empty. So I added a cheap field-type histogram and diffed two runs.

# drift.py — compare the shape of two output files, field by field
from collections import Counter, defaultdict
from pathlib import Path
import json


def field_types(path: str) -> dict[str, Counter]:
    hist: dict[str, Counter] = defaultdict(Counter)
    for line in Path(path).read_text().splitlines():
        for key, value in json.loads(line).items():
            hist[key][type(value).__name__] += 1
    return hist


if __name__ == "__main__":
    import sys
    before, after = field_types(sys.argv[1]), field_types(sys.argv[2])
    for key in sorted(set(before) | set(after)):
        b, a = before.get(key, Counter()), after.get(key, Counter())
        if b != a:
            print(f"{key}: {dict(b)} -> {dict(a)}")

Run it on yesterday's file and today's before you load anything, and it prints exactly the sentence I wanted at hour 9: summary: {'NoneType': 12, 'str': 480} -> {'str': 492}.

Hour 24: why the gate belonged on the free server, not on my laptop

The whole point of running the job on a remote box is that it runs when my laptop is closed, so validation has to live there too.

# produce first, validate second, load only on success
python extract.py --out out.jsonl \
  && python gate.py out.jsonl \
  && python drift.py last_good.jsonl out.jsonl \
  && python load.py out.jsonl

Two habits made the 48 hours survivable. First, write JSONL and one record per line, so a single bad response is one line number instead of an unparseable blob. Second, log the gate's stderr into the same file as the run summary, so the failure and the context sit next to each other.

mkdir -p logs
python extract.py --out out.jsonl 2>&1 | tee -a logs/run.log

Hour 30: what still broke anyway

• Quota and cold-start behavior are unknown until you measure them. I do not know your limits, and neither does a blog post; the gate does not care, but your retry policy should.
• Log volume is a real ceiling. Chatty validation output on a small remote disk fills it faster than you expect, so rotate logs before you need to.
• A gate that blocks the load blocks the whole night. That is the correct tradeoff for a summary table and the wrong one for a payment ledger, and only you can decide which you are running.
• The drift report cannot see semantics. A field that keeps its type and changes meaning sails right through; no tool here saves you from reading ten records by hand.

A decision table for your next swap

Change you are making	Schema gate	Drift report	Human reads 10 records
Prompt wording tweak	Required	Required	Yes
Model or provider swap	Required	Required	Yes
Schema change you authored	Required	Required	Yes
New input source	Required	Required	Yes

Every row says yes. That is not laziness; it is the observation that the cheap checks all cost minutes and the expensive check costs one coffee.

Who should not use this approach

Skip the gate-and-drift workflow if you are optimizing for p99 latency inside a request path, because two extra file passes are simply not free. Skip it if your volume makes a full-corpus validation pass impractical, and sample instead. Skip it if your data is regulated and a failed batch has legal weight — you need review, not a script. And skip it if you have no golden records at all, because a gate without a corpus just re-encodes today's assumptions.

What I would repeat in the next 48 hours

1. Write the contract down before the first model call, not after the first bad chart.
2. Decide explicitly whether absence is null, "", or a missing key, and encode that decision in the schema.
3. Keep the gate and the loader in the same && chain so a broken batch cannot half-land.
4. Snapshot one known-good output file and diff every run against it.
5. Read ten records by hand at the end, because tools check shape and humans check meaning.

Forty-eight hours is enough time to learn that a free model and a free server are not the hard part — the contract is. If you want to try the same weekend experiment, start by copying gate.py and writing your schema before you write another prompt; the free model access and free server option are there to make the experiment cheap, not to make the decision for you.