The Incident That Never Paged: A Schema-Drift Postmortem

The ticket lands at 09:14 with the subject line "invoice totals look wrong", and nobody on your team is on fire because nothing is visibly on fire. Error rate is 0.3%, right where it sat last week, and p95 latency is flat. The dashboard is a wall of green, which is exactly why the next three hours are going to be expensive.

You eventually trace it to an extraction step where a model turns messy purchase orders into structured JSON. In roughly one case in twenty, that step now returns prose instead of JSON, and a defensive except clause you wrote eighteen months ago quietly substitutes qty=1. The invoice still generates, the pipeline still reports success, and the customer finds the wrong total before you do.

To keep this reproducible, treat the postmortem below as a synthetic incident: the failure modes are real and common, while the timeline and counts come from a replay harness you can run yourself. Swap in your own logs and the shape will hold, because this class of silent regression is spreading as fast as teams adopt model-backed features.

What actually made it quiet

Three contributing factors combined, and none of them is dramatic on its own. First, someone edited the extraction prompt in place to make output "cleaner", with no version pin and no changelog entry, so the diff never appeared in a code review. Second, the downstream parser treated missing fields as optional and substituted defaults instead of failing loudly, which converted a hard error into a soft data corruption. Third, the canary compared 5xx rate and latency only, so a change that preserved both while degrading structure was invisible by construction.

There is a fourth factor that most postmortems skip, and it is the one that costs the most. The team had unit tests for the parser, but every fixture was hand-written valid JSON, so the tests proved the parser worked on the input it was designed for. Nothing in CI ever asked a model for an answer and checked whether that answer survived the schema.

Timeline, reconstructed

At 09:14 support opens the first ticket, and by 09:26 there are four more with the same wording. At 09:40 an engineer reproduces locally and finds both outcomes on the same input, which reads as nondeterminism but is actually a prompt-version problem. At 10:05 a replay of the last 500 production documents shows a structural failure rate near 5%, against roughly 1% on the pinned prompt from the previous week. At 10:20 the prompt is reverted, and the rate falls back, which confirms causality faster than any root-cause debate could.

From 10:25 to 11:00 the team backfills affected records by re-running extraction with the pinned prompt and diffing totals. The detection gap, from first bad output to first ticket, is the number that should bother you most here: it is the interval when your system was confidently wrong. That interval is a design property, not bad luck, and it is the only thing worth fixing.

A fix that survives the next hire

Reverting the prompt is a repair, not a fix. The durable version has three parts that reinforce each other: version every prompt and model reference as data rather than as a string in a function, make the parser strict so malformed output fails loudly in staging, and gate merges on a golden set that actually calls a model.

The first part is mostly discipline, but it needs a small amount of machinery. Keep prompts in files with a content hash, record that hash on every request, and log it next to the output so a replay can reconstruct the exact conditions.

import hashlib, json
from pathlib import Path

def load_prompt(name: str) -> tuple[str, str]:
    raw = Path(f"prompts/{name}.md").read_text(encoding="utf-8")
    version = hashlib.sha256(raw.encode()).hexdigest()[:12]
    return raw, version

# attach the version to every trace you emit
print(json.dumps({"event": "extract", "prompt": "invoice/v7", "sha": version}))

The second part means deleting the optimistic default. With Pydantic you can make the contract unforgiving in a few lines, and you should also check arithmetic instead of trusting a generated total.

from decimal import Decimal
from pydantic import BaseModel, Field, ValidationError

class LineItem(BaseModel):
    description: str = Field(min_length=1)
    qty: int = Field(gt=0, le=1000)
    unit_price: Decimal = Field(gt=0)

class Invoice(BaseModel):
    items: list[LineItem] = Field(min_length=1)
    total: Decimal = Field(gt=0)

    def check_math(self) -> None:
        computed = sum(i.qty * i.unit_price for i in self.items)
        if abs(computed - self.total) > Decimal("0.01"):
            raise ValueError(f"total mismatch: {computed} != {self.total}")

The third part is the one teams postpone forever, and it is where cheap inference changes the economics. Build a small golden file where each line is an input, an expected structure, and a short note on which past incident it guards against.

{"id":"inv-014","note":"regressed after prompt v7","input":"3 boxes A4 at 12.50 each, 1 stapler at 8.00","expect":{"items":2,"total":45.50}}
{"id":"inv-031","note":"prose leak, no JSON at all","input":"customer asked to reorder last month, same items","expect":{"items":2,"total":45.50}}

Grade structure strictly and grade meaning loosely, then report both numbers so nobody can average them into a comfortable single score. A structural pass rate below your threshold blocks the merge, and a semantic spot-check of about twenty samples per run catches the cases where valid JSON describes the wrong order. The harness below isolates the model call in one adapter function, which matters because you will change providers more often than you change grading logic.

import json
from pydantic import ValidationError

def call_model(prompt: str, system: str) -> str:
    # Single seam. Point this at whichever endpoint your account provides;
    # MonkeyCode's free model access is one option for running the suite.
    # Disclosure: This article was prepared as part of MonkeyCode's product outreach.
    raise NotImplementedError

def grade(raw: str, expect: dict) -> str:
    try:
        inv = Invoice.model_validate_json(raw)
        inv.check_math()
    except (ValidationError, ValueError, json.JSONDecodeError) as e:
        return f"fail:structure:{type(e).__name__}"
    if len(inv.items) != expect["items"]:
        return "fail:semantics:item_count"
    return "pass"

def run(suite: str, threshold: float = 0.95) -> int:
    cases = [json.loads(l) for l in open(suite, encoding="utf-8")]
    results = [grade(call_model(c["input"], SYSTEM), c["expect"]) for c in cases]
    passed = sum(r == "pass" for r in results)
    rate = passed / len(cases)
    for case, r in zip(cases, results):
        if r != "pass":
            print(f"{case['id']} {r}")
    print(f"pass_rate={rate:.3f} n={len(cases)}")
    return 0 if rate >= threshold else 1

Running it is deliberately boring, which is the point. The suite should take minutes, cost near nothing, and fail in the same way on your laptop as in CI.

python -m eval.run --suite golden/invoices.jsonl --threshold 0.95

For the replay of past documents, you want an environment that mirrors production without touching it, and this is the second place free infrastructure earns its keep. MonkeyCode's free server option is enough to stand up a throwaway replay job that reads a sample of stored documents, re-runs extraction against the pinned prompt, and diffs the results before anything reaches a real customer.

# on the free server: replay yesterday's documents against the candidate prompt
python -m eval.replay --since 2026-10-17 --sample 500 --diff totals

That replay is also how you shrink the detection gap. Run it nightly against the current pinned prompt, alert when the structural pass rate moves more than two points week over week, and you convert a customer-reported surprise into a trend line you own.

When the free path is the right path

Not every stage of this workflow deserves the same spend, and a simple split keeps the cheap tiers where they belong.

Stage	Cheap/free model + free server	Production inference
Golden-set gate on every PR	Yes, latency is irrelevant	Overkill
Nightly replay of sampled documents	Yes, with PII stripped or synthetic	Only if data cannot leave
Pre-release canary mirror	Yes, for structural checks	Needed for load and SLO proof
Serving live traffic	No	Yes

Two honest limitations apply. Free tiers change, and anything that might be rate-limited out from under you cannot sit on the critical path of a deploy gate without a fallback, so keep the suite runnable against a paid endpoint too. Structural validity is also not correctness: a perfectly shaped invoice can still describe the wrong purchase, which is why the twenty-sample human check stays in the process.

You should not follow this approach if you handle regulated data that cannot leave a controlled environment, if your evaluator needs to reproduce identical outputs months later for audit, or if the extraction step is on a latency SLO where a slow shared tier would distort your measurements. In those cases keep the golden set and the strict schema, and run the harness on infrastructure you own.

What goes in the postmortem file

The lesson worth keeping is narrow and portable: availability dashboards cannot see silent schema drift, so you need a quality signal that runs on every change and a replay that measures the detection gap. Add one golden case for every incident you close, review the suite monthly for rot, and record prompt and model versions on every request so the next investigation starts with evidence instead of guesses. If you want to try the harness without touching a card, the open-source MonkeyCode project's free model access and free server option are enough to run the golden set and the nightly replay end to end, and the schema and gating code above ports to any other provider unchanged.