Keep a Ledger of Model Failures Instead of Trusting the Release Notes

Keeping a model endpoint in production usually feels like a configuration change until the failures you already fixed start returning under slightly different shapes. A new model does not need to be worse on average; it only needs to forget one field constraint that your code learned the hard way, and that is exactly the case a clean release note will not mention. The cheapest protection is not a huge evaluation set or another schema draft; it is a failure ledger, a small collection of the inputs and invariants that previously produced real errors, replayed against whatever endpoint you are about to trust next.

When a parse failure disappears, most teams throw away the broken example. The prompt gets clarified, the parser gets a guard clause, the issue closes, and the only artifact left is a commit message that says something like handle missing vendor. Two releases later a new model returns the same bad shape, and the lesson has to be relearned by someone who may never have seen the original ticket. A model endpoint is not a stable library; an upgrade silently replaces the thing that failed, so the valuable part of your testing is not a snapshot of average quality but a history of the specific contracts that mattered.

Keeping that history is simple enough to do on a free schedule rather than only when somebody remembers. MonkeyCode's free model access and free server option make it practical to store a small JSONL ledger and replay it without dedicated infrastructure, but the contents still need to be curated by a human. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Start with the smallest record that captures the original failure. You do not need to keep the full model transcript, the timestamp, or the temperature. You need the prompt, a stable identifier, and the invariant that the failure violated: a required key, a type boundary, or a finite set of allowed values. Store one record per line in failures.jsonl, treating each line as a permanent reminder rather than a test for semantic greatness.

{"id": "invoice_014", "prompt": "Extract vendor and total from the following receipt...", "requires": ["vendor", "total"], "types": {"total": ["int", "float"]}}
{"id": "ticket_221", "prompt": "Convert this support email into a ticket with priority and title.", "requires": ["priority"], "values": {"priority": ["low", "medium", "high", "urgent"]}}

The replay loop is deliberately thin. For each stored prompt, call the current endpoint, parse the generated object, and check only the invariants from the record. You are not asking the model to reproduce an old exact answer, and you are not checking for style. You are asking whether the new endpoint still respects the minimum shape that once caused a real user-visible failure.

This is the opposite of a nondeterminism probe. You are not repeating one prompt many times to measure variation inside a single endpoint; you are repeating many historical prompts against new endpoints to watch for the return of known failure modes.

A small Python harness can be enough, as long as you remember that the provider-specific parsing should be adapted to your own response envelope. The example below is portable pseudocode, not a complete integration, so keep the endpoint and key in the environment rather than in the script.

import json, os, urllib.request

def call_model(prompt):
    payload = json.dumps({"prompt": prompt})
    req = urllib.request.Request(
        os.environ["MODEL_ENDPOINT"],
        data=payload.encode(),
        headers={
            "Authorization": f"Bearer {os.environ['MODEL_KEY']}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        body = json.load(resp)
        # Adapt this to the provider's actual response shape.
        return json.loads(body["choices"][0]["message"]["content"])

def invariants_pass(actual, record):
    for field in record.get("requires", []):
        if field not in actual:
            return False, f"{field} missing"
    for field, allowed in record.get("types", {}).items():
        if field in actual and type(actual[field]).__name__ not in allowed:
            return False, f"{field} is {type(actual[field]).__name__}"
    for field, allowed in record.get("values", {}).items():
        if field in actual and actual[field] not in allowed:
            return False, f"{field} has unknown value {actual[field]!r}"
    return True, "ok"

with open("failures.jsonl") as f:
    records = [json.loads(line) for line in f]

for record in records:
    ok, reason = invariants_pass(call_model(record["prompt"]), record)
    if not ok:
        print(f"{record['id']}: {reason}")

The output is not a benchmark and it will not tell you that a model is better. What it gives you is a short list of resurrection warnings. When a previously fixed failure reappears after an endpoint change, the identifier on the line lets you trace back to the original issue instead of hunting through a vague migration bug with no starting point.

You should replay the ledger when the default model is replaced, when the provider changes the response envelope, or when someone wants to move from a paid endpoint to a free one and expects the behavior to be identical. The ledger turns that expectation into a checkable statement. If the free endpoint passes the old invariants, that is useful evidence for the swap; if it fails two records in the same category, you know which normalization code to revisit before the migration continues.

There are limits to this approach, and ignoring them turns the ledger into a different kind of trap. Saving only failures means you are testing for the past, not for the future, and you can become so focused on old edge cases that you miss new classes of error. A failure ledger cannot tell you whether common-case quality improved, whether the new endpoint is faster under real traffic, or whether a different model now follows instructions more faithfully. The invariant set is only as reliable as the bug taxonomy you maintain, and adding a brittle check for every historical defect can end up constraining the model instead of protecting the user.

This is not for everyone. If your application uses a single frozen model and never changes endpoints, the ledger has little to do after the first pass. If your downstream parser is already robust enough to normalize any valid representation, you may not need a separate collection of old failures. And if nobody on the team is willing to add one record each time a parsing bug is fixed, the replay will quietly become stale and give you false confidence.

If a free model endpoint and a small place to keep a JSONL file are available to you, start the ledger with the next parsing mistake you fix. It costs almost nothing, and it turns last month's painful lesson into a test that survives the next release.