Version Golden Cases Like Schema Migrations

Silent prompt regressions rarely raise exceptions and instead arrive as answers that still parse cleanly. The durable fix is a versioned golden-case catalog whose graders fail closed rather than a larger model. Treat every fixture like a schema migration that hashes inputs and pins the grader contract before rescoring. Re-score the suite whenever the prompt, the tools, or the serving model changes underneath that catalog.

Golden cases rot for the same structural reason that database dumps rot after a messy, unsigned migration. A June transcript encodes tool names, argument shapes, and refusal phrasing that later prompts no longer share. The file still looks authoritative in git, so reviewers treat it as a baseline that was never actually pinned. Scoring new outputs against moving memory is how silent drift survives an otherwise careful product code review.

A grader contract is the checksum, and it should be much narrower than asking whether the answer reads well. Exact string match is too brittle for prose, while an unconstrained judge model is too noisy for merges. The middle path is a short stack of deterministic checks that run before any subjective scorer may speak. Schema validity, required tool calls, forbidden invented arguments, and bounded identifier overlap catch most production misses.

Consider a support agent that must look up an order before it apologizes or issues any customer credit. The golden case is not the final English sentence but the frozen user turn, allowed tools, and required assertions. If the model skips lookup and fabricates a tracking number, a payload-only reviewer may still praise the tone. A pinned grader that requires get_order with a real order_id fails the build, which is the entire point.

Store each case as JSON so the hash stays stable and reviewers can read the fixture diff directly. Avoid notebooks and screenshots as sources of truth, because they cannot be hashed without leftover execution noise. A single object per file keeps git blame useful when an assertion changes for a documented product reason.

{
  "id": "order_lookup_missing_id_v3",
  "version": 3,
  "prompt_id": "support_agent_v12",
  "input": {
    "messages": [
      {"role": "user", "content": "Where is my package? Order is 1842-AX."}
    ]
  },
  "tools": ["get_order", "create_refund"],
  "assertions": {
    "must_call": ["get_order"],
    "must_not_call": ["create_refund"],
    "args": {
      "get_order": {"order_id": {"eq": "1842-AX"}}
    },
    "output_schema": "support_reply_v2",
    "forbid_patterns": ["I assume", "tracking number I made up"]
  }
}

The version field is not decoration and should move whenever the product contract itself actually changes. When a second lookup becomes required, bump the case instead of editing the old fixture quietly in place. Keep v3 in the catalog after v4 lands, and mark the older row retired if behavior changed on purpose. That pattern mirrors schema migrations, because you do not rewrite 001_create_orders.sql after it has shipped.

Graders should read frozen assertions and the model trace, then emit a structured score record without extra network calls. Keeping them local lets a CI runner or a free remote server execute the same bytes with identical results. Pass the raw trace in, and refuse to grade when required keys such as tool_calls or final_text are missing.

import hashlib, json, re
from pathlib import Path

def case_hash(case: dict) -> str:
    payload = json.dumps(case, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]

def grade_trace(case: dict, trace: dict) -> dict:
    if "tool_calls" not in trace or "final_text" not in trace:
        return {"id": case["id"], "case_hash": case_hash(case), "pass": False,
                "failures": ["malformed_trace"]}
    failures = []
    calls = [c["name"] for c in trace.get("tool_calls", [])]
    for name in case["assertions"].get("must_call", []):
        if name not in calls:
            failures.append(f"missing_tool:{name}")
    for name in case["assertions"].get("must_not_call", []):
        if name in calls:
            failures.append(f"forbidden_tool:{name}")
    for name, spec in case["assertions"].get("args", {}).items():
        matched = next((c for c in trace.get("tool_calls", []) if c["name"] == name), None)
        if not matched:
            continue
        for key, rule in spec.items():
            actual = matched.get("arguments", {}).get(key)
            if "eq" in rule and actual != rule["eq"]:
                failures.append(f"arg_mismatch:{name}.{key}")
    text = trace.get("final_text", "")
    for pat in case["assertions"].get("forbid_patterns", []):
        if re.search(pat, text, re.I):
            failures.append(f"forbid_pattern:{pat}")
    return {"id": case["id"], "case_hash": case_hash(case),
            "pass": not failures, "failures": failures}

The harness should do bookkeeping rather than cleverness, because cleverness is where unreproducible scores usually hide. Load every JSON fixture, hash it, grade the captured trace, and write a score document that later diffs can trust. If two score files share an id but the case_hash moved, abort the comparison instead of inventing a quality trend.

def run_suite(case_dir: Path, traces: dict) -> dict:
    scores = []
    for path in sorted(case_dir.glob("*.json")):
        case = json.loads(path.read_text())
        if case["id"] not in traces:
            scores.append({"id": case["id"], "case_hash": case_hash(case),
                           "pass": False, "failures": ["missing_trace"]})
            continue
        scores.append(grade_trace(case, traces[case["id"]]))
    blob = "".join(s["case_hash"] for s in scores).encode("utf-8")
    return {
        "suite_hash": hashlib.sha256(blob).hexdigest()[:16],
        "n": len(scores),
        "passed": sum(1 for s in scores if s["pass"]),
        "failed": sum(1 for s in scores if not s["pass"]),
        "scores": scores,
    }

def diff_scores(old: dict, new: dict) -> list:
    if old["suite_hash"] != new["suite_hash"]:
        raise SystemExit("suite_hash mismatch: fixtures changed, rebase the baseline")
    by_id_old = {s["id"]: s for s in old["scores"]}
    regressions = []
    for s in new["scores"]:
        prev = by_id_old[s["id"]]
        if prev["pass"] and not s["pass"]:
            regressions.append({"id": s["id"], "failures": s["failures"]})
    return regressions

A useful operating rule is to refuse the diff when the suite hash changes between the two score documents. That refusal feels annoying in the moment, much like a failed migration checksum feels annoying during a release. The annoyance is the signal working as designed rather than a defect in the evaluation workflow itself. Rebaseline only with a reviewed commit that records why the product contract moved, not with a silent fixture edit.

Where you execute model calls is a capacity question, not a grading question, and those two concerns should stay uncoupled. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding environment with free model access and a free server option for batch scoring. The graders only need a trace JSON of tools, arguments, and text, so hashes can move to another runner later.

A typical local loop captures traces into traces.json, then scores and diffs against the last known-good file. Run the grade step on the same machine that stores the fixtures so path drift does not masquerade as a model change. Check the score JSON into artifact storage, but only rebase baseline.json when a human has accepted the new contract.

python grade_suite.py --cases ./golden --traces ./traces.json --out ./scores/2026-09-07.json
python diff_scores.py --old ./scores/baseline.json --new ./scores/2026-09-07.json

Interpret the diff as you would interpret a failing integration test rather than as a writing-quality debate among reviewers. A missing_tool:get_order failure is an actionable regression because it names the contract that the new trace broke. A complaint that the apology felt colder is not actionable unless coldness was encoded as a pattern or field. Teams that skip that encoding argue about vibes while hashes stay green, which is how regressions remain silent.

This approach has sharp limits that matter more as agent frameworks make it cheap to generate additional plausible text. Deterministic graders cannot certify that a paragraph is kind, legally safe, or factually complete beyond asserted fields. They also punish valid paraphrases if you overuse exact string equality on user-facing prose instead of identifiers. Keep equality checks on identifiers and tool arguments; treat a free server as a batch runner, not an availability contract.

Skip the method if you evaluate open-ended creative writing or cannot freeze tool names for a given release train. Skip it if the organization wants a single judge model to replace contracts, because that swaps one silent failure for another. Use it when the system already looks like an API, with named tools, typed arguments, and replies that must not invent identifiers. In that shape, versioned golden cases behave like migrations, and prompt changes become reviewable diffs instead of folklore.