Diff Last-Good Completions, Not Just Pass Rate

Silent regressions in prompt systems rarely trip a failing test because the grader still awards a pass. A pairwise harness that diffs each new completion against a last-good baseline flags those shifts even when the rubric stays green. Absolute scores answer whether a case remains acceptable under today's assertions. The baseline diff answers a different question: whether the system moved since the last accepted run.

Teams often treat golden files as unit tests and then ship a prompt edit that still satisfies every stored assertion. Answer length grows, a hedging preface appears, or a tool argument is reordered without breaking a loose grader. Production users feel that change as extra latency, a new tone, or another tool round-trip, while CI reports a stable pass rate. The missing artifact is not another golden case; it is a frozen completion from the last accepted run.

Think of the golden file as a building code and the baseline as a photograph of the finished wall. The code says the wall must be load-bearing and fire-rated, which many slightly different walls can satisfy. The photograph shows the specific wall you signed off last Tuesday, including the scuff that nobody wrote a rule for. When the photograph changes and the code still passes, you have a silent regression in the architectural sense, even if no inspector would fail the job.

The workflow below keeps those two artifacts separate on purpose. Golden cases hold invariants you are willing to enforce forever, such as required JSON keys, forbidden phrases, and tool-name allowlists. The baseline store holds the exact normalized completion from the last human-accepted run, keyed by case id and prompt hash. CI then fails closed on grader parse errors, fails hard on broken invariants, and fails as a regression when the photograph changes. That third signal is what a flat pass rate cannot see.

A reproducible core fits in one Python module and two JSON files, with no third-party packages. Completions arrive from any command that writes UTF-8 to stdout, which keeps the harness honest about not owning the model. The grader never calls another model, because an LLM-as-judge can regress in the same silent way this design is trying to catch. If a completion is not valid JSON when the case asked for JSON, the run is a failure, not a skip.

{
  "prompt_id": "billing-status-v3",
  "prompt_sha256": "replace-with-sha256-of-the-prompt-file",
  "cases": [
    {
      "id": "status-paid-en",
      "input": {"order_id": "ord_1042", "locale": "en"},
      "expect": {
        "json": true,
        "required_keys": ["status", "order_id"],
        "forbidden": ["as an AI", "I think"],
        "status_in": ["paid", "pending", "failed"]
      }
    },
    {
      "id": "status-missing-order",
      "input": {"order_id": "", "locale": "en"},
      "expect": {
        "json": true,
        "required_keys": ["error"],
        "forbidden": ["paid"],
        "error_equals": "missing_order_id"
      }
    }
  ]
}

Save that file as golden.json next to a prompt file whose bytes you hash before every run. The hash belongs in the report because a moved prompt with an unchanged golden file is a deployment event, not a comment. Pair it with baseline.json, which starts empty and is written only under an explicit --record flag after a person has read the diff. Recording from CI is how last-good photographs quietly become last-lucky photographs.

#!/usr/bin/env python3
"""Pairwise eval harness: fail-closed graders plus last-good completion diffs."""
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys
from pathlib import Path

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def normalize(text: str) -> str:
    return " ".join(text.strip().split())

def load_json(path: Path):
    return json.loads(path.read_text(encoding="utf-8"))

def complete(cmd: list[str], payload: dict) -> str:
    proc = subprocess.run(
        cmd, input=json.dumps(payload), text=True,
        capture_output=True, check=False,
    )
    if proc.returncode != 0:
        raise RuntimeError(proc.stderr[-500:])
    return proc.stdout

def grade(raw: str, expect: dict) -> list[str]:
    failures = []
    text = raw.strip()
    if expect.get("json"):
        try:
            obj = json.loads(text)
        except json.JSONDecodeError as exc:
            return [f"parse_fail: {exc}"]  # fail closed; do not continue
        for key in expect.get("required_keys", []):
            if key not in obj:
                failures.append(f"missing_key:{key}")
        allowed = expect.get("status_in")
        if allowed and obj.get("status") not in allowed:
            failures.append(f"status_not_in:{obj.get('status')!r}")
        err = expect.get("error_equals")
        if err and obj.get("error") != err:
            failures.append(f"error:{obj.get('error')!r}")
    lowered = text.lower()
    for phrase in expect.get("forbidden", []):
        if phrase.lower() in lowered:
            failures.append(f"forbidden:{phrase}")
    return failures

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--golden", type=Path, default=Path("golden.json"))
    p.add_argument("--baseline", type=Path, default=Path("baseline.json"))
    p.add_argument("--prompt", type=Path, required=True)
    p.add_argument("--cmd", nargs=argparse.REMAINDER, required=True)
    p.add_argument("--record", action="store_true")
    p.add_argument("--fail-on-diff", action="store_true")
    args = p.parse_args()
    cmd = args.cmd[1:] if args.cmd[:1] == ["--"] else args.cmd

    golden = load_json(args.golden)
    prompt_sha = sha256_bytes(args.prompt.read_bytes())
    if prompt_sha != golden.get("prompt_sha256"):
        print(f"prompt_hash_mismatch expected={golden.get('prompt_sha256')} got={prompt_sha}")
        return 2

    baseline = load_json(args.baseline) if args.baseline.exists() else {"cases": {}}
    new_base = {"prompt_sha256": prompt_sha, "cases": dict(baseline.get("cases", {}))}
    hard_fail = diff_fail = 0

    for case in golden["cases"]:
        raw = complete(cmd, {"prompt_id": golden["prompt_id"], **case["input"]})
        fails = grade(raw, case["expect"])
        digest = sha256_bytes(normalize(raw).encode())
        prev = baseline.get("cases", {}).get(case["id"], {})
        changed = prev.get("digest") not in (None, digest)
        status = "FAIL" if fails else ("DIFF" if changed else "PASS")
        print(f"{status} {case['id']} digest={digest[:12]} {fails or ''}".strip())
        if fails:
            hard_fail += 1
        elif changed:
            diff_fail += 1
            print(f"  prev={prev.get('digest', '')[:12]} len {len(raw)}")
        if args.record and not fails:
            new_base["cases"][case["id"]] = {"digest": digest, "sample": normalize(raw)[:240]}

    if args.record:
        args.baseline.write_text(json.dumps(new_base, indent=2) + "\n", encoding="utf-8")
    if hard_fail:
        return 1
    if args.fail_on_diff and diff_fail:
        return 3
    return 0

if __name__ == "__main__":
    sys.exit(main())

A stub completer makes the first run deterministic on a laptop, which is the point of a harness you can check into git. The stub is not a model. It is a fixture that proves the grader, the hash, and the exit codes before a remote completion command is wired in. Replace it only after the pairwise path is green against known bytes.

#!/usr/bin/env python3
import json, sys
payload = json.load(sys.stdin)
if not payload.get("order_id"):
    json.dump({"error": "missing_order_id"}, sys.stdout)
else:
    json.dump({"status": "paid", "order_id": payload["order_id"]}, sys.stdout)

The commands stay small enough to paste into a pull-request check. The first line records a baseline after you have read both completions. The second line is what CI should run, including --fail-on-diff, so a still-valid but moved answer cannot hide behind a green assertion set. Exit code 1 is an invariant break, 2 is a prompt-hash mismatch, and 3 is a silent move against last-good bytes.

python3 eval_harness.py --prompt prompt.txt --golden golden.json \
  --baseline baseline.json --record -- python3 stub_complete.py
python3 eval_harness.py --prompt prompt.txt --golden golden.json \
  --baseline baseline.json --fail-on-diff -- python3 stub_complete.py

Wire a real completer by changing only the command after --, keeping stdin JSON and stdout text as the contract. That is the seam where a local binary, a container, or a remote completion server can sit without teaching the grader about vendors. If the eval loop needs a completion endpoint and you do not want paid API spend on every push, MonkeyCode's free model access and free server option can host that loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness does not depend on that product; it will grade any command that honors the same stdin and stdout contract.

Normalize before hashing or the photograph becomes a flake detector for trailing newlines. The normalize function collapses whitespace so a wrap-around in a pretty-printer does not page the on-call engineer. It does not strip hedging prefaces or reorder JSON keys, because those are the silent moves this design exists to surface. If key order is unimportant, dump parsed objects with sort_keys=True before hashing, and document that choice in the golden file rather than hiding it in the grader.

Treat --record as a migration, not as a cleanup step. When a prompt hash changes, require a new baseline even if every assertion still passes, because the photograph is now of a different building. When only a completion digest changes, print the previous twelve-character prefix beside the new one so the review comment can point at a specific case id. A pass rate of 100 percent with three DIFF lines is a regression report, not a success story, and the exit code should say so in CI.

This approach has sharp limits that matter more than the happy path. Temperature above zero will churn digests even when behavior is stable, so pairwise hashing belongs on frozen, low-variance settings or on stubbed fixtures. Open-ended writing, brainstorming, and sampling-heavy agents will look permanently regressed under --fail-on-diff, which is a misuse of the signal rather than a model failure. The harness also cannot see tool traces it was never given; if the dangerous move lives in an intermediate call, a payload-only baseline will stay politely green.

Skip this design when you do not yet have invariants worth encoding, or when product intent is that answers should wander. Skip it when the completer cannot be pinned to a command with a stable contract, because the photograph will then track infrastructure noise. Skip it as a substitute for evaluation of helpfulness, citation quality, or user preference, none of which a SHA-256 digest can represent. Use it when a prompt is already a production interface and you need the difference between still-legal and still-the-same.

A flat pass rate is a building inspection. The last-good digest is the photograph you keep in the job folder after the inspector leaves. Run the stub until exit codes mean what you think they mean, then point --cmd at whatever completion path you already trust. If that path is a free completion host you are already using for MonkeyCode, keep the same JSON contract and let the pairwise report decide whether yesterday's wall is still the wall you shipped.