Model Swap or Grader Bug? A Replay-First Eval Harness for AI C++ Patches

A patch queue goes red over a weekend. The pass rate falls, nobody touched the compiler, and two people are already rewriting prompts to win the metric back. The real change was one line in a grader that stopped trimming trailing whitespace from program output.

That is not a model problem. It is an attribution problem. An eval harness that only reports a pass rate cannot tell you which of its three inputs moved: the prompt, the fixture, or the grader. The workflow below fixes that by making grading replayable, so every failure gets sorted into a bucket before anyone edits a prompt.

Three inputs, three hashes

An eval case has a prompt, a fixture (the code under test), and a grader. Each one can change independently. If the harness stores only a verdict, all three changes collapse into the same red cell in a spreadsheet.

The fix is boring and effective: hash all three inputs at record time, and hash them again at grading time. A verdict that changes with identical hashes is a model-side event. A verdict that changes with identical model output is a grader-side event. Anything else is a case edit that has to be reviewed like code.

case_id            = split-range-001
prompt_hash        = 8a3f...   (sha256 of prompt file contents)
fixture_hash       = 1c90...   (sha256 of the .cpp file)
grader_hash        = e41d...   (sha256 of the grading script)
output_hash        = 77b2...   (sha256 of the recorded completion)

Golden cases small enough to read

A case that needs a paragraph of explanation to justify its expected output will not survive a reviewer's attention span. Keep each file under a screen. One prompt, one fixture, one expected observable.

{
  "id": "split-range-001",
  "prompt_file": "prompts/fix_split_range.md",
  "fixture": "fixtures/split_range.cpp",
  "compile_flags": ["-std=c++20", "-O2", "-Wall", "-Werror"],
  "stdin": "0 1 2 3 4\n",
  "expect": { "exit_code": 0, "stdout": "0 1 2 3\n", "stderr_empty": true }
}

Note what is not in that file: no prose rubric, no model name, no temperature. Those belong to the run configuration, because they are exactly the things that get swapped without anyone recording it.

Step 1: record once, replay forever

Recording stores the completion alongside its input hashes. Replay grades the stored completion offline, with no network call. The first run of a new case is the only one that spends anything.

# evals/harness.py — reference implementation, Python 3.11+
import hashlib, json, pathlib, subprocess

ROOT = pathlib.Path(__file__).parent

def digest(*parts: str) -> str:
    h = hashlib.sha256()
    for p in parts:
        h.update(p.encode())
        h.update(b"\x00")
    return h.hexdigest()[:16]

def input_hash(case: dict) -> str:
    prompt = (ROOT / case["prompt_file"]).read_text()
    fixture = (ROOT / case["fixture"]).read_text()
    grader = (ROOT / "graders" / "io_match.py").read_text()
    return digest(prompt, fixture, grader)

def grade(case: dict, completion: str) -> dict:
    src = (ROOT / case["fixture"]).read_text() + "\n" + completion
    build = ROOT / "build" / case["id"]
    build.mkdir(parents=True, exist_ok=True)
    (build / "main.cpp").write_text(src)
    cc = subprocess.run(
        ["g++", *case["compile_flags"], "main.cpp", "-o", "a.out"],
        cwd=build, capture_output=True, text=True,
    )
    if cc.returncode != 0:
        return {"compiles": False, "stderr": cc.stderr[-2000:]}
    run = subprocess.run(
        ["./a.out"], cwd=build, input=case.get("stdin", ""),
        capture_output=True, text=True, timeout=10,
    )
    exp = case["expect"]
    ok = (
        run.returncode == exp["exit_code"]
        and run.stdout == exp["stdout"]
        and (not exp.get("stderr_empty", True) or run.stderr == "")
    )
    return {"compiles": True, "ok": ok, "stdout": run.stdout}

The compile step is the expensive part, and it is also the part people skip when they grade on stdout alone. A patch that prints the right bytes but fails -Werror is not a pass.

Step 2: split replay from live

Two subcommands, one report format. The replay arm never touches the network; the live arm calls whatever endpoint the run configuration points at.

# no tokens spent; grades recorded completions
python evals/harness.py replay --report out/replay.json

# spends tokens; records new completions into cassettes/
python evals/harness.py live --endpoint "$EVAL_ENDPOINT" --report out/live.json

# attribute every difference
python evals/harness.py diff out/replay.json out/live.json

Run replay on every pull request. Run live on a schedule, or when a model or prompt change is proposed. That ratio matters: replay is deterministic, cheap, and suitable for a hard gate; live is a sampling instrument and should never block a merge on a single flip.

Step 3: where free capacity actually helps

MonkeyCode's free model access and free server option are the two pieces that make the live arm affordable for an individual maintainer. The value is not the free tier itself; it is that repeated grading runs become cheap enough to do often. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest constraints. First, quota and lifetime terms on a free tier change, so read the project's current page instead of trusting a blog post — including this one. Second, a free workspace is the wrong place to store anything you would not publish, so keep cassettes free of proprietary fixtures.

The harness itself is provider-shaped, not provider-bound. --endpoint is a shell variable for a reason: swap it and the diff logic keeps working.

Step 4: classify the drift, do not just count failures

This is the whole point. One function, three buckets, checked in order.

def classify(case, recorded, replayed, live) -> str:
    if input_hash(case) != recorded["input_hash"]:
        return "CASE_DRIFT"      # prompt, fixture, or grader was edited
    if replayed["ok"] != recorded["ok"]:
        return "GRADER_DRIFT"    # same inputs, same output, different verdict
    if live["ok"] != replayed["ok"]:
        return "MODEL_DRIFT"     # new completion, new verdict
    return "STABLE"

Order is not cosmetic. If the case hash moved, everything downstream is uninterpretable, so it gets reported first and the run is marked "not comparable" rather than "failed".

The resulting report reads like a triage list:

split-range-001   STABLE
parse-args-004    GRADER_DRIFT   verdict ok->fail, output identical
wide-int-002      MODEL_DRIFT    verdict fail->ok
sort-stable-007   CASE_DRIFT     fixture changed; run not comparable

GRADER_DRIFT on a passing case is the most valuable line in the file. It means a grader edit silently reclassified work that nobody asked to re-evaluate.

Step 5: gate on transitions, not totals

A CI gate that fails on "pass rate below 90%" will block merges for reasons unrelated to the merge. Gate on the classification instead.

- run: python evals/harness.py replay --report out/replay.json
- run: python evals/harness.py diff out/replay.json out/last-green.json --fail-on GRADER_DRIFT

MODEL_DRIFT on a live run should open an issue, not fail a build. CASE_DRIFT should require a reviewer to look at the fixture diff. Only grader-side movement deserves a hard stop, because it corrupts every future comparison until someone explains it.

One caveat on sensitivity: this harness assumes your graders can already fail when they should. Proving that is a separate problem with its own fixtures, and it is worth solving before you trust any bucket above.

When to replay and when to go live

Question	Replay	Live
Did a grader edit change verdicts?	yes	no
Did a new model change outputs?	no	yes
Is this cheap enough for every PR?	yes	no
Is this deterministic?	yes	no
Can it catch a fixture typo?	yes	yes

Limitations and who should skip this

Cassettes rot. A recorded completion is only valid for the fixture and flags it was recorded against, so a fixture edit invalidates it by design — which is good, but it means cassettes are not a long-term archive.

Cases that depend on wall-clock time, thread scheduling, or unseeded randomness will produce MODEL_DRIFT noise that is really nondeterminism in the test. Fix the fixture first or drop the case.

Finally, this harness measures individual case flips, not statistical significance. A team trying to decide whether a model is "better" by a few percentage points needs many samples and a proper interval; a single replay run will happily report a difference that is noise. And anyone using this as a security or correctness sign-off for shipped code should stop here — golden cases guard a pipeline, they do not certify a binary.

If the goal is to keep the live arm running without owning a runner, the replay path is the place to start: it costs nothing, it catches the failure mode that burns the most time, and it works before any endpoint is configured.