Prompt Diff, Eval Drift: Lock-File the Prompt Half of Your AI C++ Eval Harness

A typical failure looks like this. Someone tightens one sentence in the system prompt — prefer becomes must — and the next sweep of the C++ golden set drops fourteen points. The model gets blamed, a migration gets proposed, and nobody notices that the harness never recorded which prompt produced the baseline.

The numbers in that sketch are illustrative, not a benchmark. The failure shape, though, is common: eval harnesses version their golden cases and their graders, and leave the prompt as an untracked string in a YAML file.

This post covers the missing half. It is a lock-file discipline for everything on the prompt side that can move a verdict, plus a paired replay that tells you whether a drop came from the prompt, the model, or the box.

Why the prompt is the unstable half

A C++ patch harness has two halves. The execution half — compiler, standard flag, sanitizers, test runner — is usually pinned, because someone got burned by a -O2-only failure once. The generation half is usually not pinned at all.

That asymmetry is backwards. Compiler versions change a few times a year. Prompt templates change on Tuesday afternoon, in a PR titled "clarify instructions", with no reviewer looking at eval impact.

Three things drift independently:

1. The prompt template text, including whitespace and example ordering.
2. The decoding parameters, which often live in a different file from the template.
3. The provider-reported model identifier string, which can change meaning under a stable name.

Only the third is outside your control. The first two are pure version-control hygiene, and they cause most of the confusion.

Step 1: Hash every input that can move a verdict

The artifact is a single JSON file committed next to the harness. It stores hashes, not content, so the diff stays readable and the prompts can still be edited normally.

{
  "prompt_template_sha256": "<sha256 of harness/prompt.tmpl>",
  "system_prompt_sha256": "<sha256 of harness/system.txt>",
  "golden_manifest_sha256": "<sha256 over sorted golden/*.json>",
  "grader_version": "2.3.1",
  "decode": { "temperature": 0.2, "top_p": 1.0, "max_tokens": 4096 },
  "model_id_reported": "<the string the provider returned, verbatim>",
  "toolchain": {
    "cc": "clang",
    "cc_banner": "<first line of clang --version>",
    "std": "c++20",
    "sanitizers": ["address", "undefined"]
  }
}

Two details matter more than they look. Record the model identifier exactly as reported, never a friendly alias you invented — you want to detect a silent re-point, not describe your intent. And hash the golden manifest as a tree, so adding, removing, or renaming a case all register as drift.

Step 2: Fail the build on drift, not on vibes

The check is short enough that there is no excuse for skipping it. Plain Python 3.11, standard library only.

#!/usr/bin/env python3
"""tools/lockcheck.py - fail CI when a verdict-moving input changed."""
import hashlib, json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
HARNESS = ROOT / "harness"


def sha256_file(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def sha256_tree(paths) -> str:
    h = hashlib.sha256()
    for p in sorted(paths):
        h.update(str(p.relative_to(ROOT)).encode())
        h.update(p.read_bytes())
    return h.hexdigest()


lock = json.loads((HARNESS / "harness.lock.json").read_text())
current = {
    "prompt_template_sha256": sha256_file(HARNESS / "prompt.tmpl"),
    "system_prompt_sha256": sha256_file(HARNESS / "system.txt"),
    "golden_manifest_sha256": sha256_tree((ROOT / "golden").glob("*.json")),
}

drift = {k: (lock[k], v) for k, v in current.items() if lock[k] != v}
for key, (locked, now) in drift.items():
    print(f"DRIFT {key}\n  locked: {locked}\n  now:    {now}", file=sys.stderr)

banner = subprocess.run(
    [lock["toolchain"]["cc"], "--version"],
    capture_output=True, text=True, check=True,
).stdout.splitlines()[0].strip()
if banner != lock["toolchain"]["cc_banner"]:
    print(f"WARN toolchain changed: {banner}", file=sys.stderr)

sys.exit(1 if drift else 0)

Toolchain changes warn instead of failing, because a container rebuild should not block a prompt fix. Prompt and golden-case changes fail hard, because those two are the ones people edit casually.

Step 3: Replay both versions, paired

A before/after comparison across two nights is worthless. CI machines differ, the provider's load differs, and C++ tests are timing-sensitive at the margins. Run the same golden set, same order, same repeat count, on both prompt versions, back to back, in one job.

Store one file per case per repeat, runs/<run-id>/<case>/<k>.json, containing at minimum {"verdict": "pass" | "fail"}. Then build a flip table.

#!/usr/bin/env python3
"""tools/flip.py A_DIR B_DIR - paired verdict comparison."""
import json, pathlib, sys
from collections import defaultdict


def pass_rate(run_dir):
    rates = defaultdict(list)
    for f in sorted(pathlib.Path(run_dir).glob("*/*.json")):
        rec = json.loads(f.read_text())
        rates[f.parent.name].append(rec["verdict"] == "pass")
    return {k: sum(v) / len(v) for k, v in rates.items()}


a, b = pass_rate(sys.argv[1]), pass_rate(sys.argv[2])
rows = []
for case in sorted(set(a) | set(b)):
    pa, pb = a.get(case), b.get(case)
    if pa is None or pb is None:
        kind = "missing"
    elif pa == pb:
        kind = "stable"
    elif pa == 1.0 and pb < 1.0:
        kind = "regressed"
    elif pb == 1.0 and pa < 1.0:
        kind = "fixed"
    else:
        kind = "unstable"
    rows.append((case, pa, pb, kind))

for case, pa, pb, kind in rows:
    print(f"{kind:9s} {case:44s} A={pa:.2f} B={pb:.2f}")
summary = defaultdict(int)
for *_, kind in rows:
    summary[kind] += 1
print("\nsummary:", dict(summary))

Run it with at least five repeats per case. With fewer, 1.00 -> 0.50 is indistinguishable from noise and the table will lie to you in a confident font.

Step 4: Read the flip table before reading the diff

The four categories answer different questions, and conflating them is where teams lose days.

Category	Meaning	Next action
regressed	Was always passing, now not	Real signal — read the prompt diff
fixed	Was always failing, now passes	Verify it fixed the intended bug, not the grader
unstable	Passing inconsistently on both sides	Harness bug, not a prompt bug
stable	No change	Ignore, however tempting

unstable deserves emphasis. If a case cannot pass ten times out of ten under the locked prompt, it never belonged in a regression gate. Every unstable row you find is a case that has been injecting randomness into every comparison you have run.

Step 5: Where a free tier and a free server actually help

A paired replay multiplies work: two prompt versions, N golden cases, k repeats, plus the compile-and-sanitize step for each patch. That is the point where a cost gate quietly turns five repeats into two, and two repeats produce an unstable column you cannot trust.

MonkeyCode provides free model access and a free server option, both operator-supplied as of writing. In this workflow the free model access removes the billing reason to lower k, and the free server option is a reasonable place to park the nightly job: lockcheck.py on every commit, the paired replay once per night. Quotas, available models, and server specs change without notice, so check the project page for current numbers rather than trusting a figure quoted in any article, this one included.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution. The product is a means to run the loop, not a substitute for it. A free tier that lets you run two thousand generations is worthless if the golden set has forty cases and the grader is a regex.

Step 6: Know what the lock does not cover

The lock file pins your inputs. It says nothing about three failure modes worth naming.

First, grader bugs. A grader that silently accepts patches with uninitialized reads will show a beautiful fixed column. Canary cases — deliberately broken patches the grader must reject — belong in the manifest, not in a comment.

Second, upstream model drift. When the provider changes what a model identifier means, your lock file records only that the string stayed the same. The flip table catches the effect; the lock file cannot predict it. Re-baseline deliberately, and record the baseline run id in the lock file so the next person knows which sweep the numbers came from.

Third, test-environment leakage. Address-space randomization, thread scheduling, and filesystem ordering can flip a C++ case without any prompt change at all. Randomize case order between the two arms of the replay — if the flip table changes shape, the harness is leaking state.

Who should skip this

Teams with fewer than roughly twenty golden cases should not build a lock-file pipeline yet. With a set that small, every case flip is a conversation, not a statistic, and the ceremony costs more than it returns.

Solo developers running one-off generations should skip it too. The discipline pays off when a failure gets misattributed to a model — which requires at least two people and a shared baseline.

And anyone whose real problem is a flaky C++ test suite should fix that first. A harness cannot stabilize tests it did not write; it can only record how unstable they are.

Start with the cheap half

Before touching prompts again, commit a lock file and a twenty-line check. The first time a passing sweep drops fourteen points, you will know within one command whether the prompt changed, the toolchain changed, or the model moved beneath a stable name — and you will spend the afternoon on the actual bug instead of on a migration proposal.

If you want to run the paired replay somewhere it does not compete with your CI minutes, MonkeyCode's free tier is a low-stakes place to start.