Mutation-Score the Grader: A Prompt Lockfile for AI C++ Patch Evals

A mid-size systems team merged an AI-written patch into a lock-free queue. Unit tests stayed green. Two weeks later a prompt edit asked the model to prefer simpler atomics. The new patch compiled. The golden case still passed, because it never ran two producers. Production dropped items under load. The harness did not fail. It had never been mutation-scored, and the prompt had never been pinned.

Cheap C++ patches create a second class of debt. Golden cases rot. Graders accept weaker oracles. Prompt text drifts in a chat window with no hash. The rest of this article describes a small, reproducible eval harness that treats the prompt as a locked dependency and mutation-tests the grader itself.

What this harness is for

The workflow targets AI-generated C++ patches that look locally correct. It does not replace code review. It catches three silent failures: prompt drift, oracle rot, and patches that satisfy only the tests the team already showed the model.

A demo test repeats the happy path the model already saw. A contract case states what must remain true after a patch the model has not been coached to pass. The difference is the whole method.

Artifact: a prompt lockfile plus a mutation pass

The artifact has four files.

1. prompt.lock.json — hashes of the system prompt, the user template, and the eval corpus.
2. cases/ — golden cases with inputs, forbidden diffs, and grader commands.
3. mutate_grader.py — a script that weakens oracles on purpose and expects the harness to fail.
4. run_eval.sh — a thin runner that compiles, links sanitizers, and records grades.

Each file is small enough to keep in the same repository as the C++ library under test. The lockfile is the product. The mutation pass is how the team learns the product is not theater.

Step 1: Pin the prompt like a lockfile

Prompts change faster than headers. A one-line edit can alter which includes the model emits. The lockfile records cryptographic hashes, not the prompt prose, so review diffs stay short.

{
  "schema": "cpp-eval-lock/v1",
  "prompt": {
    "system_sha256": "replace-with-sha256-of-system.txt",
    "user_template_sha256": "replace-with-sha256-of-user.md.tmpl",
    "style_rules_sha256": "replace-with-sha256-of-invariants.md"
  },
  "corpus": {
    "cases_sha256": "replace-with-sha256-of-cases-tree",
    "grader_sha256": "replace-with-sha256-of-mutate_grader.py"
  },
  "language": "c++17",
  "sanitizers": ["address", "undefined"]
}

A pre-commit hook recomputes the hashes. A mismatch is a failed eval, even if every test is still green. That is the point. Green tests with a drifted prompt are a silent regression of the harness, not of the library.

#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import hashlib, json, pathlib, sys
lock = json.loads(pathlib.Path("prompt.lock.json").read_text())
def sha(p):
    return hashlib.sha256(pathlib.Path(p).read_bytes()).hexdigest()
checks = {
    "system_sha256": sha("system.txt"),
    "user_template_sha256": sha("user.md.tmpl"),
    "style_rules_sha256": sha("invariants.md"),
}
bad = [k for k, v in checks.items() if lock["prompt"][k] != v]
if bad:
    print("lock mismatch:", ", ".join(bad))
    sys.exit(2)
print("prompt lock ok")
PY

The snippet is a proposal. Teams should replace the placeholder hashes before using it as a gate. Unsigned prose in a chat log is not a pin.

Step 2: Write golden cases as contracts, not demos

Each case records three oracles: compile flags, forbidden diff substrings, and a behavior command. Forbidden diffs catch deletions the model likes when a prompt says "simplify." Behavior still matters. A patch that keeps the atomics and loses items must fail the command oracle.

{
  "id": "queue-mpmc-no-loss",
  "source_files": ["src/queue.hpp", "src/queue.cpp"],
  "forbidden_diff": [
    "std::memory_order_relaxed on head or tail",
    "removal of mutex or atomic around published index"
  ],
  "compile": {
    "std": "c++17",
    "flags": ["-Wall", "-Werror", "-fsanitize=address,undefined"]
  },
  "behavior": {
    "cmd": ["./queue_mpmc_test", "--producers", "2", "--consumers", "2", "--n", "10000"],
    "expect_exit": 0,
    "expect_stdout_contains": ["lost=0"]
  }
}

A minimal C++ golden driver looks like the following. It is an example driver, not a proven lock-free queue.

// cases/queue_mpmc_test.cpp — example driver, not production queue code
#include "queue.hpp"
#include <atomic>
#include <iostream>
#include <thread>
#include <vector>

int main() {
  const int producers = 2, consumers = 2, n = 10000;
  Queue<int> q;
  std::atomic<int> produced{0}, consumed{0};
  std::vector<std::thread> ts;
  for (int i = 0; i < producers; ++i) {
    ts.emplace_back([&] {
      for (int k = 0; k < n; ++k) {
        q.push(k);
        produced.fetch_add(1, std::memory_order_relaxed);
      }
    });
  }
  for (int i = 0; i < consumers; ++i) {
    ts.emplace_back([&] {
      int v;
      while (consumed.load(std::memory_order_relaxed) < producers * n) {
        if (q.try_pop(v))
          consumed.fetch_add(1, std::memory_order_relaxed);
      }
    });
  }
  for (auto& t : ts) t.join();
  const int lost = produced.load() - consumed.load();
  std::cout << "lost=" << lost << "\n";
  return lost == 0 ? 0 : 1;
}

Single-thread tests would have kept the original silent failure green. Contention is part of the oracle. A case that never contends is a demo wearing a contract badge.

Step 3: Grade in layers, then stop early

Numbered graders keep the failure mode readable.

1. Lock grade. Prompt hashes match prompt.lock.json.
2. Compile grade. -Wall -Werror plus ASan and UBSan.
3. Diff grade. Forbidden substrings do not appear in the unified diff.
4. Behavior grade. The golden driver exits 0 with the expected tokens.

A runner can be a shell script. Keep it boring.

#!/usr/bin/env bash
set -euo pipefail
case_id="${1:?case id}"
patch="${2:?unified diff}"

./check_lock.sh
git apply --check "$patch"
git apply "$patch"

g++ -std=c++17 -Wall -Werror -fsanitize=address,undefined \
  -o queue_mpmc_test src/queue.cpp cases/queue_mpmc_test.cpp

python3 grade_diff.py --case "$case_id" --diff "$patch"
./queue_mpmc_test --producers 2 --consumers 2 --n 10000 | tee /tmp/out.txt
grep -q 'lost=0' /tmp/out.txt

Early stop matters. A compile failure should not be recorded as a behavior pass. Mixed grades are how silent regressions hide in dashboards. One CSV line per patch is enough.

Step 4: Mutation-score the grader

If the expected stdout is deleted and the harness still passes, the oracle is decoration. Mutation scoring applies hostile edits to the case definition, not to the C++ library. The harness must fail each mutant.

# mutate_grader.py — proposal: mutants target the oracle, not the product
import copy, json, pathlib, sys

def load(path):
    return json.loads(pathlib.Path(path).read_text())

def mutants(case):
    out = []
    c = copy.deepcopy(case)
    c["behavior"]["expect_exit"] = 1
    out.append(("invert-exit", c))
    c = copy.deepcopy(case)
    c["behavior"]["cmd"] = ["./queue_mpmc_test", "--producers", "1",
                            "--consumers", "1", "--n", "1"]
    out.append(("drop-contention", c))
    c = copy.deepcopy(case)
    c["forbidden_diff"] = []
    out.append(("drop-forbidden", c))
    c = copy.deepcopy(case)
    c["compile"]["flags"] = ["-O0"]
    out.append(("drop-sanitizers", c))
    return out

def harness_fails(case) -> bool:
    # Labeled stub: wire this to run_eval.sh against the mutant case.
    return False

def main():
    case = load(sys.argv[1])
    killed = 0
    rows = mutants(case)
    for name, mutant in rows:
        failed = harness_fails(mutant)
        print(f"{name}: {'killed' if failed else 'SURVIVED'}")
        killed += int(failed)
    print(f"mutation score: {killed}/{len(rows)}")
    sys.exit(0 if killed == len(rows) else 3)

if __name__ == "__main__":
    main()

A surviving mutant means the grader would accept a quieter case. That is how the original queue patch escaped. The single-thread golden case was a surviving mutant of a contention oracle.

Teams should refuse to expand the corpus until the mutation score is full. Extra green cases with dead oracles increase confidence theater. The score measures the harness. It does not measure the library.

Step 5: Run the loop where the patch is actually produced

The lockfile and the mutation pass are local files. They do not require a particular vendor. They do require that each generated patch is saved as a diff, then graded offline.

When a team uses MonkeyCode, the same user.md.tmpl should be the only user prompt sent, and run_eval.sh plus mutate_grader.py should be the commands that actually grade the diff. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator-described free-tier model access and a free server option are enough to exercise that loop. They are not a production SLA, a quota, a hardware claim, or a benchmark.

A practical cadence:

1. Freeze prompt files and refresh prompt.lock.json.
2. Request a patch for one failing library test, using the locked template.
3. Save the unified diff.
4. Run run_eval.sh.
5. Run mutate_grader.py against the case that just graded the patch.
6. Record lock hash, patch hash, grades, and mutation score in one CSV line.

date,lock,patch,lock_grade,compile,diff,behavior,mutants_killed,mutants_total
2026-09-07,a1b2,c3d4,pass,pass,fail,pass,3,4

The sample row is illustrative. A diff fail with a behavior pass is the interesting class: the model restored function and deleted an invariant the tests did not encode. CSV is enough. A dashboard is optional.

Decision table: when to treat green as a pass

Signal	Treat green as pass	Action
Lock mismatch, tests pass	No	Restore prompt files or re-hash deliberately
Mutation score incomplete	No	Repair oracles before adding cases
Compile pass, sanitizer flags dropped	No	Restore ASan and UBSan in the case
Behavior pass, forbidden diff hit	No	Treat as invariant regression
All grades pass, mutation score full	Conditionally	Review remaining design risk

"Conditionally" is intentional. The harness does not prove lock-freedom, linearizability, or API taste. It proves the oracles still bite.

Limitations and who should skip this

The approach is weak for GUI code, for generated protobufs, and for patches whose correctness is visual. It is also weak when the golden driver itself is wrong. Mutation scoring detects quiet oracles. It does not detect a confidently wrong expected value.

Sanitizer-backed C++ tests need a toolchain that actually links libasan. Windows MSVC setups will not paste the sample flags unchanged. The example queue driver can hang if try_pop never returns and the stop condition is sloppy. Timeouts belong in the runner. They are omitted above for brevity.

Teams that already have a mature property-based test suite may find the JSON cases redundant. Teams that cannot pin prompts because every request is a fresh chat with no files should not pretend they have this harness. Do not use this workflow as a substitute for a memory-model review on lock-free structures. Do not publish mutation scores as marketing metrics. A 4/4 kill rate on four crude mutants is not a safety case.

Keep the CSV boring

Cases the model never saw are a separate control. This article is about graders that fail to fail. Pin the prompt. Mutate the oracle. Keep the CSV boring.

Teams that already store diffs from a coding assistant can drop prompt.lock.json and mutate_grader.py beside those diffs and reuse the same runner on free-tier model access or a free server option. That is the whole loop.

MonkeyCode provides free models that can run this workflow.