Mutants Before Merges: A Two-Sided Sensitivity Gate for C++ Eval Harnesses

An illustrative trace. A 40-line C++ patch clears the compiler, passes 41 golden cases and merges. Two weeks later a downstream consumer reports wrong scaling on a load path none of the goldens touch. Nothing in the pipeline crashed. The harness was blind — every test passed because no assertion ever inspected the broken path.

That trace is a composite pattern, not a report on a named team. The defect class behind it is measurable all the same. This article builds a gate that asks whether a C++ eval harness is capable of failing, and that refuses to grade a patch when the answer is no.

Golden cases measure agreement, not sensitivity

Golden cases answer one question: does current output match recorded output? That is agreement. It says nothing about whether the harness would notice if the behavior underneath changed.

Two properties matter. Sensitivity is the probability that the harness fails when behavior breaks. Specificity is the probability that it passes when behavior is correct. A harness that fails on every run has perfect sensitivity and no usefulness; a harness that passes everything is the failure mode in the opening trace.

More golden cases raise agreement coverage. They raise sensitivity only if the new cases touch code that was previously unasserted, and stdout diffs frequently do not. A case that checks a final total can pass while an intermediate bound check was deleted.

The gate below measures both properties before a patch is graded, using mutants as calibrated faults.

Step 1 — Freeze the harness contract

A gate needs a fixed pass/fail definition, or the mutants measure noise. Pin the build type, the runner and the wall-clock ceiling in one place, and treat any change to them as a change to the gate itself.

cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j2
ctest --test-dir build --output-on-failure   # the bench command

Everything downstream calls exactly these commands. If this patch class needs sanitizers, keep a second profile and run the gate twice, one ledger per profile.

Step 2 — Run a flake pre-pass

Kill rates computed on a flaky harness are fiction. Before generating any mutant, run the bench three times on the untouched tree and require identical results.

for i in 1 2 3; do
  ctest --test-dir build --output-on-failure >/dev/null 2>&1 || echo "flake run $i"
done

Any deviation stops the gate. Ordering dependence, timing assumptions and shared temp files surface here far more cheaply than inside a mutant run.

Step 3 — Build a two-class mutant seed set

Mutants are small deliberate faults. They fall into two classes with opposite expected results.

Class	Expectation	Purpose	Examples
sensitivity	harness must FAIL	proves the harness can see a break	<= to <, sign flip, deleted early return, dropped const, rounded float literal
inert	harness must PASS	proves the harness is not failing on trivia	reformatted comment, reordered #include, renamed local, added dead branch

Hand-write ten sensitivity mutants and three inert ones. They cost nothing and they encode what the team actually cares about. Model-drafted mutants extend the set afterwards; the note further down explains why they are candidates rather than ground truth.

[
  {"id": "M01", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "if (n <= cap)", "repl": "if (n < cap)"},
  {"id": "M02", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "return a - b;", "repl": "return b - a;"},
  {"id": "M03", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "  const auto lim = limit;", "repl": "  auto lim = limit;"},
  {"id": "I01", "class": "inert", "file": "src/scale.cpp",
   "find": "// clamp input", "repl": "// clamp input to cap"}
]

Each anchor must match exactly once. An ambiguous anchor is a skipped mutant, never a silent edit. That rule keeps the ledger honest when a patch moves code around.

Step 4 — Drive the gate

The driver below is an illustrative reference implementation, not code executed for this article. It restores each source file in a finally block, so a crash cannot leave a mutant in the tree.

#!/usr/bin/env python3
# mutgate.py - two-sided sensitivity gate for a C++ eval harness
import json, pathlib, subprocess, sys, time

ROOT = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
KILL_FLOOR = float(sys.argv[2]) if len(sys.argv) > 2 else 0.60
BUILD = "cmake --build build -j2"
BENCH = "ctest --test-dir build --output-on-failure"
TIMEOUT = 90

def run(cmd, timeout=TIMEOUT):
    t0 = time.time()
    try:
        p = subprocess.run(cmd, cwd=ROOT, shell=True, capture_output=True,
                           text=True, timeout=timeout)
        return p.returncode, (p.stdout + p.stderr)[-1200:], round(time.time() - t0, 1)
    except subprocess.TimeoutExpired:
        return 124, "wall clock exceeded", timeout

def observe():
    rc, log, secs = run(BUILD)
    if rc != 0:
        return "BUILD_ERROR", log, secs
    rc, log, secs2 = run(BENCH)
    return ("FAIL" if rc != 0 else "PASS"), log, secs + secs2

def main():
    mutants = json.loads((ROOT / "mutants.json").read_text())
    ledger = {"clean_control": None, "mutants": [], "verdict": None}

    state, log, secs = observe()
    ledger["clean_control"] = {"state": state, "secs": secs}
    if state != "PASS":
        ledger["verdict"] = "REJECT_HARNESS"
        (ROOT / "mutgate-ledger.json").write_text(json.dumps(ledger, indent=2))
        print("REJECT_HARNESS: clean tree does not pass")
        return

    for m in mutants:
        src = ROOT / m["file"]
        original = src.read_text()
        hits = original.count(m["find"])
        if hits != 1:
            ledger["mutants"].append({"id": m["id"], "class": m["class"],
                                      "state": "SKIPPED", "note": f"anchor x{hits}"})
            continue
        src.write_text(original.replace(m["find"], m["repl"], 1))
        try:
            state, log, secs = observe()
        finally:
            src.write_text(original)
        ledger["mutants"].append({"id": m["id"], "class": m["class"],
                                  "state": state, "secs": secs})

    sens = [x for x in ledger["mutants"]
            if x["class"] == "sensitivity" and x["state"] != "BUILD_ERROR"]
    killed = [x for x in sens if x["state"] == "FAIL"]
    inert_noise = [x for x in ledger["mutants"]
                   if x["class"] == "inert" and x["state"] == "FAIL"]
    build_errors = [x for x in ledger["mutants"] if x["state"] == "BUILD_ERROR"]
    rate = len(killed) / len(sens) if sens else 0.0

    if build_errors:
        ledger["verdict"] = "REVIEW_BUILD_ANCHORS"
    elif inert_noise:
        ledger["verdict"] = "REVIEW_GRADER_NOISE"
    elif rate >= KILL_FLOOR:
        ledger["verdict"] = "ACCEPT"
    else:
        ledger["verdict"] = "UNKNOWN_HARNESS_BLIND"

    (ROOT / "mutgate-ledger.json").write_text(json.dumps(ledger, indent=2))
    print(json.dumps({"kill_rate": round(rate, 2), "verdict": ledger["verdict"]}))

if __name__ == "__main__":
    main()

Run it as python3 mutgate.py . 0.60. The threshold is a policy choice, not a discovered constant. Start at 0.6, raise it for safety-critical patches, and record the value beside the ledger.

Step 5 — Read the verdict table, not just the number

Clean control	Inert mutants	Kill rate	Verdict
PASS	all PASS	at or above floor	ACCEPT - the harness earns the right to grade this patch
PASS	all PASS	below floor	UNKNOWN - survivors mark unasserted paths; write cases instead of merging on green
PASS	any FAIL	any	REVIEW - harness fails on harmless change; fix specificity first
PASS	-	-	REVIEW - an anchor or mutation caused a build error; kills are not counted
FAIL	-	-	REJECT - the harness itself is broken; no patch can be graded

UNKNOWN is the useful state. It converts a false green into an explicit work item, and it costs one line in the review checklist: survivors list, case inventory, owner.

Step 6 — Turn survivors into cases

Every sensitivity mutant that survives is a concrete assertion the harness does not have. Map it to the smallest case that would fail: an input where n == cap, a negative operand, a value that must not be rounded. Group survivors by the function they touch and open one case-writing task per group rather than one per mutant.

Keep the ledger as a per-patch artifact. Two runs of the same patch with different verdicts usually mean the mutant set changed, not the code.

Where disposable mutants and a free runner fit

A mutation gate needs a lot of throwaway work: dozens of candidate mutants per patch, plus a runner that rebuilds each time in a sandbox. Both halves tolerate imperfection, which changes the economics.

Model-drafted mutants are cheap candidates, not ground truth. A model produces edits that look plausible, which correlates with the bugs a reviewer would miss, but the distribution is biased toward what the model finds natural. It will miss the boundary case a maintainer has been burned by. The workflow that holds up: hand-write the sensitivity and inert seed set, ask a model for thirty more candidates per patch, then keep only those whose anchors resolve and whose class is unambiguous.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states the current offering includes free model access and a free server option, with a free token allowance advertised at the time of writing as 10 million tokens. Allowances, hardware and terms change, and the project is the only authority on its own limits, so verify the current terms before depending on them.

The free server option fits the runner half because mutation runs are disposable by construction: a container that rebuilds a mutant, runs the bench and is discarded holds no state worth keeping. Keep the merge-blocking golden run in whatever CI the team already trusts, and spend the disposable capacity on the sensitivity suite.

Limitations and who should not use this

Kill rate is a heuristic, not a probability of catching real defects. Mutants and real bugs are different distributions, and a high kill rate can coexist with an unasserted path that no mutant happened to hit.

BUILD_ERROR mutants are excluded from both numerator and denominator on purpose. Counting a compile failure as a kill inflates the rate and hides the case where a refactor broke the anchors.

Mutant runs multiply CI time by the size of the set. With 40 sensitivity mutants and a 30-second bench, the gate costs roughly 20 minutes plus rebuilds. Budget for it, or run it nightly against a sample of recent patches.

Skip this approach if the harness is already a long-running fuzzing or sanitizer campaign, if the codebase needs formal guarantees rather than sampled faults, or if a two-person team has fewer than a dozen golden cases to begin with. In that state, writing cases teaches more than measuring them.

Closing

A harness that cannot fail cannot be trusted when it passes. Build the two-sided gate once, keep the mutant set small and hand-seeded, and read UNKNOWN as a scheduled task rather than a failure. If the runner half is what is blocking you, MonkeyCode's free model access and free server option are a reasonable place to try it on real patches.