DEV Community: Finley Li

Kill-Matrix Pruning: Cutting the Golden Cases That Never Catch an AI C++ Patch

Finley Li — Tue, 27 Oct 2026 12:48:26 +0000

A team running an AI patch pipeline once showed me their nightly eval: 214 golden cases, a clean green dashboard, and a production regression that the suite had walked straight past. The postmortem took an hour. Then someone asked the obvious question, and nobody had an answer: how many of those 214 cases had ever failed for a reason worth knowing?

We instrumented the suite instead of arguing about it. Sixty-one cases had never failed on any patch the pipeline had ever scored. Thirty-three failed on nearly every patch, which made them noise rather than signal. The suite looked thorough and was mostly ceremony.

This article describes the instrument we built: a kill matrix. It is a small, reproducible way to find out which golden cases actually earn their runtime, and which ones you are paying for out of habit. A free-tier model and a free-tier runner make the measurement cheap enough to run weekly, but the method stands on its own.

Step 1: Define the mutants before you look at the cases

A kill matrix needs a fixed oracle side. That side is a set of known-wrong C++ patches, called mutants, seeded by hand or by a mutation tool. The rule: every mutant must be a bug a reviewer would reject in a real PR.

Start with twenty. Small, boring, semantics-bearing mutations work best:

// original
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i < n ? i : n - 1;
}

// mutant M07: boundary flipped, one-past-end reachable
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i <= n ? i : n - 1;
}

// mutant M12: unsigned underflow when n == 0
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i < n ? i : n - 1;  // unchanged, but n == 0 now allowed by caller
}

Record each mutant as a patch file plus a one-line description of the defect. Do not include anything you cannot defend in a code review. Style-only mutants produce a matrix full of false kills.

Step 2: Build the matrix with a fixed seed and a fixed compiler

The matrix is a two-dimensional table: mutants down the rows, golden cases across the columns. Each cell answers one question — did this case fail when it saw this mutant?

The runner below is deliberately dumb. It compiles the mutant, applies the case, and writes a boolean. No model, no grader, no rubric. The point of this pass is to measure the cases, not the model.

# build_kill_matrix.py
import json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).parent
MUTANTS = sorted((ROOT / "mutants").glob("*.patch"))
CASES = sorted((ROOT / "cases").glob("case_*.py"))

def run_case(case: pathlib.Path) -> bool:
    """Return True when the case FAILS (i.e. the mutant is killed)."""
    proc = subprocess.run(
        [sys.executable, str(case)],
        cwd=ROOT / "build",
        capture_output=True,
        timeout=120,
    )
    return proc.returncode != 0

matrix = {}
for mutant in MUTANTS:
    subprocess.run(["git", "apply", str(mutant)], cwd=ROOT, check=True)
    subprocess.run(["cmake", "--build", "build", "-j"], cwd=ROOT, check=True)
    row = {}
    for case in CASES:
        row[case.stem] = run_case(case)
    matrix[mutant.stem] = row
    subprocess.run(["git", "checkout", "--", "."], cwd=ROOT, check=True)

(ROOT / "kill_matrix.json").write_text(json.dumps(matrix, indent=2))
print(f"{len(MUTANTS)} mutants x {len(CASES)} cases written")

Two hygiene rules keep the matrix honest. First, freeze the compiler and flags for the whole run; a matrix built across two toolchains is two matrices stapled together. Second, run each cell once and record timing, because a case that takes four minutes to kill one mutant is a different purchase than one that takes four seconds.

Step 3: Read the matrix in four buckets

The raw JSON is not the deliverable. The classification is. Sort each case by how many mutants it kills, then look for the four shapes that matter.

Shape	Signal in the matrix	Action
Discriminating	Kills at least one mutant no other case kills	Keep, and keep it fast
Redundant	Its kills are a strict subset of another case's kills	Fold into the stronger case
Never fires	Zero kills across the whole mutant set	Investigate or delete
Always fires	Kills every mutant, including the baseline	Broken case, or it is testing the build

The "never fires" bucket is where the silent regressions hide. A case with zero kills is either asserting something no realistic bug can violate, or it is silently skipping itself — a missing fixture, a disabled #ifdef, a test binary that swallowed a nonzero exit. Both need a human, not a deletion.

Step 4: Compress with greedy set cover, then validate on a holdout

Once you know which cases kill which mutants, the minimal suite is a set cover problem. Greedy selection gets you within a constant factor of optimal and takes about thirty lines.

# prune_suite.py
import json

matrix = json.load(open("kill_matrix.json"))
mutants = list(matrix)
cases = list(next(iter(matrix.values())))

killers = {c: {m for m in mutants if matrix[m][c]} for c in cases}
chosen, uncovered = [], set(mutants)

while uncovered:
    best = max(cases, key=lambda c: len(killers[c] & uncovered), default=None)
    if best is None or not (killers[best] & uncovered):
        break
    chosen.append(best)
    uncovered -= killers[best]

print(f"kept {len(chosen)}/{len(cases)} cases; uncovered mutants: {sorted(uncovered)}")
json.dump(chosen, open("suite.json", "w"), indent=2)

Split the mutant set 70/30 before you run this. Prune on the 70 percent, then check how much of the 30 percent the reduced suite still kills. A pruned suite that drops below roughly ninety percent holdout coverage was fit to the mutants, not to the code.

Step 5: Put the sweep on a free runner and measure the spend

This is where the harness meets infrastructure. The account's outreach brief points at two MonkeyCode options — free model access and a free server option — and both are relevant here for one reason: the kill matrix is a volume job. Twenty mutants across a hundred cases is two thousand executions, and the interesting question is not whether you can afford it once, but whether you can afford it every week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tier as an unverified budget until your own numbers exist. The brief cites a free token allowance; that figure is a marketing claim until a script counts what your sweep actually consumes. Log tokens per mutant, wall-clock per case, and failures per run, then compare against the allowance. If the ratio is comfortable, schedule the sweep weekly. If not, run it on the top twenty cases only and prune quarterly.

# one sweep, three numbers worth keeping
/usr/bin/time -f "wall=%e s" python build_kill_matrix.py 2>> sweep.log
grep -c '"true"' kill_matrix.json >> sweep.log
wc -l sweep.log

The same caution applies to the free server. Nothing in the brief says how long it stays up, how fast it builds, or whether it survives a -j build of your project. Measure those three things on day one, and keep a fallback runner configured so a capacity surprise costs you one sweep, not one quarter.

Step 6: Guard the pruning against its own success

Aggressive compression has a failure mode: the suite becomes fast, green, and blind. Three cheap guards prevent it.

Holdout mutants. Keep the thirty percent permanently out of selection, as in Step 4.
Canary cases. Add two cases that must fail on every run, one by design and one by an injected marker. If either passes, the harness is broken, not the code.
Quarterly re-seed. Models drift and so do bugs. Add ten fresh mutants every quarter drawn from real review rejections, and re-run the matrix.

Who should not do this

Skip this method if your mutant set would be invented rather than harvested from real rejections — a matrix of synthetic bugs measures your imagination, not your suite. Skip it if you have fewer than roughly fifteen cases, because set cover over a tiny universe just renames the problem. And skip it if the eval runs on a shared cluster where a two-thousand-execution sweep would starve other jobs; the measurement is not worth the queue.

What remains is a cheap habit. Once a quarter, you learn which of your golden cases still catch something, which never did, and what a full sweep actually costs on whatever free capacity you are leaning on. The dashboard stops being a number and starts being an argument.

Shuffle the Golden Cases: Catching Eval State Leaks That Order Hides

Finley Li — Tue, 20 Oct 2026 09:32:28 +0000

Shuffle the Golden Cases: Catching Eval State Leaks That Order Hides

A patch-eval suite can be green for months and still be lying. The lie usually has nothing to do with the patch. It has to do with the sequence.

Here is the failure shape. Case 12 compiles src/vec.cpp with -DEVAL_VARIANT=sse and drops build/vec.o into a scratch directory shared by the whole run. Case 17 wants the scalar variant, but its build script skips compilation when build/vec.o already exists. Case 17 links the SSE object, passes its test, and reports PASS. The patch under review never touched vector code.

Nothing in case 17's log mentions case 12. Nothing in the diff mentions either. The suite is order-dependent, and because the order was fixed in the manifest, the dependency was invisible.

1. Treat order as an input you must vary

Most eval harnesses treat the case list as configuration. It is really an input variable — as much as the patch, the compiler flags, or the model version.

If a verdict changes when that input changes, the suite has a leak. If it does not, order independence is a property you can assert, not a hope you maintain.

The gate is simple: run the suite in a canonical order twice, then in a handful of seeded shuffles. Compare verdict vectors, not logs. Flag any case whose verdict moves.

2. Make the manifest the only source of order

Cases live in a JSON manifest. Each entry owns its command, its timeout, and its expected verdict. No shell script reorders anything downstream.

[
  {"id": "c01_vec_basic",   "cmd": ["bash", "cases/c01.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c12_vec_sse",     "cmd": ["bash", "cases/c12.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c17_vec_scalar",  "cmd": ["bash", "cases/c17.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c22_asan_owner",  "cmd": ["bash", "cases/c22.sh"], "expect": "FAIL", "timeout": 300}
]

Two details matter. First, expect is recorded even for cases that must fail — a negative case that starts passing is a regression too. Second, every case runs in the same scratch root per suite run. Isolating each case in a fresh container would hide exactly the bug this gate exists to find.

3. Run canonical twice, then seeded shuffles

The harness below is the whole gate, roughly sixty lines. It is plain Python on purpose: it needs to be readable enough that someone debugging a flip can trust it.

# shuffle_gate.py — order-dependence gate for a patch-eval suite
import json, random, shutil, subprocess, sys, time
from pathlib import Path

CASES = {c["id"]: c for c in json.loads(Path("cases.json").read_text())}
ROOT  = Path("scratch")

def run_suite(order, tag):
    work = ROOT / tag
    if work.exists():
        shutil.rmtree(work)          # reset once per run, never per case
    work.mkdir(parents=True)
    rows = []
    for cid in order:
        case = CASES[cid]
        t0 = time.monotonic()
        try:
            p = subprocess.run(case["cmd"], cwd=work, capture_output=True,
                               text=True, timeout=case.get("timeout", 300))
            got = "PASS" if p.returncode == 0 else "FAIL"
        except subprocess.TimeoutExpired:
            got = "TIMEOUT"
        rows.append({"id": cid, "verdict": got,
                     "secs": round(time.monotonic() - t0, 2)})
    return rows

def vector(rows):
    return {r["id"]: r["verdict"] for r in rows}

Canonical order runs twice with different tags so a TIMEOUT or a flaky model call shows up before shuffling begins. Shuffles use a fixed seed list committed to the repo, because a flip you cannot reproduce is a flip you cannot fix.

def main():
    base_order = list(CASES)
    a = run_suite(base_order, "canon_a")
    b = run_suite(base_order, "canon_b")
    report = {"canonical": {}, "shuffles": []}

    va, vb = vector(a), vector(b)
    for cid in base_order:
        if va[cid] != vb[cid]:
            report["canonical"][cid] = {"a": va[cid], "b": vb[cid]}

    for seed in (17, 41, 1234, 90210):
        order = base_order[:]
        random.Random(seed).shuffle(order)
        rows = run_suite(order, f"shuf_{seed}")
        v = vector(rows)
        flips, prev = [], None
        for r in rows:
            if v[r["id"]] != va[r["id"]]:
                flips.append({"case": r["id"], "base": va[r["id"]],
                              "now": v[r["id"]], "after": prev})
            prev = r["id"]          # heuristic attribution, see step 5
        report["shuffles"].append({"seed": seed, "order": order, "flips": flips})

    Path("leaks.json").write_text(json.dumps(report, indent=2))
    dirty = report["canonical"] or any(s["flips"] for s in report["shuffles"])
    sys.exit(1 if dirty else 0)

if __name__ == "__main__":
    main()

The exit code is the point. A gate that prints a warning gets ignored by week three.

4. Read the verdict vector, not the log tail

Logs are noisy. Four hundred lines of compiler output and one ASan report look the same whether a case leaked state or genuinely regressed.

The vector is small and diffable. On a clean suite, leaks.json carries empty objects and four empty flip lists. On a dirty one it names the case, the canonical verdict, the shuffled verdict, and the case executed immediately before it.

The after field is a lead, not a verdict. A leak can travel through the filesystem or the environment and surface many cases later — the SSE object in step 1 survives three unrelated cases before case 17 consumes it.

5. Confirm by isolation, then quarantine

Attribution only becomes evidence when the pair reproduces in isolation. Run the two cases alone, in the suspected order, with the same scratch root. If the flip returns, the pair is real.

Three artefacts come out of confirmation:

A minimal reproducer — two case IDs and a seed, added as a permanent assertion in tests/test_order.sh.
A quarantine entry in leaks.json with a note pointing at the shared resource: build/vec.o, port 8080, ASAN_OPTIONS, /tmp/eval_cache.
A fix on the offending side, not the victim. Case 12 should build out-of-tree; case 17 should not be silently patched to clean the world first.

Quarantine means the entry stays visible and the gate keeps failing until the fix lands. Deleting the case removes the signal and restores the false green.

6. Shuffling is not free, and that changes the budget

A four-shuffle matrix on a twenty-case suite is five full suite runs. On a shared build machine that cost lands on wall clock and disk, not just CPU.

This is where free model access changes the triage loop rather than the verdict. Confirmed flips are rare; most shuffled runs are clean and never need the expensive model. Re-grading triage runs with a free-tier model through MonkeyCode keeps the matrix affordable, and the free server option gives the harness a persistent box for the seed list, the scratch root, and leaks.json between runs. The strong model is then reserved for the few flips that survive isolation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat about shared boxes: a free server is also a shared-resource magnifier. Ports, /tmp, and disk quotas are the most common leak carriers in exactly this environment, which makes the gate more valuable there, not less.

7. Decision table for the flip report

Observation	Likely cause	Action
Flip only under seeded shuffles, stable in both canonical runs	Filesystem, port, or env leak between cases	Confirm pair in isolation; fix the producing case
Flip between `canon_a` and `canon_b`, same order	Case nondeterminism or unseeded model output	Pin seed and flags; re-check the grader with a canary
Case passes alone, fails after any neighbour	Environment mutation by whoever ran first	Diff `env` and `cwd` before and after the neighbour
Whole suite flips in one run	Harness-level state: disk full, port held, stale scratch root	Reset the scratch root once per run, never per case
Flip appears only in `-O2` builds	Optimization-sensitive undefined behaviour	Route it to an opt-split gate, not this one

8. Limits, and who should skip this

The gate finds order coupling. It says nothing about whether a grader is correct, whether golden cases cover the patched line, or whether the model chose a sound fix.

Skip it when every case already runs in a fresh container or VM — isolation removes the leak class by construction, and the shuffle matrix is pure cost. Skip it for suites so slow that five runs exceed the CI budget; sample ordered pairs instead of full shuffles. Skip it below roughly ten cases, where a human reading the manifest beats a matrix.

Adopt it when cases share a workspace, a port, a cache, or an environment — which is to say, on most real C++ patch suites running on one machine.

A suite that only passes in one order is not a suite. It is a script with a fixed cast, and the patch you are grading is not the thing deciding the outcome. Commit the seed list alongside the cases, so the next person sees the same flips you did.

Flaky Is Not Failed: A Repeat-k Triage Loop for AI-Written C++ Patches

Finley Li — Mon, 19 Oct 2026 08:17:05 +0000

Six golden cases turn red in the same CI run. The team spends two days reverting a patch that was correct, and the single case that had genuinely regressed gets fixed last, almost by accident.

That scenario is a composite — the timeline is illustrative, the pattern is not. Any harness that grades model-written C++ against a fixed set of golden cases will eventually produce a run where the same case disagrees with itself. Two things follow. A two-state pass/fail result cannot describe that world, and a harness without a vocabulary for "I do not know" will convert infrastructure noise into engineering decisions.

The artifacts below are templates. They target Python 3.11 and a CMake/CTest tree, and they were not executed for this article, so treat paths and the runner call as starting points.

Step 1 — Make the result three-state

A golden case can fail for reasons that have nothing to do with the patch: a rate-limited model call, a build that timed out on a shared box, a runner that lost its cache. Grading those as fail is how noise becomes a revert. The runner below records infra_unknown as a distinct outcome and refuses to reason about it later.

# triage.py — repeat each golden case K times, then classify. Template.
import hashlib, os, re, subprocess, tempfile
from dataclasses import dataclass

K = int(os.environ.get("REPEAT_K", "5"))
INFRA_RE = re.compile(r"(429|503|connection reset|build timeout|no space left)", re.I)

@dataclass
class Attempt:
    case: str
    repeat: int
    status: str   # pass | fail | infra_unknown
    exit_code: int
    digest: str
    note: str = ""

def digest(blob: str) -> str:
    # Timing and pid lines track the scheduler, not the behaviour.
    cleaned = "\n".join(
        line for line in blob.splitlines()
        if not re.search(r"(total time|pid=|elapsed)", line, re.I)
    )
    return hashlib.sha256(cleaned.encode()).hexdigest()[:12]

Step 2 — Reuse the case, not the process

Every repeat must start from a clean state. The environment pinning in this runner is deliberate: LC_ALL=C and TZ=UTC remove two classic sources of "it passed on the other machine", and a per-repeat TMPDIR keeps stale artifacts from leaking across attempts.

def run_case(case: str, repeat: int) -> Attempt:
    with tempfile.TemporaryDirectory() as td:
        proc = subprocess.run(
            ["ctest", "--test-dir", "build", "-R", f"^{case}$",
             "--output-on-failure"],
            capture_output=True, text=True, timeout=900,
            env={**os.environ, "LC_ALL": "C", "TZ": "UTC",
                 "REPEAT": str(repeat), "TMPDIR": td},
        )
    blob = proc.stdout + proc.stderr
    if INFRA_RE.search(blob):
        return Attempt(case, repeat, "infra_unknown",
                       proc.returncode, "", blob[-400:])
    return Attempt(case, repeat,
                   "pass" if proc.returncode == 0 else "fail",
                   proc.returncode, digest(blob))

Step 3 — Classify into four verdicts

The interesting bucket is not flaky. It is stable_pass_varying: a case that passes every single repeat while printing two different digests. That case is a silent regression waiting for a slightly different machine, and it is invisible to any harness that only reads exit codes.

def classify(attempts: list[Attempt]) -> dict:
    graded = [a for a in attempts if a.status != "infra_unknown"]
    infra = len(attempts) - len(graded)
    if infra * 2 > len(attempts):
        return {"verdict": "inconclusive", "infra": infra}
    passes = sum(a.status == "pass" for a in graded)
    digests = len({a.digest for a in graded if a.status == "pass"})
    if passes == 0:
        verdict = "stable_fail"
    elif passes < len(graded):
        verdict = "flaky"
    elif digests > 1:
        verdict = "stable_pass_varying"
    else:
        verdict = "stable_pass"
    return {"verdict": verdict, "infra": infra,
            "passes": passes, "digests": digests}

Step 4 — Bind each verdict to a CI action

A verdict that does not change what CI does is a log line, not a gate.

verdict	CI action
`stable_pass`	green
`stable_pass_varying`	yellow: normalise the output before it becomes a red
`flaky`	red on `main`, quarantineable on a branch
`stable_fail`	red: the regression the harness exists to find
`inconclusive`	no grade at all; re-run the sweep later

That last row carries the most weight. Against a shared model endpoint, inconclusive will be common, and a harness that turns it into a failure teaches reviewers to ignore failures.

Step 5 — Quarantine with an expiry date

Unstable cases need somewhere to live that is not /dev/null and not a permanently ignored test.

# quarantine.yaml
- case: containers/unordered_dedup
  reason: iteration order leaks into stdout digest
  owner: finley
  opened: 2026-10-19
  expires: 2026-11-02

# gate.py — an expired quarantine is a build failure, not a reminder
import datetime as dt, sys, yaml

today = dt.date.today()
bad = [q for q in yaml.safe_load(open("quarantine.yaml"))
       if dt.date.fromisoformat(q["expires"]) < today]
if bad:
    sys.exit("expired quarantine: " + ", ".join(q["case"] for q in bad))

Step 6 — Treat the source of the nondeterminism

Repeat-k is triage, not treatment. Three C++ changes remove most of the churn seen in practice.

// 1. Order leaks into output: per-process hashing, different bytes each run.
std::unordered_set<std::string> seen;     // digest churns
std::set<std::string>           seen_ok;  // stable bytes, assertable

// 2. Wall clock: inject it, then assert on it.
struct Clock {
    virtual ~Clock() = default;
    virtual std::time_t now() const = 0;
};
struct FixedClock final : Clock {
    std::time_t t;
    std::time_t now() const override { return t; }
};

// 3. Worker count: 3 threads locally, 32 in CI, different interleavings.
const unsigned workers =
    std::min(4u, std::max(1u, std::thread::hardware_concurrency()));

Add -fsanitize=address,undefined to the triage build rather than the shipping build. Uninitialised reads and signed overflow produce exactly the intermittent failures that repeat-k flags, and a sanitizer report names the bug instead of leaving a pass-rate puzzle. setarch -R is also worth a try when a crash reproduces under one address layout only.

What k buys, and what it does not

k	model calls per case	catches a ~50% flake	catches a ~10% flake
2	2	often	rarely
5	5	~94%	~41%
10	10	~99.9%	~65%

The arithmetic is the binomial one: a 10% flake survives ten repeats with probability 0.9^10 ≈ 0.35. So repeat-k is a coarse filter, and it is honest to say so before anyone wires k=20 into every pull request. Rare nondeterminism is caught better by digest divergence on green runs and by a nightly sweep, not by raising k in the merge path.

Where free model access and a free server fit

A k=10 sweep over 40 golden cases is 400 model calls, and on a metered endpoint that arithmetic appears on an invoice. MonkeyCode's free model access and free server option make such a sweep affordable for a side project or a first pass on a feature branch, which is the cheapest moment to discover that six of your cases were never stable.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two constraints shape how the tool is used here. Free capacity is shared and rate-limited, so the INFRA_RE branch in Step 1 is not decoration — it is the point of the three-state design. The free tier also belongs in the triage loop, not the release gate, because a merge decision should never depend on capacity someone else controls. The offer itself (the operator states 10M free tokens plus a free server option) is worth confirming on the project's own page before planning around it.

To see the classifier fire before wiring it into CI, point it at the two cases you already suspect and spend the first sweep on a scratch branch.

Who should skip this

Projects whose evals are already deterministic: fixed seeds, ordered containers, injected clocks, no concurrency. Repeat-k would buy nothing there.
Teams that cannot store per-attempt records. Without digests and repeat indices, the classifier has no input and no audit trail.
Anyone planning k=20 on every pull request. The wall time is real, and nightly is the honest cadence.
Harnesses that grade prose rather than behaviour. Output digests are meaningful when the program prints bytes, not when a model writes a paragraph.

The habit worth keeping is small: a red run should produce a verdict with a name, and flaky and inconclusive are names. Without them every red run looks identical, and the team slowly learns to treat all of them as noise — including the one that was real.

Model Swap or Grader Bug? A Replay-First Eval Harness for AI C++ Patches

Finley Li — Sat, 17 Oct 2026 06:02:53 +0000

A patch queue goes red over a weekend. The pass rate falls, nobody touched the compiler, and two people are already rewriting prompts to win the metric back. The real change was one line in a grader that stopped trimming trailing whitespace from program output.

That is not a model problem. It is an attribution problem. An eval harness that only reports a pass rate cannot tell you which of its three inputs moved: the prompt, the fixture, or the grader. The workflow below fixes that by making grading replayable, so every failure gets sorted into a bucket before anyone edits a prompt.

Three inputs, three hashes

An eval case has a prompt, a fixture (the code under test), and a grader. Each one can change independently. If the harness stores only a verdict, all three changes collapse into the same red cell in a spreadsheet.

The fix is boring and effective: hash all three inputs at record time, and hash them again at grading time. A verdict that changes with identical hashes is a model-side event. A verdict that changes with identical model output is a grader-side event. Anything else is a case edit that has to be reviewed like code.

case_id            = split-range-001
prompt_hash        = 8a3f...   (sha256 of prompt file contents)
fixture_hash       = 1c90...   (sha256 of the .cpp file)
grader_hash        = e41d...   (sha256 of the grading script)
output_hash        = 77b2...   (sha256 of the recorded completion)

Golden cases small enough to read

A case that needs a paragraph of explanation to justify its expected output will not survive a reviewer's attention span. Keep each file under a screen. One prompt, one fixture, one expected observable.

{
  "id": "split-range-001",
  "prompt_file": "prompts/fix_split_range.md",
  "fixture": "fixtures/split_range.cpp",
  "compile_flags": ["-std=c++20", "-O2", "-Wall", "-Werror"],
  "stdin": "0 1 2 3 4\n",
  "expect": { "exit_code": 0, "stdout": "0 1 2 3\n", "stderr_empty": true }
}

Note what is not in that file: no prose rubric, no model name, no temperature. Those belong to the run configuration, because they are exactly the things that get swapped without anyone recording it.

Step 1: record once, replay forever

Recording stores the completion alongside its input hashes. Replay grades the stored completion offline, with no network call. The first run of a new case is the only one that spends anything.

# evals/harness.py — reference implementation, Python 3.11+
import hashlib, json, pathlib, subprocess

ROOT = pathlib.Path(__file__).parent

def digest(*parts: str) -> str:
    h = hashlib.sha256()
    for p in parts:
        h.update(p.encode())
        h.update(b"\x00")
    return h.hexdigest()[:16]

def input_hash(case: dict) -> str:
    prompt = (ROOT / case["prompt_file"]).read_text()
    fixture = (ROOT / case["fixture"]).read_text()
    grader = (ROOT / "graders" / "io_match.py").read_text()
    return digest(prompt, fixture, grader)

def grade(case: dict, completion: str) -> dict:
    src = (ROOT / case["fixture"]).read_text() + "\n" + completion
    build = ROOT / "build" / case["id"]
    build.mkdir(parents=True, exist_ok=True)
    (build / "main.cpp").write_text(src)
    cc = subprocess.run(
        ["g++", *case["compile_flags"], "main.cpp", "-o", "a.out"],
        cwd=build, capture_output=True, text=True,
    )
    if cc.returncode != 0:
        return {"compiles": False, "stderr": cc.stderr[-2000:]}
    run = subprocess.run(
        ["./a.out"], cwd=build, input=case.get("stdin", ""),
        capture_output=True, text=True, timeout=10,
    )
    exp = case["expect"]
    ok = (
        run.returncode == exp["exit_code"]
        and run.stdout == exp["stdout"]
        and (not exp.get("stderr_empty", True) or run.stderr == "")
    )
    return {"compiles": True, "ok": ok, "stdout": run.stdout}

The compile step is the expensive part, and it is also the part people skip when they grade on stdout alone. A patch that prints the right bytes but fails -Werror is not a pass.

Step 2: split replay from live

Two subcommands, one report format. The replay arm never touches the network; the live arm calls whatever endpoint the run configuration points at.

# no tokens spent; grades recorded completions
python evals/harness.py replay --report out/replay.json

# spends tokens; records new completions into cassettes/
python evals/harness.py live --endpoint "$EVAL_ENDPOINT" --report out/live.json

# attribute every difference
python evals/harness.py diff out/replay.json out/live.json

Run replay on every pull request. Run live on a schedule, or when a model or prompt change is proposed. That ratio matters: replay is deterministic, cheap, and suitable for a hard gate; live is a sampling instrument and should never block a merge on a single flip.

Step 3: where free capacity actually helps

MonkeyCode's free model access and free server option are the two pieces that make the live arm affordable for an individual maintainer. The value is not the free tier itself; it is that repeated grading runs become cheap enough to do often. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest constraints. First, quota and lifetime terms on a free tier change, so read the project's current page instead of trusting a blog post — including this one. Second, a free workspace is the wrong place to store anything you would not publish, so keep cassettes free of proprietary fixtures.

The harness itself is provider-shaped, not provider-bound. --endpoint is a shell variable for a reason: swap it and the diff logic keeps working.

Step 4: classify the drift, do not just count failures

This is the whole point. One function, three buckets, checked in order.

def classify(case, recorded, replayed, live) -> str:
    if input_hash(case) != recorded["input_hash"]:
        return "CASE_DRIFT"      # prompt, fixture, or grader was edited
    if replayed["ok"] != recorded["ok"]:
        return "GRADER_DRIFT"    # same inputs, same output, different verdict
    if live["ok"] != replayed["ok"]:
        return "MODEL_DRIFT"     # new completion, new verdict
    return "STABLE"

Order is not cosmetic. If the case hash moved, everything downstream is uninterpretable, so it gets reported first and the run is marked "not comparable" rather than "failed".

The resulting report reads like a triage list:

split-range-001   STABLE
parse-args-004    GRADER_DRIFT   verdict ok->fail, output identical
wide-int-002      MODEL_DRIFT    verdict fail->ok
sort-stable-007   CASE_DRIFT     fixture changed; run not comparable

GRADER_DRIFT on a passing case is the most valuable line in the file. It means a grader edit silently reclassified work that nobody asked to re-evaluate.

Step 5: gate on transitions, not totals

A CI gate that fails on "pass rate below 90%" will block merges for reasons unrelated to the merge. Gate on the classification instead.

- run: python evals/harness.py replay --report out/replay.json
- run: python evals/harness.py diff out/replay.json out/last-green.json --fail-on GRADER_DRIFT

MODEL_DRIFT on a live run should open an issue, not fail a build. CASE_DRIFT should require a reviewer to look at the fixture diff. Only grader-side movement deserves a hard stop, because it corrupts every future comparison until someone explains it.

One caveat on sensitivity: this harness assumes your graders can already fail when they should. Proving that is a separate problem with its own fixtures, and it is worth solving before you trust any bucket above.

When to replay and when to go live

Question	Replay	Live
Did a grader edit change verdicts?	yes	no
Did a new model change outputs?	no	yes
Is this cheap enough for every PR?	yes	no
Is this deterministic?	yes	no
Can it catch a fixture typo?	yes	yes

Limitations and who should skip this

Cassettes rot. A recorded completion is only valid for the fixture and flags it was recorded against, so a fixture edit invalidates it by design — which is good, but it means cassettes are not a long-term archive.

Cases that depend on wall-clock time, thread scheduling, or unseeded randomness will produce MODEL_DRIFT noise that is really nondeterminism in the test. Fix the fixture first or drop the case.

Finally, this harness measures individual case flips, not statistical significance. A team trying to decide whether a model is "better" by a few percentage points needs many samples and a proper interval; a single replay run will happily report a difference that is noise. And anyone using this as a security or correctness sign-off for shipped code should stop here — golden cases guard a pipeline, they do not certify a binary.

If the goal is to keep the live arm running without owning a runner, the replay path is the place to start: it costs nothing, it catches the failure mode that burns the most time, and it works before any endpoint is configured.

Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

Finley Li — Fri, 16 Oct 2026 05:00:35 +0000

Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

A pipeline starts scoring C++ patches with a model-backed rubric. Diffs that change printed precision, drop a noexcept guarantee, or leak a file descriptor now receive a 1-10 score before a human opens them. For a few weeks the dashboard looks reasonable, reviewers trust the number, and nobody asks whether the grader still separates good patches from bad ones. Then a prompt edit lands, the upstream model is swapped, and every patch scores 8. Nothing crashes. The grader simply stops discriminating, and the pipeline keeps reporting green.

That failure mode is independent of which model you call. A rubric grader is a program with an untested dependency, and it deserves the treatment the code under it already gets: golden cases, a canary set, a hard gate, and a ledger. The harness below is standard-library Python, a few JSON canary files, and one cron line. It is deliberately boring, because the interesting part is that the grader fails loudly instead of silently.

Layer 0 thinks, layer 1 judges

Split grading into two layers before writing any code. The deterministic layer answers questions with one correct answer: does the patched tree compile, does the golden case exit zero, does stdout match after a fixed locale and precision. The rubric layer answers questions with a range of defensible answers: is the naming consistent with the header it implements, does the commit message describe the actual change, does the new overload respect the documented contract.

Most silent regressions live on the boundary. A grader that is only deterministic cannot see prose contracts. A grader that is only a rubric cannot see a changed std::setprecision. Layer 0 gates the build, layer 1 explains the patch, and each layer needs its own canaries.

A canary here means a labeled example with a fixed expected answer that is committed to the repository. It is not a test of the patch. It is a test of the test.

Step 1 - Freeze golden cases for the deterministic layer

Start with one golden case per documented output contract. Keep the case small enough that a failure points at one behavior, and pin everything that could drift: locale, precision, and the toolchain described in your own build docs.

// golden/report_precision.cpp - contract: ratio printed with 3 decimals
#include <iostream>
#include <iomanip>

int main() {
    const double ratio = 2.0 / 3.0;
    std::cout << std::setprecision(3) << std::fixed << ratio << std::endl;
    return 0;
}

Run it through the same command the eval batch uses, so the grader and the patch pipeline cannot disagree about flags:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target report_precision
./build/report_precision > out.txt
diff -u golden/report_precision.expected out.txt

The diff is the grader. If it passes, the patch is not yet good, but it is at least not obviously broken.

Step 2 - Commit canaries in both directions

A canary set that only contains bad patches rewards a paranoid grader. A canary set that only contains good patches rewards a lenient one. Include both, and store each patch as its own file so the JSON stays readable and diffs stay reviewable.

eval/
  canaries/
    precision-3-to-2.json
    honest-refactor.json
    tie-case-review.json
    diffs/
      precision-3-to-2.patch
      honest-refactor.patch
      tie-case-review.patch

Each canary declares what it expects and why. The note field is the part humans actually maintain.

{
  "id": "precision-3-to-2",
  "kind": "rubric",
  "direction": "must-fail",
  "patch_path": "diffs/precision-3-to-2.patch",
  "expected": 3,
  "note": "Breaks the output contract in docs/report.md. Any score above 5 means the rubric is not reading the contract."
}

{
  "id": "honest-refactor",
  "kind": "rubric",
  "direction": "should-pass",
  "patch_path": "diffs/honest-refactor.patch",
  "expected": 8,
  "note": "Pure rename plus a comment fix, output byte-identical. Below 6 means the rubric punishes chore diffs."
}

{
  "id": "precision-3-to-2-build",
  "kind": "build",
  "patch_path": "diffs/precision-3-to-2.patch",
  "golden_cmd": "./build/report_precision | diff -u golden/report_precision.expected -",
  "note": "Must exit non-zero. If this passes, the deterministic layer is not running."
}

Keep the count honest. Twenty labeled canaries drawn from real review comments are worth more than two hundred generated ones, because generated canaries encode the same assumptions as the rubric prompt.

Step 3 - Measure agreement before trusting scores

Agreement is the only cheap health signal a rubric grader has. Exact match is strict; within-one match tolerates the ordinary noise of a 1-10 scale. Both numbers belong in the gate.

#!/usr/bin/env python3
# grade_gate.py - run canaries before trusting rubric scores.
import json, os, pathlib, subprocess, sys

CANARY_DIR = pathlib.Path(os.environ.get('CANARY_DIR', 'eval/canaries'))
LEDGER = pathlib.Path(os.environ.get('GRADER_LEDGER', 'eval/grader_ledger.jsonl'))
RUBRIC_ID = os.environ.get('GRADER_VERSION', 'unversioned')


def load_canaries():
    for path in sorted(CANARY_DIR.glob('*.json')):
        canary = json.loads(path.read_text())
        patch = (CANARY_DIR / canary['patch_path']).resolve()
        canary['diff_text'] = patch.read_text()
        yield canary


def call_model(prompt):
    # Transport layer. Plug in whichever client your runtime provides.
    # This function is untrusted by design; that is why canaries exist.
    raise NotImplementedError('wire your client here')


def rubric_score(diff_text, rubric):
    reply = call_model(rubric + '\n\nDIFF:\n' + diff_text)
    return json.loads(reply)

Now the comparison itself, plus the gate. The thresholds are policy defaults, not measured results: a canary set of forty concrete review-derived cases can usually carry a higher bar than one of twelve fuzzy ones. Tune them once, write the number in the repo, and treat any change to it as a reviewable commit.

def check_rubric_canaries(canaries, rubric, min_exact=0.80, min_within_one=0.95):
    exact = within_one = 0
    misses = []
    for canary in canaries:
        if canary.get('kind') != 'rubric':
            continue
        expected = canary['expected']
        got = rubric_score(canary['diff_text'], rubric)['score']
        if got == expected:
            exact += 1
        else:
            if abs(got - expected) == 1:
                within_one += 1
            misses.append({'id': canary['id'], 'expected': expected,
                           'got': got, 'delta': got - expected})
    total = sum(1 for c in canaries if c.get('kind') == 'rubric') or 1
    rates = {'exact': exact / total, 'within_one': (exact + within_one) / total}
    ok = rates['exact'] >= min_exact and rates['within_one'] >= min_within_one
    return ok, rates, misses

Build canaries run the golden command instead of a model, and check the opposite thing: they must fail.

def check_build_canaries(canaries, workdir):
    results = []
    for canary in canaries:
        if canary.get('kind') != 'build':
            continue
        subprocess.run(['git', 'apply', canary['patch_path']], cwd=workdir, check=True)
        proc = subprocess.run(canary['golden_cmd'], cwd=workdir, shell=True)
        subprocess.run(['git', 'restore', '.'], cwd=workdir, check=True)
        results.append({'id': canary['id'], 'exit': proc.returncode,
                        'ok': proc.returncode != 0})
    return results

Step 4 - Keep a grader ledger next to the token ledger

A score is only meaningful with the grader revision and the model identity attached. Write one JSON line per gate run, including the token count, so drift and cost are visible in the same file.

def log_run(rates, misses, model_id, tokens):
    record = {'grader': RUBRIC_ID, 'model': model_id, 'rates': rates,
              'misses': misses, 'tokens': tokens}
    with LEDGER.open('a') as fh:
        print(json.dumps(record), file=fh)

if __name__ == '__main__':
    rubric = pathlib.Path(sys.argv[1]).read_text()
    canaries = list(load_canaries())
    ok, rates, misses = check_rubric_canaries(canaries, rubric)
    build = check_build_canaries(canaries, '.')
    log_run(rates, misses, os.environ.get('MODEL_ID', 'unknown'),
            int(os.environ.get('TOKENS_USED', '0')))
    healthy = ok and all(r['ok'] for r in build)
    print(json.dumps({'healthy': healthy, 'rates': rates, 'misses': misses,
                      'build': build}, indent=2))
    sys.exit(0 if healthy else 1)

The exit code is the whole interface. A healthy gate returns 0 and the eval batch proceeds; an unhealthy gate returns 1 and the batch stops before it spends anything on scoring.

Step 5 - Run the gate where nobody has to remember it

A gate that depends on a human typing a command is a gate that will be skipped during the week it matters most. Two things make the scheduled version practical: a machine that stays on, and model access that does not require a per-run decision.

MonkeyCode, described by its maintainers as an open-source project, offers free model access and a free server option, with the free tier described as including 10M tokens. That combination is enough to host a weekly canary run plus the rubric calls it makes, without reserving a dedicated box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# cron: every Monday 06:17, on the free server option
17 6 * * 1 cd /srv/eval && python3 grade_gate.py rubrics/patch_v3.md >> logs/canary.log 2>&1

Read the ledger rather than the estimate. The claim worth trusting is not a predicted token count; it is the tokens field on last week's line, compared with the line before it.

Which layer owns which signal

Signal to grade	Layer 0 (deterministic)	Layer 1 (rubric)
Compiles, links, exits zero	Yes	No
stdout under fixed locale and precision	Yes	No
Wall-clock or allocation budget	Yes	No
Naming consistent with the header	Partly, via lint rules	Yes
Prose output contract in docs	No	Yes
Commit message matches the change	No	Yes
Whether the change is worth merging	No	Advisory only, never binding

If a row can be pushed into layer 0, push it. Every signal moved out of the rubric layer removes a source of drift, and the rubric layer keeps only the judgments a compiler cannot express.

Limitations and who should skip this

Canaries are a sample, and agreement is not correctness. If the labels are wrong, the gate certifies a wrong grader with a confident number. Re-label a handful of canaries every few months, especially the ones that never fail.

Rubric scores drift when the model or the prompt changes, and only a scheduled run of the ledger detects that. Single-run scores stay noisy even with a fixed prompt, so treat a one-point swing as a signal to check the ledger, not as a verdict on the patch.

The quota and server claims above are operator descriptions of the offer, not measurements taken for this article. Verify current terms before wiring a paid-critical pipeline to a free tier, and keep the gate able to run against a different client.

Skip this approach if you have fewer than roughly twenty real diffs to label, because reading the patches is then cheaper than grading them. Skip it for safety-critical, licensing, or compliance judgments, which need deterministic evidence and a named reviewer. Skip it if diff text cannot leave your network, unless every layer runs locally. And skip the rubric layer entirely if your grading budget exceeds the cost of the human review it is meant to save.

The harness is small on purpose. Golden cases, canaries in both directions, one agreement gate, one ledger. Start with a single canary that should fail and confirm the gate returns non-zero; if it returns zero, the grader was never watching.

Mutants Before Merges: A Two-Sided Sensitivity Gate for C++ Eval Harnesses

Finley Li — Thu, 15 Oct 2026 11:15:02 +0000

An illustrative trace. A 40-line C++ patch clears the compiler, passes 41 golden cases and merges. Two weeks later a downstream consumer reports wrong scaling on a load path none of the goldens touch. Nothing in the pipeline crashed. The harness was blind — every test passed because no assertion ever inspected the broken path.

That trace is a composite pattern, not a report on a named team. The defect class behind it is measurable all the same. This article builds a gate that asks whether a C++ eval harness is capable of failing, and that refuses to grade a patch when the answer is no.

Golden cases measure agreement, not sensitivity

Golden cases answer one question: does current output match recorded output? That is agreement. It says nothing about whether the harness would notice if the behavior underneath changed.

Two properties matter. Sensitivity is the probability that the harness fails when behavior breaks. Specificity is the probability that it passes when behavior is correct. A harness that fails on every run has perfect sensitivity and no usefulness; a harness that passes everything is the failure mode in the opening trace.

More golden cases raise agreement coverage. They raise sensitivity only if the new cases touch code that was previously unasserted, and stdout diffs frequently do not. A case that checks a final total can pass while an intermediate bound check was deleted.

The gate below measures both properties before a patch is graded, using mutants as calibrated faults.

Step 1 — Freeze the harness contract

A gate needs a fixed pass/fail definition, or the mutants measure noise. Pin the build type, the runner and the wall-clock ceiling in one place, and treat any change to them as a change to the gate itself.

cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j2
ctest --test-dir build --output-on-failure   # the bench command

Everything downstream calls exactly these commands. If this patch class needs sanitizers, keep a second profile and run the gate twice, one ledger per profile.

Step 2 — Run a flake pre-pass

Kill rates computed on a flaky harness are fiction. Before generating any mutant, run the bench three times on the untouched tree and require identical results.

for i in 1 2 3; do
  ctest --test-dir build --output-on-failure >/dev/null 2>&1 || echo "flake run $i"
done

Any deviation stops the gate. Ordering dependence, timing assumptions and shared temp files surface here far more cheaply than inside a mutant run.

Step 3 — Build a two-class mutant seed set

Mutants are small deliberate faults. They fall into two classes with opposite expected results.

Class	Expectation	Purpose	Examples
sensitivity	harness must FAIL	proves the harness can see a break	`<=` to `<`, sign flip, deleted early return, dropped `const`, rounded float literal
inert	harness must PASS	proves the harness is not failing on trivia	reformatted comment, reordered `#include`, renamed local, added dead branch

Hand-write ten sensitivity mutants and three inert ones. They cost nothing and they encode what the team actually cares about. Model-drafted mutants extend the set afterwards; the note further down explains why they are candidates rather than ground truth.

[
  {"id": "M01", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "if (n <= cap)", "repl": "if (n < cap)"},
  {"id": "M02", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "return a - b;", "repl": "return b - a;"},
  {"id": "M03", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "  const auto lim = limit;", "repl": "  auto lim = limit;"},
  {"id": "I01", "class": "inert", "file": "src/scale.cpp",
   "find": "// clamp input", "repl": "// clamp input to cap"}
]

Each anchor must match exactly once. An ambiguous anchor is a skipped mutant, never a silent edit. That rule keeps the ledger honest when a patch moves code around.

Step 4 — Drive the gate

The driver below is an illustrative reference implementation, not code executed for this article. It restores each source file in a finally block, so a crash cannot leave a mutant in the tree.

#!/usr/bin/env python3
# mutgate.py - two-sided sensitivity gate for a C++ eval harness
import json, pathlib, subprocess, sys, time

ROOT = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
KILL_FLOOR = float(sys.argv[2]) if len(sys.argv) > 2 else 0.60
BUILD = "cmake --build build -j2"
BENCH = "ctest --test-dir build --output-on-failure"
TIMEOUT = 90

def run(cmd, timeout=TIMEOUT):
    t0 = time.time()
    try:
        p = subprocess.run(cmd, cwd=ROOT, shell=True, capture_output=True,
                           text=True, timeout=timeout)
        return p.returncode, (p.stdout + p.stderr)[-1200:], round(time.time() - t0, 1)
    except subprocess.TimeoutExpired:
        return 124, "wall clock exceeded", timeout

def observe():
    rc, log, secs = run(BUILD)
    if rc != 0:
        return "BUILD_ERROR", log, secs
    rc, log, secs2 = run(BENCH)
    return ("FAIL" if rc != 0 else "PASS"), log, secs + secs2

def main():
    mutants = json.loads((ROOT / "mutants.json").read_text())
    ledger = {"clean_control": None, "mutants": [], "verdict": None}

    state, log, secs = observe()
    ledger["clean_control"] = {"state": state, "secs": secs}
    if state != "PASS":
        ledger["verdict"] = "REJECT_HARNESS"
        (ROOT / "mutgate-ledger.json").write_text(json.dumps(ledger, indent=2))
        print("REJECT_HARNESS: clean tree does not pass")
        return

    for m in mutants:
        src = ROOT / m["file"]
        original = src.read_text()
        hits = original.count(m["find"])
        if hits != 1:
            ledger["mutants"].append({"id": m["id"], "class": m["class"],
                                      "state": "SKIPPED", "note": f"anchor x{hits}"})
            continue
        src.write_text(original.replace(m["find"], m["repl"], 1))
        try:
            state, log, secs = observe()
        finally:
            src.write_text(original)
        ledger["mutants"].append({"id": m["id"], "class": m["class"],
                                  "state": state, "secs": secs})

    sens = [x for x in ledger["mutants"]
            if x["class"] == "sensitivity" and x["state"] != "BUILD_ERROR"]
    killed = [x for x in sens if x["state"] == "FAIL"]
    inert_noise = [x for x in ledger["mutants"]
                   if x["class"] == "inert" and x["state"] == "FAIL"]
    build_errors = [x for x in ledger["mutants"] if x["state"] == "BUILD_ERROR"]
    rate = len(killed) / len(sens) if sens else 0.0

    if build_errors:
        ledger["verdict"] = "REVIEW_BUILD_ANCHORS"
    elif inert_noise:
        ledger["verdict"] = "REVIEW_GRADER_NOISE"
    elif rate >= KILL_FLOOR:
        ledger["verdict"] = "ACCEPT"
    else:
        ledger["verdict"] = "UNKNOWN_HARNESS_BLIND"

    (ROOT / "mutgate-ledger.json").write_text(json.dumps(ledger, indent=2))
    print(json.dumps({"kill_rate": round(rate, 2), "verdict": ledger["verdict"]}))

if __name__ == "__main__":
    main()

Run it as python3 mutgate.py . 0.60. The threshold is a policy choice, not a discovered constant. Start at 0.6, raise it for safety-critical patches, and record the value beside the ledger.

Step 5 — Read the verdict table, not just the number

Clean control	Inert mutants	Kill rate	Verdict
PASS	all PASS	at or above floor	ACCEPT - the harness earns the right to grade this patch
PASS	all PASS	below floor	UNKNOWN - survivors mark unasserted paths; write cases instead of merging on green
PASS	any FAIL	any	REVIEW - harness fails on harmless change; fix specificity first
PASS	-	-	REVIEW - an anchor or mutation caused a build error; kills are not counted
FAIL	-	-	REJECT - the harness itself is broken; no patch can be graded

UNKNOWN is the useful state. It converts a false green into an explicit work item, and it costs one line in the review checklist: survivors list, case inventory, owner.

Step 6 — Turn survivors into cases

Every sensitivity mutant that survives is a concrete assertion the harness does not have. Map it to the smallest case that would fail: an input where n == cap, a negative operand, a value that must not be rounded. Group survivors by the function they touch and open one case-writing task per group rather than one per mutant.

Keep the ledger as a per-patch artifact. Two runs of the same patch with different verdicts usually mean the mutant set changed, not the code.

Where disposable mutants and a free runner fit

A mutation gate needs a lot of throwaway work: dozens of candidate mutants per patch, plus a runner that rebuilds each time in a sandbox. Both halves tolerate imperfection, which changes the economics.

Model-drafted mutants are cheap candidates, not ground truth. A model produces edits that look plausible, which correlates with the bugs a reviewer would miss, but the distribution is biased toward what the model finds natural. It will miss the boundary case a maintainer has been burned by. The workflow that holds up: hand-write the sensitivity and inert seed set, ask a model for thirty more candidates per patch, then keep only those whose anchors resolve and whose class is unambiguous.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states the current offering includes free model access and a free server option, with a free token allowance advertised at the time of writing as 10 million tokens. Allowances, hardware and terms change, and the project is the only authority on its own limits, so verify the current terms before depending on them.

The free server option fits the runner half because mutation runs are disposable by construction: a container that rebuilds a mutant, runs the bench and is discarded holds no state worth keeping. Keep the merge-blocking golden run in whatever CI the team already trusts, and spend the disposable capacity on the sensitivity suite.

Limitations and who should not use this

Kill rate is a heuristic, not a probability of catching real defects. Mutants and real bugs are different distributions, and a high kill rate can coexist with an unasserted path that no mutant happened to hit.

BUILD_ERROR mutants are excluded from both numerator and denominator on purpose. Counting a compile failure as a kill inflates the rate and hides the case where a refactor broke the anchors.

Mutant runs multiply CI time by the size of the set. With 40 sensitivity mutants and a 30-second bench, the gate costs roughly 20 minutes plus rebuilds. Budget for it, or run it nightly against a sample of recent patches.

Skip this approach if the harness is already a long-running fuzzing or sanitizer campaign, if the codebase needs formal guarantees rather than sampled faults, or if a two-person team has fewer than a dozen golden cases to begin with. In that state, writing cases teaches more than measuring them.

Closing

A harness that cannot fail cannot be trusted when it passes. Build the two-sided gate once, keep the mutant set small and hand-seeded, and read UNKNOWN as a scheduled task rather than a failure. If the runner half is what is blocking you, MonkeyCode's free model access and free server option are a reasonable place to try it on real patches.

Debug Totals Lie at Release: An Opt-Split Gate for C++ Patch Evals

Finley Li — Tue, 13 Oct 2026 00:25:07 +0000

Debug Totals Lie at Release: An Opt-Split Gate for C++ Patch Evals

The failure mode is ordinary, and it hides in a green log. A grader accepts a C++ patch because the debug binary prints 18432. A later release build of the same sources prints 18416. Nobody edited the prompt between those two runs. The optimizer met a signed addition that had already left the representable range.

That split is a silent regression. A single golden print, captured under one flag set, cannot see it.

This note describes an opt-split gate for AI C++ patches. One frozen input tape, one hand-written oracle, and the candidate built at -O0 and -O2. The gate fails when the oracle stays stable and the candidate does not. A prompt hash pin stops a later draft from changing the test without a recorded edit.

What a single print misses

Generated C++ often looks finished inside a driver that prints one number. The driver and the patch share a translation unit. Warnings are quiet, or they were never enabled. The build uses whatever optimization the local default happens to be, which is frequently closer to -O0 than to the flags a package will ship.

Signed overflow, a read of an uninitialized scalar, and a dependency on evaluation order can all survive that setup. A second flag set sometimes keeps the arithmetic inside the range the oracle demands. Sometimes it stops hiding the bug. The opt-split gate does not prove the absence of undefined behavior. It catches the cheaper mismatch: debug output and release output disagree, while a checked oracle prints the same bytes twice.

Files the gate owns

Four inputs define a run, and none of them is a dashboard screenshot. prompt.txt is the only prose the draft step may read. prompt.pin stores the SHA-256 of that file from the last accepted wording change. tape.txt lists signed integers, one token per line, and ends with END.

oracle.cpp is written by a person. candidate.cpp is the patch under test, and it must export the same signature the oracle exports. The contract stays narrow so the grade is about values, not about prose.

// contract.hpp
#pragma once
#include <cstdint>
#include <optional>

std::optional<std::int64_t> fold_checked(const std::int64_t* data, std::size_t n);

A returned integer is the exact sum. std::nullopt means the fold is rejected. The candidate is scored against the oracle decision, not against a golden string sitting in the prompt.

Step 1. Pin the prompt before a draft exists

A changed prompt is a changed test, even when every tape byte is identical. The pin check refuses to grade on drift. It does not try to be clever about whitespace.

python3 - << 'PY'
import hashlib, pathlib, sys
prompt = pathlib.Path("prompt.txt").read_bytes()
digest = hashlib.sha256(prompt).hexdigest()
pin = pathlib.Path("prompt.pin")
if not pin.exists():
    pin.write_text(digest + "\n", encoding="ascii")
    print("pinned", digest)
    sys.exit(0)
expected = pin.read_text(encoding="ascii").strip()
if digest != expected:
    print("prompt pin mismatch")
    print("expected", expected)
    print("actual  ", digest)
    sys.exit(2)
print("prompt pin ok", digest)
PY

Treat a mismatch as a harness failure, not as a patch failure. Update prompt.pin only in a commit that quotes the new wording. Nightly jobs must not rewrite the pin to make themselves green.

Step 2. Freeze the tape outside the driver

The driver reads tape.txt, calls fold_checked, and prints REJECT or a decimal integer. It contains no expected answers. A model that can see the golden line inside the driver will often echo it, and the echo will match at every optimization level.

// tape_driver.cpp
#include "contract.hpp"
#include <fstream>
#include <iostream>
#include <string>
#include <vector>

int main() {
    std::ifstream in("tape.txt");
    std::vector<std::int64_t> data;
    std::string tok;
    while (in >> tok) {
        if (tok == "END") {
            break;
        }
        data.push_back(std::stoll(tok));
    }
    std::optional<std::int64_t> got = fold_checked(data.data(), data.size());
    if (!got) {
        std::cout << "REJECT\n";
        return 0;
    }
    std::cout << *got << "\n";
    return 0;
}

Start with three tapes, and keep them in separate files. One sum fits in 64 bits. One sum crosses the top of the signed 64-bit range. One file contains only END, the empty range.

The samples below are unexecuted fixtures until a local compiler has produced the four binaries. The method does not depend on a hosted runner.

10
20
30
END

9223372036854775807
1
END

Step 3. Build both sides at two levels

Hold the standard, the warning set, and the include path fixed. Change only the optimization flag. Record the compiler identity before the first object file exists, so a later green row can be tied to a toolchain.

mkdir -p out
g++ --version | head -n 1 | tee out/compiler.txt
CXXFLAGS="-std=c++20 -Wall -Wextra -Werror -I."
g++ $CXXFLAGS -O0 -o oracle_o0 oracle.cpp tape_driver.cpp
g++ $CXXFLAGS -O2 -o oracle_o2 oracle.cpp tape_driver.cpp
g++ $CXXFLAGS -O0 -o cand_o0 candidate.cpp tape_driver.cpp
g++ $CXXFLAGS -O2 -o cand_o2 candidate.cpp tape_driver.cpp

A candidate that fails to compile receives a compile grade, and the matrix stops. Partial binaries are not compared. An oracle that fails to compile means the gate is misconfigured. That result is not a patch defect.

The oracle itself should be boring. This implementation is a proposal for the fixture, not a claim that a model emitted it, and not a measured result.

#include "contract.hpp"
#include <cstdint>
#include <limits>
#include <optional>

std::optional<std::int64_t> fold_checked(const std::int64_t* data, std::size_t n) {
    __int128 acc = 0;
    for (std::size_t i = 0; i < n; ++i) {
        acc += data[i];
        if (acc > std::numeric_limits<std::int64_t>::max()) {
            return std::nullopt;
        }
        if (acc < std::numeric_limits<std::int64_t>::min()) {
            return std::nullopt;
        }
    }
    return static_cast<std::int64_t>(acc);
}

Run each binary from a directory that contains only the chosen tape.txt. Capture stdout bytes. Do not trim, and do not pass the text through a shell that rewrites newlines.

./oracle_o0 > out/oracle_o0.txt
./oracle_o2 > out/oracle_o2.txt
./cand_o0 > out/cand_o0.txt
./cand_o2 > out/cand_o2.txt
cmp -s out/oracle_o0.txt out/oracle_o2.txt

Step 4. Score a four-cell matrix

Compare the files with cmp. A missing newline is a mismatch. A numeric parser that forgives spaces will hide the drift this gate exists to catch.

Oracle `-O0` vs `-O2`	Candidate vs oracle at `-O0`	Candidate vs oracle at `-O2`	Grade
differ	any	any	oracle unstable; do not score the patch
match	differ	any	functional miss
match	match	differ	opt-split fail
match	match	match	pass for this tape

The third row is the silent regression. The patch met the contract in the debug build and broke it, or changed it, at -O2. Store the four stdout files beside the prompt hash. A later job that flips only cand_o2 is a regression even when the debug line still matches the earlier example.

Repeat the matrix for every tape. The patch passes the gate only when every tape passes. One opt-split fail fails the run. Do not average rows into a soft score. A mean will bury a release-only break under three comfortable debug matches.

Step 5. Attach a draft host without baking in its limits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The matrix does not need a hosted model. A local editor can drop candidate.cpp into the tree, and the four g++ lines still apply. MonkeyCode enters only when a team wants a draft step or a remote build host. It is an open-source project. Operator materials describe free model access and a free server option. Both are availability claims, so confirm them in the current project docs on the day of the run.

Keep the draft blind. It may read prompt.txt and contract.hpp. It may not read oracle.cpp, the tapes, or earlier stdout. Showing the oracle to the model turns the grade into a transcription exercise, and transcription will pass at both optimization levels for the wrong reason.

Do not hard-code a token allowance, a machine size, or a rental duration into prompt.pin or the grader. Those numbers move, and a stale ceiling will outlive the plan that published it. If the account cannot start a build that day, fail the job closed and leave the previous pin untouched.

The free server, when it is actually available, is a fit for the four compile-and-run lines. The pass/fail rule should stay in a script the reviewer can diff on a laptop. A clean split is local pin check, remote compile, local grade. The host returns stdout files and the compiler version line, and the reviewer's copy of the matrix script assigns the grade.

Teams that already keep prompt.pin can send the frozen prompt to the current free model access and run this opt-split matrix on the free server, after reading that day's limits in the project docs. The grade script should still run unchanged if that host is offline.

Limitations, and who should skip the gate

Agreement across -O0 and -O2 is a screen, not a proof. Undefined behavior can survive both flag sets, and it can disappear under both. Sanitizer builds belong in a separate gate. This matrix also ignores throughput. A patch may pass every tape and still be too expensive to ship, so grade cost with a different fixture.

Skip the approach when the specification allows divergence under fast-math flags, or when the contract is approximate and a tolerance is the real oracle. Skip it when the oracle and the candidate are supposed to reject different edges. That mismatch is a spec argument, not a regression.

Skip it for patches that cannot be built as one extra translation unit, such as a plugin ABI or a generated lexer. The sample driver uses std::stoll and will throw on a corrupt tape. That is a harness bug, and the run should fail before any patch grade is written.

The __int128 accumulator is a GCC and Clang extension, not portable C++. On a toolchain without it, replace the oracle with a checked add written in standard C++. Pin that choice next to the compiler version line. Otherwise two machines will disagree about the oracle and the matrix will blame the patch.

What the log must hold

Each run record needs the prompt hash, the compiler version string, the four flag lines, and the stdout hashes. A green debug line without those fields is how a single total survives review. The release build was never graded, so nobody had a file to open.

The opt-split gate makes that file mandatory. Same tape, two optimization levels, and a human oracle that does not travel inside the prompt. When the release total moves and the oracle does not, the patch does not ship.

One Attempt Must Carry Every Grade: A noexcept Ledger for C++ Patches

Finley Li — Sun, 11 Oct 2026 20:54:13 +0000

Consider a Tuesday review of a small C++ frame parser. The candidate patch prints the same payload length as the golden file, and the happy-path binary exits zero. Reviewers merge it. A downstream caller, already compiled against the old noexcept promise, later aborts on a short frame.

The prints never mentioned exceptions. The green run never fed the parser an empty buffer. That gap is what the ledger below is built to catch.

What a matching printout leaves ungraded

A happy-path fixture checks observable text. It does not check the exception specification a caller already compiled against. Those are different contracts, and a patch can satisfy one while breaking the other.

Dropping noexcept from a header can leave the happy output unchanged. Keeping noexcept on the declaration while throwing from the body can do the same, until a bad input calls std::terminate. Either miss is silent next to a golden text file.

The workflow grades both misses on a single attempt id. It does not claim those grades prove the absence of undefined behavior, data races, or leaks.

Split the contract from the sample

The proposed tree keeps golden files outside the directory a model is allowed to overwrite. Each sample gets a fresh attempt folder. Object files from an earlier sample never leak into the next link.

golden/frame.hpp is the last reviewed declaration. The attempt include file starts as a copy of that header, then receives only the edits the prompt permitted. A comment-only difference is not a failure. A consumer translation unit decides whether the declaration is still non-throwing.

tests/contract_tu.cpp includes the candidate header. tests/happy.cpp and tests/malformed.cpp link the candidate object and print one line. The malformed program is the runtime half of the contract: exit code 0 and the sentinel 0 plus a newline mean a short frame was reported without an abort.

This layout is a proposal plus an unexecuted example. It assumes a POSIX g++ that accepts -std=c++20. On Windows the numeric abort code differs, and any non-zero status still fails the column.

Steps that refuse a stitched green build

Create attempts/001/include, attempts/001/src, and attempts/001/build. Copy golden/frame.hpp into the attempt include tree before applying the candidate.
Place that sample's payload.cpp under attempts/001/src. Leave earlier attempt directories untouched when a previous sample already failed.
Compile the candidate to payload.o. A non-zero status fails patch_compile and blocks promotion for this attempt only.
Compile tests/contract_tu.cpp against the candidate header and the new object file. A failed static_assert fails noexcept_contract.
Link tests/happy.cpp, run it, and compare stdout with golden/happy.out byte for byte, including the trailing newline.
Link tests/malformed.cpp, run it with a short timeout, and require exit code 0 plus golden/malformed.out. A signal, an escaped exception, or a timeout fails malformed_run.
Append one JSON object to ledger.jsonl. Include the attempt id, a hash of payload.cpp, and every column. Promote only when that same object marks each required column ok.
Keep failed rows. A later sample must pass on its own columns. A stdout pass from attempt 1 plus a contract pass from attempt 2 is not evidence.

The stitch is the review habit this ledger blocks. A retried sample looks like progress when the tool prints only the newest green line and discards the attempt that actually held the contract.

Declarations, fixtures, and the grader

Reviewed contract, stored as golden/frame.hpp:

#pragma once
#include <string_view>
#include <cstddef>

struct FrameView {
    std::string_view bytes;
};

// Short frames return 0. They do not throw.
std::size_t payload_size(FrameView frame) noexcept;

Consumer check in tests/contract_tu.cpp. The noexcept operator checks the declaration the compiler can see. It does not execute the body.

#include "frame.hpp"
#include <type_traits>

static_assert(noexcept(payload_size(FrameView{})));
static_assert(std::is_nothrow_invocable_v<decltype(payload_size), FrameView>);

int main() {
    return 0;
}

Happy path in tests/happy.cpp. The two-byte buffer avoids a hex escape so the second byte stays the value 4.

#include "frame.hpp"
#include <iostream>
#include <string_view>

int main() {
    const char raw[] = {1, 4};
    FrameView frame{std::string_view(raw, 2)};
    std::cout << payload_size(frame) << '\n';
    return 0;
}

Short-frame path in tests/malformed.cpp:

#include "frame.hpp"
#include <iostream>

int main() {
    FrameView frame{std::string_view()};
    std::cout << payload_size(frame) << '\n';
    return 0;
}

A body that still matches the header and reports a short frame without throwing:

#include "frame.hpp"

std::size_t payload_size(FrameView frame) noexcept {
    if (frame.bytes.size() < 2) {
        return 0;
    }
    return static_cast<unsigned char>(frame.bytes[1]);
}

That function is a fixture for the harness, not a parser. For the two-byte input, golden/happy.out contains the character 4 and a newline. For an empty view, golden/malformed.out contains 0 and a newline.

If a definition is marked noexcept and still throws, the program compiles and then calls std::terminate. Python's subprocess module reports a POSIX signal as a negative return code, often -6 for SIGABRT. The grader treats every non-zero code as a failed column, so it does not special-case 134 or -6. If the candidate removes noexcept from the header and the definition together, the contract translation unit fails even when both text fixtures still match.

Unexecuted grader, saved as grade_attempt.py next to golden/ and tests/:

#!/usr/bin/env python3
"""Unexecuted example. One attempt is promotable only when every column passes."""
import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
LEDGER = ROOT / "ledger.jsonl"
REQUIRED = (
    "patch_compile",
    "noexcept_contract",
    "happy_link",
    "happy_stdout",
    "malformed_link",
    "malformed_run",
)

def file_sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def run(cmd, timeout):
    proc = subprocess.run(
        cmd, cwd=ROOT, text=True, capture_output=True, timeout=timeout
    )
    return proc.returncode, proc.stdout, proc.stderr

def compile_one(columns, label, args):
    code, _out, err = run(args, 20)
    columns[label] = {"ok": code == 0, "stderr_tail": err[-300:]}
    return code == 0

def checked(cmd, timeout):
    try:
        code, out, _err = run(cmd, timeout)
        return code, out
    except subprocess.TimeoutExpired:
        return "timeout", ""

def grade(attempt: Path) -> dict:
    build = attempt / "build"
    build.mkdir(parents=True, exist_ok=True)
    inc = attempt / "include"
    src = attempt / "src" / "payload.cpp"
    columns = {}
    if not src.is_file():
        return {
            "attempt": attempt.name,
            "promotable": False,
            "columns": {"payload": {"ok": False}},
        }
    obj = build / "payload.o"
    compile_one(
        columns,
        "patch_compile",
        ["g++", "-std=c++20", "-c", str(src), "-I", str(inc), "-o", str(obj)],
    )
    if columns["patch_compile"]["ok"]:
        compile_one(
            columns,
            "noexcept_contract",
            [
                "g++", "-std=c++20", str(ROOT / "tests/contract_tu.cpp"),
                str(obj), "-I", str(inc), "-o", str(build / "contract"),
            ],
        )
        if compile_one(
            columns,
            "happy_link",
            [
                "g++", "-std=c++20", str(ROOT / "tests/happy.cpp"),
                str(obj), "-I", str(inc), "-o", str(build / "happy"),
            ],
        ):
            code, out = checked([str(build / "happy")], 5)
            expected = (ROOT / "golden/happy.out").read_text()
            columns["happy_stdout"] = {
                "ok": code == 0 and out == expected,
                "got": out,
                "returncode": code,
            }
        if compile_one(
            columns,
            "malformed_link",
            [
                "g++", "-std=c++20", str(ROOT / "tests/malformed.cpp"),
                str(obj), "-I", str(inc), "-o", str(build / "malformed"),
            ],
        ):
            code, out = checked([str(build / "malformed")], 5)
            expected = (ROOT / "golden/malformed.out").read_text()
            columns["malformed_run"] = {
                "ok": code == 0 and out == expected,
                "returncode": code,
                "got": out,
            }
    promotable = all(columns.get(name, {}).get("ok") for name in REQUIRED)
    return {
        "attempt": attempt.name,
        "payload_sha256": file_sha256(src),
        "promotable": promotable,
        "columns": columns,
    }

def main():
    if len(sys.argv) != 2:
        raise SystemExit("usage: grade_attempt.py attempts/001")
    row = grade(Path(sys.argv[1]))
    with LEDGER.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(row, sort_keys=True) + "\n")
    print(json.dumps({"attempt": row["attempt"], "promotable": row["promotable"]}))
    raise SystemExit(0 if row["promotable"] else 1)

if __name__ == "__main__":
    main()

Commands that create the first attempt and append one ledger row, after the golden header and the three test files are saved:

mkdir -p attempts/001/include attempts/001/src
cp golden/frame.hpp attempts/001/include/frame.hpp
cp candidates/001/payload.cpp attempts/001/src/payload.cpp
printf '4\n' > golden/happy.out
printf '0\n' > golden/malformed.out
g++ --version | head -n 1 > attempts/001/compiler.txt
python3 grade_attempt.py attempts/001

A second sample repeats the copy into attempts/002. It does not edit the row already written for 001.

How to read the row

Row shape	Meaning	Promote
All six required columns ok on this attempt	Same sample held the declaration and both runs	Candidate for review, not an automatic merge
Happy stdout ok, noexcept contract failed	Text matched after the header dropped noexcept	No
Happy stdout ok, malformed run non-zero	Abort or escaped exception on a short frame	No
Attempt A passed stdout, attempt B passed the contract	Stitched evidence	No
Malformed run timeout	Hang, or a limit too tight for the machine	No; inspect, then rerun that same attempt
Missing payload.cpp	Incomplete sample	No

Promotion here means the ledger process exits zero. It does not mean a maintainer should skip the diff. The payload hash lets a reviewer confirm the file on disk is the file that was graded.

A failed noexcept_contract row usually ends with a static-assertion note from the compiler. A failed patch_compile row often names an exception-specification mismatch, which means the header and the definition disagreed. A failed malformed_run with a negative return code is the terminate path.

Read that order before asking a model for another sample. Another sample does not repair a ledger that was graded against the wrong header copy.

A sample source is not a grade

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can supply the candidate payload.cpp, and a replacement header if the prompt allows one. The free server option can store attempts/ and ledger.jsonl, which keeps the failed rows off a single laptop disk and away from a production runner. The product does not grade the patch. The script does.

Current limits belong in the product documentation, not in a remembered campaign line. This walkthrough states no token allowance, no model name, no hardware shape, no retention window, and no promise that a free tier remains fixed. Before a shared review depends on that workspace, an operator should confirm the compiler version, the disk rules, and whether private sources are allowed there.

Leave keys and live packet captures out of the attempt tree. The two fixtures above are enough to exercise the columns. Record the first line of g++ --version beside the attempt when the server compiler and the laptop compiler differ, because a row graded on one driver is weak evidence for the other.

The script does not call a model and does not log in to a server. Keeping network steps outside the grader means a timeout from an API cannot be written down as a contract pass.

Who should not run this gate

The declaration check is blind to the body. The malformed process covers one input, not the input space. Leaks and races need their own gates, and a five-second timeout confuses a slow disk with a deadlock until a person reads the row.

Codebases that allow selected functions to throw should not aim static_assert(noexcept(...)) at those functions. Every honest edit would fail. Teams that need a formal proof, or a full audit of undefined behavior, need a heavier tool than six process columns.

A change whose purpose is to relax the public contract should not be smashed against the old golden header. Update golden/frame.hpp and the expected files in a reviewed commit, then sample against the new contract. Blocking an approved API edit is a process mistake.

The example has not been executed in this article. Expect to adjust include paths if the scratch tree differs from the layout above.

Keep the failed attempt on disk

A scratch repository and one candidate file are enough to watch a stitched green build exit non-zero. Readers who already use a MonkeyCode workspace can park the attempt directories on the free server option, then check the current docs for limits before treating that workspace as review storage.

Fifteen Tenths Print Two Ways: A Three-Column Grader for C++ Patches

Finley Li — Sat, 10 Oct 2026 17:35:28 +0000

A nightly billing export can stay green after an AI-written C++ patch changes rounding. The candidate binary prints 1 for 15 tenths. The golden file prints 1 as well. A frozen reference, stored in another tree, still prints 2.

Reviewers who only diff the candidate against the golden see a match and accept the row. That match is the bug this harness is built to catch. Silent regressions in patch evals often look like agreement, because the expected file moved with the candidate or was already stale. A two-column check has no independent voice left.

What the third column changes

This workflow keeps three outputs for every case. The reference comes from a frozen implementation the patch is not allowed to edit. The candidate comes from the patch under test. The golden comes from a file that the run itself must not rewrite.

Candidate-to-golden agreement is not a pass on its own. Reference-to-candidate agreement against a different golden is not an automatic file update either. The grade is the pattern of the three comparisons, under one equality rule declared before the run.

Integer and text cases use exact output equality. Money in this example stays in scaled integers so formatting cannot hide a one-unit error. Teams that must grade floats should pick one policy, such as a fixed distance in units in the last place, and apply that policy to every column. A tight compare on the golden and a loose compare on the reference rebuilds the blind spot.

Step 1 — Freeze the reference, then list boundary rows

Build the reference from a tree the candidate patch cannot see. Record a hash of that binary next to the case list. A later grade that names a different hash is a different experiment, even if the source path looks familiar.

# Unexecuted build sketch. Flags are illustrative, not a tuned recipe.
c++ -std=c++17 -O2 -o bin/ref_billing src/ref_main.cpp
c++ -std=c++17 -O2 -o bin/cand_billing src/cand_main.cpp
sha256sum bin/ref_billing > build/ref_billing.sha256
sha256sum -c build/ref_billing.sha256

Do not paste a digest that this tree did not just produce. A copied sample hash looks like evidence and is not. The sidecar file should be updated only when a human accepts a new reference, not when a candidate run fails.

Cases live in JSONL. Each row carries an id, argv, and the golden text. The golden is data for the grader. It is not an argument passed to either binary.

{"id":"tenths-0","args":["0"],"golden":"0"}
{"id":"tenths-4","args":["4"],"golden":"0"}
{"id":"tenths-5","args":["5"],"golden":"1"}
{"id":"tenths-15","args":["15"],"golden":"2"}
{"id":"tenths-neg-5","args":["-5"],"golden":"-1"}
{"id":"tenths-neg-15","args":["-15"],"golden":"-2"}

The reference used here rounds half away from zero on magnitude. It is a specification stand-in for the workflow, not a claim about any live billing system.

// Frozen reference. Unexecuted illustration. Kept outside the candidate tree.
long long ref_cents_from_tenths(long long tenths) {
    long long sign = tenths < 0 ? -1 : 1;
    long long mag = tenths < 0 ? -tenths : tenths;
    return sign * ((mag + 5) / 10);
}

Two candidate-shaped mistakes show up often in generated arithmetic. Truncation toward zero maps 15 to 1 and -15 to -1. Adding five before dividing looks right on positives and still maps -15 to -1.

// Candidate-shaped bugs. Unexecuted illustrations.
long long trunc_toward_zero(long long tenths) { return tenths / 10; }

long long half_up_breaks_on_sign(long long tenths) { return (tenths + 5) / 10; }

If a generator also rewrites tenths-15 so the golden becomes 1, a grader that only reads candidate and golden goes green. The reference still prints 2. The third column is what keeps that row red.

Step 2 — Run both binaries inside one replay envelope

Both processes get the same locale and timezone. The grader reads stdout only. Stderr is stored beside the row and does not join the equality check, so a warning cannot impersonate a result. Neither binary receives the golden path.

export LC_ALL=C
export TZ=UTC
python3 grade_three_col.py ./bin/ref_billing ./bin/cand_billing

The script is an unexecuted proposal. It is a debugging workflow, not a recorded benchmark and not evidence about any model.

# Unexecuted example. Proposal only; not a measured run.
import json, subprocess, sys
from pathlib import Path

def run(bin_path, args):
    proc = subprocess.run(
        [bin_path, *args],
        capture_output=True,
        text=True,
        check=False,
    )
    return proc.returncode, proc.stdout.strip(), proc.stderr

def classify(ref_out, cand_out, golden):
    same = ref_out == cand_out
    cand_gold = cand_out == golden
    ref_gold = ref_out == golden
    if same and cand_gold:
        return "pass_all_agree"
    if same and not cand_gold:
        return "golden_rot"
    if cand_gold and not ref_gold:
        return "coupled_or_stale_golden"
    if ref_gold and not same:
        return "candidate_regression"
    return "three_way_split"

def main():
    rows = [
        json.loads(line)
        for line in Path("cases.jsonl").read_text().splitlines()
        if line.strip()
    ]
    failures = []
    for row in rows:
        rc_r, ref_out, err_r = run(sys.argv[1], row["args"])
        rc_c, cand_out, err_c = run(sys.argv[2], row["args"])
        if rc_r != 0 or rc_c != 0:
            failures.append({
                "id": row["id"],
                "kind": "nonzero_exit",
                "rc_ref": rc_r,
                "rc_cand": rc_c,
            })
            continue
        kind = classify(ref_out, cand_out, row["golden"])
        if kind != "pass_all_agree":
            failures.append({
                "id": row["id"],
                "kind": kind,
                "reference": ref_out,
                "candidate": cand_out,
                "golden": row["golden"],
                "stderr_ref": err_r,
                "stderr_cand": err_c,
            })
    print(json.dumps({"failures": failures, "graded": len(rows)}, indent=2))
    return 1 if failures else 0

if __name__ == "__main__":
    sys.exit(main())

A failing row for the coupled case looks like this. Reference says 2, candidate says 1, golden says 1. The next action is to reject the patch and to restore the golden if this run, or the generator, wrote that 1.

{"id":"tenths-15","kind":"coupled_or_stale_golden","reference":"2","candidate":"1","golden":"1"}

Step 3 — Classify the pattern before any file edit

Four patterns need different handling. A single red count hides that difference. Automatic golden refresh hides it further, so this workflow does not include one.

Pattern	What matches	What to do
`candidate_regression`	Reference matches golden. Candidate differs.	Reject the patch. Leave the golden untouched.
`coupled_or_stale_golden`	Candidate matches golden. Reference differs.	Reject the patch. Check whether the golden was rewritten from this candidate.
`golden_rot`	Reference matches candidate. Both differ from golden.	Stop. A person may refresh the golden only after a reviewed specification change.
`three_way_split`	All three differ.	Stop. Do not pick a winner from the three strings.

coupled_or_stale_golden is the quiet pattern. The row looks reviewed because two artifacts match. Those two are not independent when the same generation step could edit both. Independence comes from a reference binary the generator cannot read and a golden file the grader cannot write.

Exit status is nonzero if any row is not pass_all_agree. A note that says most rows passed is a log line, not the gate. The worst pattern in the batch decides the run.

Step 4 — Reject a swapped reference before trusting outputs

A prompt checksum does not prove the reference binary is the one the goldens were written against. Compare the current bin/ref_billing digest with build/ref_billing.sha256 before classify runs. On mismatch, emit reference_identity_drift and skip output grading.

A swapped reference can make every old golden look rotten. An automatic update in that state would launder the swap into the expected file. Keep the digest in a sidecar, not inside cases.jsonl, so a golden edit cannot silently retarget the oracle.

Step 5 — Send failing rows forward, not the oracle

A later candidate can be requested from the failing ids, the inputs, the reference outputs, and the pattern names. The request should not grant write access to cases.jsonl or to the reference sources. The following run uses the same frozen binary hash and the same golden bytes.

That loop is what makes a silent retry visible. A generator that repairs the grade by editing expected text fails coupled_or_stale_golden again, because the reference did not move. A generator that changes the C++ so 15 yields 2 and -15 yields -2 can reach pass_all_agree only when the golden already said those values.

Keep the prompt payload small and dull. Include the mismatch. Omit instructions to update tests, relax comparisons, or delete rows. Those edits create green runs without a behavior fix, and this classifier is there to make them obvious.

Where a free generation host fits

MonkeyCode is described by the operator as an open-source project that offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Both availability claims are operator-supplied. This article does not state a token quota, a hardware shape, a time limit, or a promise that the offer stays open. Model names and current terms should be taken from the project's own materials at publish time, not from memory of an older post.

Inside the loop, free model access is only a place to request the next patch from failing rows. The free server option is only a place to run the two binaries and grade_three_col.py away from the generator. The grader host should mount the reference as an executable, not as a source tree. A candidate that can open ref_main.cpp can copy it and manufacture agreement.

The same three-column rule works if the patch is produced somewhere else. Vendor choice does not permit an automatic golden write. Removing every product name from this page would leave the classifier, the case file, and the rejection rules intact.

Limitations

A wrong reference certifies wrong behavior. This harness does not invent a specification. It only reports when the candidate, the golden, and a chosen reference stop agreeing.

Exact stdout checks miss bugs outside the case list. The six tenths rows cover zero, a below-half value, a half value, a larger positive, and the signed pair. They do not cover overflow near the ends of long long, and they say nothing about threads, exceptions, or allocators.

Nonzero exits fail the row even when stdout matches. That will reject a program that prints the right answer and then returns a warning code. Teams with a different contract should encode it in classify, not by ignoring exit status for one convenient patch.

This gate does not replace a sanitizer run, an assertion census, a cost check, or a link against an untouched consumer. Those questions stay open after pass_all_agree. Folding their logs into the golden file would couple independent evidence to the same writable text the classifier distrusts.

Free access can change. A plan that needs a fixed capacity number should wait for a current written limit. An old quota repeated from a previous article is not evidence for this week.

Who should not use this gate

Teams with no trusted reference should not write one in the same session that writes the patch. A second function from the same prompt is another candidate. It does not supply a third column.

Intentional specification changes need a human golden edit, landed outside the candidate run. Until that edit lands, the grader will report golden_rot or three_way_split. That is a sequencing constraint, not a reason to turn the classifier off for the day.

Output-identical refactors and pure speed work can pass this gate while still being wrong for cost, linkage, or build hygiene. Use another check for those risks. Do not weaken the three-column compare so a performance patch can change printed results by a little.

Operators who already freeze a reference can point failing rows at a generator they trust, then rerun this classifier on a host that cannot read reference sources. Confirm current access terms before that host is treated as available.

Matching Prints Hide Freed Storage: A Twin-Run Sanitizer Gate for C++ Patches

Finley Li — Fri, 09 Oct 2026 13:30:08 +0000

A coding-model case accepted a C++ patch because the program printed case-7 and exited zero. The same binary, started again with the same arguments, printed that line a second time. Nothing in the functional golden file moved. A sibling build, compiled with AddressSanitizer and UndefinedBehaviorSanitizer, aborted while materializing the return value. The characters in the golden file had been read from storage that the patch had already deleted.

A green print is a contract check. It is not a memory check, and it is not a stability check. The workflow below records three bits for each case: functional match, twin-run stability, and a sanitizer-clean lane. A lucky first sample cannot close the case by itself.

What the three bits mean

The harness grades one fixture at a time. It does not rank vendors, and a full accept is not a production sign-off. Every listing in this article is a proposed lab script. None of it was executed against a named model for this draft, and none of it is a benchmark table.

The model may replace src/impl.cpp only. The driver, the expected stdout, and the grader stay read-only. A patch that touches those paths is rejected before any bit is computed. Permission bits on the oracle are a tripwire for accidental edits, not a security boundary. A separate account or a read-only mount is what actually keeps an untrusted patch away from the expected file.

1. Freeze a fixture the patch cannot edit

The operator creates the case tree once, then removes write permission from the driver and the golden stdout before any generation step.

mkdir -p cases/label7/src cases/label7/expected cases/label7/work
cat > cases/label7/src/main.cpp << 'EOF'
#include <iostream>
#include <string>

std::string label_for(int id);

int main() {
  std::cout << label_for(7) << '\n';
  return 0;
}
EOF
printf 'case-7\n' > cases/label7/expected/stdout.txt
chmod a-w cases/label7/src/main.cpp cases/label7/expected/stdout.txt

The implementation under test still tends to emit the expected line on a quiet allocator. That tendency is not a guarantee. Use after free is undefined on the functional build as well, which is why a single green print was never evidence. The copy into out is a decoy. The returned object is built from a freed buffer.

#include <string>

std::string label_for(int) {
  std::string* heap = new std::string("case-7");
  std::string out = *heap;
  delete heap;
  return *heap;
}

A second fixture covers drift that is not a heap error. The accumulator is uninitialized, so agreement between two runs is a property to measure, not a courtesy to assume.

int fold_ids(int n) {
  int acc;
  for (int i = 0; i < n; ++i) acc += i;
  return acc;
}

The prompt for this second case asks for the sum of 0 .. n-1. A first print of 45 when n is 10 can collide with a golden file. That collision is one sample. The twin bit exists so the sample cannot stand alone.

2. Build a functional binary and a sanitized sibling

Both binaries compile from the same pair of translation units. The functional build turns warnings into errors, so a patch that only survives under -w never reaches the behavior bits. The sanitized sibling keeps frame pointers and debug info so an abort names a line.

CASE=cases/label7
mkdir -p "$CASE/work"
g++ -std=c++20 -O1 -Wall -Wextra -Werror \
  -o "$CASE/work/fn" "$CASE/src/main.cpp" "$CASE/src/impl.cpp"
g++ -std=c++20 -O1 -g -fno-omit-frame-pointer \
  -fsanitize=address,undefined \
  -o "$CASE/work/san" "$CASE/src/main.cpp" "$CASE/src/impl.cpp"

Failure of either command records build_fail and skips the behavior bits. A binary that never linked does not receive a partial behavior score. The fragments in this article are sequential parts of one lab session. They are not a packaged installer, and they assume the previous variables are still in the shell.

3. Require the same stdout on two runs

The functional binary runs twice from the case work directory. Locale is pinned. Arguments stay identical. The first stdout must match the golden file, and the second stdout must match the first byte for byte.

export LC_ALL=C
( cd "$CASE/work" && ./fn > run1.out )
status1=$?
( cd "$CASE/work" && ./fn > run2.out )
status2=$?
diff -q "$CASE/expected/stdout.txt" "$CASE/work/run1.out"
func_match=$?
diff -q "$CASE/work/run1.out" "$CASE/work/run2.out"
twin_stable=$?

A non-zero process status fails the functional bit even when the captured text looks plausible. This script does not bound runtime. Operators who need a bound wrap each launch with timeout 2 and treat status 124 as a functional failure. That limit is local lab policy, not a property of any model host.

4. Score sanitizer stderr as its own bit

The sanitized binary is not asked to reproduce the golden file for the accept decision. Its exit status and diagnostic stderr are the signal. A zero exit and no sanitizer diagnostic set the bit. Any other outcome clears it.

( cd "$CASE/work" && ./san > san.out 2> san.err )
san_status=$?
if [[ $san_status -eq 0 ]] && ! grep -E -q 'AddressSanitizer|UndefinedBehaviorSanitizer|runtime error:' san.err; then
  sanitizer_clean=1
else
  sanitizer_clean=0
fi

Leak detection stays enabled. A case that intentionally keeps a process-lifetime buffer should say so in the prompt and may export ASAN_OPTIONS=detect_leaks=0 for that case alone. Turning leaks off for the whole suite would erase a defect class this lane is meant to keep visible. Partial stdout from an aborted sanitizer run is ignored. The accept bit does not reward a crash that happened to flush the golden line first.

5. Fold the bits into one verdict file

The recorder stores a single JSON object. Later summaries may count accepts. They may not promote one true bit into a pass.

functional=false
twin=false
sanitizer=false
if [[ $func_match -eq 0 && $status1 -eq 0 && $status2 -eq 0 ]]; then
  functional=true
fi
if [[ $twin_stable -eq 0 && $status1 -eq 0 && $status2 -eq 0 ]]; then
  twin=true
fi
if [[ $sanitizer_clean -eq 1 ]]; then
  sanitizer=true
fi
accept=false
if [[ $functional == true && $twin == true && $sanitizer == true ]]; then
  accept=true
fi
cat > "$CASE/work/verdict.json" << EOF
{
  "case": "$CASE",
  "functional": $functional,
  "twin": $twin,
  "sanitizer": $sanitizer,
  "accept": $accept
}
EOF

functional	twin	sanitizer	accept	reading
true	true	true	true	stdout held twice and instrumentation stayed quiet
true	true	false	false	expected text hid a memory or undefined-behavior fault
true	false	true	false	the first print matched and the second print drifted
false	any	any	false	the functional contract failed; other bits stay diagnostic
build failed	—	—	false	no behavior score is recorded

An accept means the case may move to human review. It does not mean the patch is free of data races, of logic errors outside the fixture, or of faults inside uninstrumented libraries.

6. Archive the failure before touching the prompt

When accept is false, sanitizer stderr is read first if the sanitizer bit is false. The two stdout files are compared next if the twin bit is false. The prompt stays frozen in that same step. Editing the prompt and the patch together makes the next verdict incomparable to the one just archived.

mkdir -p logs
cp "$CASE/work/verdict.json" logs/label7-verdict.json
if [[ -f "$CASE/work/san.err" ]]; then
  cp "$CASE/work/san.err" logs/label7-san.err
fi

The log directory sits outside the case tree so a later patch cannot overwrite the evidence. Only after that copy does the operator delete work/ and request another implementation. A rerun that changes compilers without noting the compiler identity will not be comparable either, so the archive step should also store the output of g++ --version beside the verdict.

Where free model access and a free server fit

The gate stands without a particular vendor. Something still has to draft impl.cpp, and the two binaries still have to run outside the editor that stored the prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant. The operator supplied two availability claims for this draft: free model access, and a free server option. No token quota, hardware size, rental duration, or permanence claim is stated here. Those figures were not checked against a current primary document, and unpublished token counts are not treated as measurements.

One workable split sends the prompt through free model access and writes the reply into src/impl.cpp. The grade then runs in a disposable tree, with the expected file mounted read-only. The free server option matters when that tree should not live on a laptop. Sanitizer links often demand more memory than an ordinary -O1 binary. If the free server cannot host the sanitized sibling, generation can remain on the free model path and the sanitizer bit should move to a machine that can link it. Dropping the third bit to squeeze onto a small host recreates the original miss: a matching print, and an untested heap.

Keys do not belong in the case directory. The script reads sources and writes verdict.json only.

Limitations

AddressSanitizer and UndefinedBehaviorSanitizer miss logic bugs on paths the fixture never calls. They also miss data races unless a separate ThreadSanitizer build is added, and this script does not build one. A twin run will not expose a bug that is stably wrong. Two identical wrong lines still fail the functional bit, which is why the golden file sits outside the patch tree.

False sanitizer reports show up when a case links prebuilt objects that were not instrumented the same way. The commands assume a GNU-style g++, diff, and bash. A host without those tools will not produce comparable bits. Untrusted patches belong in a disposable environment with no credentials and no network. This article does not specify that sandbox. It only refuses to treat a local green print as a finished review.

Who should skip this gate

Language suites without a comparable instrumentation story will not gain a third bit by renaming the script. Leaderboards that need latency, throughput, or cross-model ranks need a different artifact. This gate covers a narrower procedural point: whether one patch kept its stdout contract across two runs without tripping the sanitizers already installed. Operators who cannot isolate untrusted builds should not aim a free server at raw patches and call the JSON a review.

Teams that already store golden stdout can add the twin diff and the sanitized sibling without rewriting prompts. If the current MonkeyCode documentation still lists free model access and a free server option, either can feed this case tree. The verdict file, not a product banner, decides whether the print was enough.

Build the Patch as a Stranger: A Consumer-Link Gate for C++ Codegen

Finley Li — Thu, 08 Oct 2026 09:28:18 +0000

A packaging job can fail on a Thursday morning while the eval board stays green. In that pattern, the candidate patch repairs normalize() inside one translation unit, and the grader compiles that unit together with the tests. The public header never gains the declaration. Downstream, consumer.cpp includes only the shipped header, links libpatch.a, and stops on an undefined reference.

That miss is structural, not a fluke of one model. In-tree tests often see private headers, extra -I paths, and a link line the author of the harness invented. A stranger's binary does not. The consumer-link gate builds the patch the way a later client would, then classifies the failure before anyone treats a green board as a merge signal.

What the in-tree binary hides

A single test executable can pass for boring reasons. The test file may declare the missing function locally. The build may compile patch.cpp straight into the test binary, so an archive was never produced. A macro set in the test harness may change the header, while a clean client never defines that macro.

Behavior checks still matter. They do not answer whether the patch is installable. Link errors, one-definition-rule collisions, and include-path accidents are silent regressions relative to a grader that never leaves its own tree.

The example below is synthetic. It exists so a team can reproduce the failure class on a laptop before any model is scored. No timing numbers, pass rates, or customer incidents are claimed for it.

Four outcomes worth naming

The gate records one class per golden case. Names stay stable so a later diff of JSON is readable.

header_missing means the consumer could not include the public header with the published -I path.
link_undefined means the header compiled and the archive did not provide a required symbol.
odr_or_duplicate means two translation units disagreed on an inline function, a macro, or a weak symbol.
behavior_mismatch means link and start succeeded, and the process exit or stdout disagreed with the golden file.

Timeouts stay in a fifth bucket, hang, so a stalled binary is not scored as a wrong answer. Optimization matrices and assertion censuses belong in other gates. This one only asks whether a stranger can compile, link, and run.

Step 1 — Split the library from the client

Keep the patch sources, the public header, and the consumer in separate directories. The consumer may include patch.hpp and nothing under src/. Private helpers stay out of the client file even when that makes the case look blunt.

CXX ?= c++
CXXFLAGS ?= -std=c++20 -Wall -Wextra -Werror -Iinclude
.PHONY: clean gate

build/libpatch.a: src/patch.cpp include/patch.hpp
    mkdir -p build
    $(CXX) $(CXXFLAGS) -c src/patch.cpp -o build/patch.o
    ar rcs $@ build/patch.o

build/consumer: consumer/consumer.cpp include/patch.hpp build/libpatch.a
    $(CXX) $(CXXFLAGS) consumer/consumer.cpp build/libpatch.a -o $@

gate: build/consumer
    ./build/consumer > build/consumer.out

clean:
    rm -rf build

The archive step is the point. If the model left the definition in a file the test binary used to compile directly, ar never sees that symbol, and the consumer link fails on purpose. A green make gate after that split is a stronger claim than a green in-tree unit test.

A minimal client looks like this. It is a golden case, not a demo to extend with private helpers.

#include "patch.hpp"
#include <iostream>

int main() {
  auto out = demo::normalize("  Ok \n");
  if (out != "ok") return 2;
  std::cout << out << "\n";
  return 0;
}

The matching header must stand alone. A forward declaration stuffed into consumer.cpp would hide the bug the gate exists to catch, so the grader rejects consumer files that declare demo:: symbols themselves. That check is a text scan, cheap and strict, and it runs before make.

Step 2 — Pin the golden contract

Each case is a small JSON file. Paths are relative to the case root. Expected classes describe the seed bug, not every future patch. A correct patch should land on pass.

{
  "id": "normalize-trim",
  "public_header": "include/patch.hpp",
  "archive_sources": ["src/patch.cpp"],
  "consumer": "consumer/consumer.cpp",
  "expect_stdout": "ok\n",
  "expect_exit": 0,
  "forbid_consumer_decls": ["demo::"],
  "compiler_note": "pin c++ --version in the run manifest"
}

Seed a known-bad patch beside it, with the declaration omitted from the header, and expect link_undefined. That seed proves the grader can fail. A harness that has never seen red will happily mark every later accident as pass.

Store the seed under a name that cannot be confused with the fix, such as goldens/normalize-trim.missing-decl.json. Reviewers should be able to run the red case in one command and read the class without opening a dashboard.

Step 3 — Grade the log, not the vibe

The following grader is a proposed workflow, not a report of a measured run. It shells out to make, then maps compiler text to the classes above. Diagnostic strings vary by compiler, so the patterns are explicit and the compiler version is written into the same JSON as the grade.

import json, re, subprocess, sys
from pathlib import Path

PATTERNS = [
    ("header_missing", re.compile(r"fatal error: .*No such file or directory")),
    ("link_undefined", re.compile(r"undefined reference to")),
    ("odr_or_duplicate", re.compile(r"multiple definition of")),
]

def classify(log: str, exit_code: int, stdout: str, golden: dict) -> str:
    for name, pat in PATTERNS:
        if pat.search(log):
            return name
    if exit_code != 0 and "timed out" in log:
        return "hang"
    if stdout != golden["expect_stdout"] or exit_code != golden["expect_exit"]:
        return "behavior_mismatch"
    return "pass"

def main() -> int:
    golden = json.loads(Path(sys.argv[1]).read_text())
    subprocess.run(["make", "clean"], check=False, text=True, capture_output=True)
    proc = subprocess.run(["make", "gate"], check=False, text=True, capture_output=True)
    log = proc.stdout + proc.stderr
    out_path = Path("build/consumer.out")
    stdout = out_path.read_text() if out_path.exists() else ""
    kind = classify(log, proc.returncode, stdout, golden)
    report = {"id": golden["id"], "class": kind, "exit": proc.returncode}
    Path("build").mkdir(exist_ok=True)
    Path("build/grade.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report))
    return 0 if kind == "pass" else 1

if __name__ == "__main__":
    raise SystemExit(main())

Run it against one case, then against the known-bad seed.

python3 grade_consumer.py goldens/normalize-trim.json
python3 grade_consumer.py goldens/normalize-trim.missing-decl.json
nm -g build/libpatch.a

The second command should print "class": "link_undefined". If it prints pass, the consumer is still compiling the patch source by accident, and the Makefile is wrong. Fix that before scoring any model output. nm is the tie-breaker when the log is noisy: a missing global symbol confirms the archive, not the regex.

Step 4 — Keep the worker poorer than the grader

Candidate patches should be generated and built in a directory that contains include/, src/, consumer/, and the Makefile, and nothing else. Ambient headers from a developer's home directory are a common way for a broken patch to look healthy. Copy the case tree into an empty folder and export a minimal PATH before calling make.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project and reports free model access plus a free server option. No primary source dated 2026-10-08 was attached for a permanent token ceiling, hardware shape, model list, or duration, so this workflow does not print a quota as a fact. The run manifest stores only what the account response and the build worker showed on that run: a timestamp, the model id string if the API returns one, the compiler version, and the grade JSON.

Use the free server option as the poor worker when it is actually available. Copy the case tree up, run make gate, and pull build/grade.json back. A local laptop remains the reference. Disagreement between the two grade files is a harness bug or an environment bug, not a reason to average the scores.

A practical next step is to diff those two JSON files after one local run and one free-server run, and to treat a mismatch as a failed eval rather than as a successful demo.

How a failing class is reviewed

A link_undefined result is not yet a verdict on the model's reasoning. The reviewer opens the public header and the archive member list. nm -g build/libpatch.a shows whether the symbol was compiled at all. If the symbol exists but is local, the model likely omitted the declaration or hid the definition in an anonymous namespace. If the symbol is absent, the source file was not part of archive_sources, or the definition was wrapped in a macro the consumer never sets.

A header_missing result sends the reviewer to the include path only. Adding -I until the board turns green is the wrong fix. The consumer command in the Makefile is the contract. Change the patch to place the header where that command already looks.

A behavior_mismatch after a clean link goes back to the golden string. Update the golden file only when the contract truly changed, and record the reason in the case notes. Silent edits to expected stdout are how a harness learns to approve regressions.

Decision table

Log fragment	Class	Do not do
`No such file or directory` on the public header	`header_missing`	Add the grader's private `-I` to the consumer
`undefined reference to`	`link_undefined`	Compile `src/*.cpp` into the consumer
`multiple definition of`	`odr_or_duplicate`	Delete the second TU to make the board green
Exit `2` or wrong stdout	`behavior_mismatch`	Rewrite the golden string to match the patch
No exit before the timeout	`hang`	Score it as a behavioral near-miss

Limitations

This gate does not prove algorithmic correctness beyond the golden stdout and exit code. It does not replace sanitizer builds, cross-optimization checks, or a review of whether the model deleted assertions. Floating-point cases need an explicit tolerance field this sample omits. Compiler diagnostics in languages other than English, or from a compiler that phrases undefined references differently, will fall through to behavior_mismatch unless the patterns are extended.

Header-only libraries gain little from an archive step. Skip the gate there, or replace the archive with a second translation unit that includes only the public header. Teams that already install the package and compile an external smoke client in CI will find this redundant. Patches that are mid-refactor, with an intentionally broken public header, should not be scored as regressions against a stable golden case.

The Python snippet is a sketch. It does not isolate the build in a container, does not bound CPU, and trusts make output. Production use needs a timeout wrapper, a clean PATH, and a pinned compiler recorded in the same JSON as the grade. Anyone who needs a measured quality claim for a named model should run the seed case first and publish the manifest, not a slogan.

Closing

A green in-tree test means the grader's translation unit was satisfied. It does not mean a later client can include the header and link the archive. Split those builds, name the failure, and keep the worker from seeing headers the client will never have. The score that remains is smaller, and harder to fake.

Clean Includes, Separate Grades: Probing AI C++ Patches Outside the Driver

Finley Li — Thu, 08 Oct 2026 06:59:10 +0000

A composite night-build scene, not a measured incident from this account, still matches a failure C++ evals keep shipping. A grader accepted a patch just after 02:00. Every golden assertion passed, and the diff touched only inventory.h plus inventory.cpp.

The next consumer file included that header and nothing else. Compilation stopped because std::string was unknown. The eval driver had included <string>, <vector>, and a project prelude before it loaded the header under test. The patch looked finished because the driver had already done the include work.

That failure mode is quiet. Functional graders never see it. A standalone translation-unit gate does.

What the green run actually proved

The driver is a convenience program. It exists to call functions and check return values. It is a poor witness for header hygiene.

A model can emit a public header that uses std::vector without including <vector>, then still compile, because the driver's first lines already pulled that header in. The same blind spot hides missing forward declarations, macros supplied by a unity prefix, and a precompiled header a free-standing consumer will never receive. A passing driver compile therefore proves less than the status line suggests.

A useful grade separates two checks. One check asks whether behavior matches the golden cases. The other asks whether a stranger can include the public header in a clean translation unit and compile a tiny call. A single green exit code blends those checks and hides the second failure.

The probe the harness should emit

The files below are a proposed workflow. They are not a recorded run, and they are not a benchmark. The operator points them at a patch directory, a public-header list, and a compiler command. That command can target a local toolchain or a clean remote tree.

Keep the probe beside the existing functional grader, not inside it. public_headers.txt names only headers a downstream file is allowed to include:

include/inventory.h

The broken header that motivates the gate looks like this:

#pragma once
namespace inventory {
struct Item {
  Item(std::string name, int qty);
  const std::string& label() const;
  std::string name;
  int qty;
};
}

A driver that already included <string> will compile a call through that header. A standalone consumer will not. The repaired form includes what it uses:

#pragma once
#include <string>
namespace inventory {
struct Item {
  Item(std::string name, int qty);
  const std::string& label() const;
  std::string name;
  int qty;
};
}

probe_main.cpp.in carries one call already present in the golden case. The harness substitutes the header path. It does not ask the model for a new specification, and it does not embed an expected output table in the header.

#include "HEADER_PATH"

int main() {
  inventory::Item item{"bolt", 3};
  return item.label().empty() ? 1 : 0;
}

The return value is only a liveness check so the compiler must instantiate the call. It is not the behavioral oracle. Behavioral truth stays in the golden-case lane.

Numbered steps

Copy the patched tree into a scratch directory. Drop prefix-header flags such as -include prelude.h for this lane only, and do not reuse a precompiled header or a unity build file. Leave the functional lane on its original flags so a logic regression still has its own signal.
Read public_headers.txt. Render one probe_main.cpp per public header from the template. The rendered file may include that single public header. It must not include the eval driver, the prelude, or a catch-all project header.
Compile the probe as its own translation unit. Pin the language mode in the command so a remote default cannot silently change the meaning of a pass. This invocation is a starting point, not a universal CI policy:

sed "s|HEADER_PATH|inventory.h|" probe_main.cpp.in > build/probe_inventory.cpp
c++ -std=c++20 -Wall -Wextra -Werror -I include -c build/probe_inventory.cpp -o build/probe_inventory.o

Classify the compiler result before reading the functional grade. A missing declaration, an incomplete type, or an unknown name is a hygiene failure even when every driver assertion passed. A successful object file is a pass for this lane alone.
Append a one-line record the next run can diff. The record is the gate. A dashboard is optional.

case=inventory header=include/inventory.h lane=standalone_tu status=fail class=missing_declaration

diff -u grades/last.txt grades/this.txt

Run the functional golden cases in a separate process with the original driver. Do not skip them after a hygiene failure. A patch can be self-sufficient and still wrong. A patch can be behaviorally right and still unusable from a clean include. The report should show both rows.
Repeat the compile on a second tree that has never been built on the developer machine, when such a tree exists. The useful property is the absence of local precompiled headers and cached unity outputs. Host size, region, and uptime are out of scope for the grade.

A small classifier

Map compiler text to a stable class in a wrapper the repo owns. The patterns below are proposals. They need adjustment when the compiler or the language pack changes.

#!/bin/sh
log="$1"
if grep -E -q "error: .*(does not name a type|was not declared|incomplete type|no member named)" "$log"; then
  echo "missing_declaration"
  exit 2
fi
if grep -E -q "error:" "$log"; then
  echo "other_compile_error"
  exit 3
fi
echo "clean"
exit 0

Treat the exit code as the contract. Prose in the log is evidence, not the grade key. A later compiler upgrade can rename a sentence without changing the exit status the harness stores.

Decision table

Probe class	Functional grade	Report	Merge?
clean	pass	both lanes green	eligible for review
missing_declaration	pass	hygiene fail, behavior pass	no
missing_declaration	fail	both lanes red	no
other_compile_error	pass	flag or API drift	no
clean	fail	include ok, logic wrong	no
probe skipped	pass	incomplete grade	no

Skip is not green. A harness that cannot find a compiler, a header list, or a scratch tree must record an incomplete grade. Otherwise a missing tool becomes a silent pass, which is the same class of mistake as a driver that includes the world.

Where a hosted compile fits

Iteration is the costly part of the lane. Each failed probe names a missing include or an incomplete type. A coding model can suggest the repair. The probe accepts or rejects that repair.

The model is not the grader. A greener suggestion still has to survive the clean translation unit. That split is the whole point of keeping golden cases and the include probe in different processes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The outreach describes free model access and a free server option for this loop: request a header repair, then compile the standalone probe on a clean tree that does not share the laptop's precompiled headers. Those two availability claims are operator-supplied.

This article does not state a token quota, a hardware spec, a duration, or a permanent offer, because no primary source for those details was attached. Confirm the current limits, model list, and server terms on the project page before a team depends on them. A local compiler is enough to run the gate.

Use the hosted compile as a second witness, not as the only witness. If the remote log and the local log disagree, keep both. Disagreement is a result. It often means a hidden -include flag, a different default language mode, or a header found through an ambient include path.

Limitations

This gate does not replace include-what-you-use, a link test, or a sanitizer build. A header can include <string> and still return a reference into a temporary. The sample's const std::string& accessor is a reminder of that gap, not a fix for it. Macro-heavy headers can pass one probe call and fail under another feature flag.

The classifier's regular expressions are compiler-specific and will miss a diagnostic emitted in another language. The workflow also assumes the operator knows which headers are public. Point it at an internal header and it will punish legal layering.

Teams that already compile every public header inside a hermetic image with a pinned compiler gain little from a duplicate of that job. Projects whose documented contract is a single umbrella header should probe the umbrella, not each internal file. Readers who need a reproducible claim about a free allowance or a free host should use the current project documentation, not this walkthrough, as the quote.

Closing

A driver that includes the world will keep congratulating incomplete headers. Split the grade. One lane checks behavior against golden cases. The other compiles a stranger's file that includes only the public header.

Record both rows. Send the patch to review only when neither lane had to borrow an include from the harness. A green assertion log, by itself, is not evidence that the header can stand alone.