DEV Community: Finley Li

Zero Hits on the Hunk: A Coverage Gate for AI C++ Patch Evals

Finley Li — Thu, 17 Sep 2026 05:31:01 +0000

A C++ service accepted an AI patch on a Friday afternoon. The golden suite stayed green. The reviewer saw a plausible rewrite of decode_frame() and merged.

Monday's on-call page was a truncated header from a partner. decode_frame() had never run under the goldens. Those cases only constructed payloads that skipped the new branch. The harness had measured a function the patch did not touch.

This failure mode is not a flaky test. It is a blind one. When models outgrow the cases used to score them, a green golden file can mean the suite never called the hunk.

Blind goldens are a different bug class

Output digests catch silent changes in printed bytes. Resource envelopes catch RSS spikes. Flag matrices catch -O2 surprises. None of those checks ask whether any golden executed the lines the model rewrote.

Coverage is a weak proxy for correctness. It is a strong proxy for contact. A hunk with zero hits was not evaluated. Treating that result as a pass is a category error.

The rest of this article is a small, reproducible envelope. It fails the eval when patched executable lines report zero hits under the existing goldens. The method is a harness design, not a published benchmark.

Decision table before any flags

The gate is a predicate on two facts the CI already has: the golden result, and hit counts on the diff.

Golden result	Scored hunk lines	Envelope
red	any	reject; the suite already failed
green	every listed line count > 0	coverage accept only
green	some listed line counts = 0	reject
green	no executable patched lines scored	reject; suite and patch did not overlap

A coverage accept is not a merge. Digest checks, sanitizers, and must-not-compile cases still apply. This table only fills the hole where the suite never touched the rewrite.

Artifact: a coverage envelope

The envelope has five steps. Each fragment can be copied into a CI job. Replace the sample paths with the repository under test.

1. Record the hunk, not the whole tree

Coverage of untouched files is noise. The gate should score the patch.

#!/usr/bin/env bash
# record_hunk.sh — example workflow, not a measured production run
set -euo pipefail
BASE="${1:-origin/main}"
git diff -U0 "$BASE"...HEAD -- '*.cpp' '*.cc' '*.cxx' '*.h' '*.hpp' \
  > /tmp/patch.diff

python3 - <<'PY'
import re, json
hunks = {}
path = None
for line in open("/tmp/patch.diff"):
    if line.startswith("+++ b/"):
        path = line[6:].strip()
        hunks.setdefault(path, [])
        continue
    m = re.match(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", line)
    if m and path:
        start = int(m.group(1))
        length = int(m.group(2) or "1")
        hunks[path].extend(range(start, start + length))
json.dump(hunks, open("/tmp/hunk_lines.json", "w"), indent=2)
print("wrote /tmp/hunk_lines.json")
PY

The JSON map is the only input the grader trusts for what changed. Untracked generated files stay out of the map unless the operator adds them on purpose.

2. Rebuild with instrumentation

Compile the test binary the same way the goldens already compile it, plus coverage flags. Mixing -O2 and --coverage is allowed, but the flag set should stay pinned. A coverage build is a different translation. Do not compare its timings against a release binary.

cmake -S . -B build-cov \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage -fno-inline -fno-elide-constructors"
cmake --build build-cov -j"$(nproc)"

-fno-inline keeps hunk lines from vanishing into a caller. That flag is a measurement choice. It is not a claim that inlining is unsafe in production.

3. Run the existing goldens, nothing extra

Do not add tests in this step. The point is to measure the suite that already claims to accept the patch.

cd build-cov
find . -name '*.gcda' -delete
ctest --output-on-failure

If the goldens fail, the coverage gate does not run. A red suite is already a reject. The envelope only speaks when the goldens are green.

4. Map hits onto the patch

GCC can emit gcov JSON. Parse that, not the text reports. GCC 9 and later accept --json-format.

# from the build tree, after goldens
while IFS= read -r gcno; do
  gcov -b -c --json-format "$gcno" >/dev/null
done < <(find . -name '*.gcno')

A compact grader then joins hunk_lines.json with the gcov JSON.

# cov_gate.py — example grader, not an executed leaderboard
from __future__ import annotations

import json
import sys
from pathlib import Path

def load_gcov(dir: Path) -> dict[str, dict[int, int]]:
    hits: dict[str, dict[int, int]] = {}
    for p in dir.rglob("*.gcov.json"):
        data = json.loads(p.read_text())
        for f in data.get("files", []):
            src = f["file"]
            line_hits = hits.setdefault(src, {})
            for ln in f.get("lines", []):
                line_hits[int(ln["line_number"])] = int(ln.get("count", 0))
    return hits

def basename_match(hunk_path: str, gcov_path: str) -> bool:
    return Path(hunk_path).name == Path(gcov_path).name

def main() -> int:
    hunks = json.loads(Path("/tmp/hunk_lines.json").read_text())
    hits = load_gcov(Path("."))
    missed = []
    scored = 0
    for path, lines in hunks.items():
        file_hits = {}
        for gpath, lh in hits.items():
            if basename_match(path, gpath) or gpath.endswith(path):
                file_hits = lh
                break
        for n in lines:
            # skip lines gcov never lists (comments, braces-only)
            if n not in file_hits:
                continue
            scored += 1
            if file_hits[n] == 0:
                missed.append(f"{path}:{n}")
    report = {
        "scored_executable_lines": scored,
        "zero_hit_lines": missed[:50],
        "zero_hit_count": len(missed),
    }
    Path("/tmp/cov_gate.json").write_text(json.dumps(report, indent=2))
    if scored == 0:
        print("cov_gate: no executable patched lines were scored", file=sys.stderr)
        return 2
    if missed:
        print(f"cov_gate: {len(missed)} patched lines had zero hits", file=sys.stderr)
        return 1
    print("cov_gate: every scored patched line was hit")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Exit code 2 means the suite and the patch did not overlap at all. Exit code 1 means some patched executable lines stayed cold. Both are rejects under this envelope.

5. Fail closed, then keep the report

Store /tmp/cov_gate.json next to the golden digest. A later prompt pin or model swap can compare zero-hit counts the same way it compares hashes. The coverage gate is a predicate, not a leaderboard.

A toy program makes the trap obvious.

// src/codec.cpp
int decode_body(const char* p, int n) { return n > 0 ? p[0] : -1; }

int decode_frame(const char* p, int n) {
    if (n < 2) return -1;
    return (static_cast<unsigned char>(p[0]) << 8) |
           static_cast<unsigned char>(p[1]);
}

// tests/golden_codec.cpp
#include <cassert>
int decode_body(const char*, int);
int decode_frame(const char*, int);
int main() {
    const char body[] = "x";
    assert(decode_body(body, 1) == 'x');
    return 0;
}

An AI patch that "simplifies" decode_frame still leaves this golden green. The coverage gate reports zero hits on that function and rejects the eval. That is the entire point.

How this relates to models that outgrow their tests

Public debate in mid-September 2026 returned to a familiar claim: coding models already beat most developers, or that passing a test suite is the same as engineering. Those slogans are not a spec. A green golden file is evidence that the cases ran. It is not evidence that the cases touched the patch.

When a model learns the suite, it can keep outputs stable while moving behavior into untested functions. Coverage-on-the-hunk is a cheap tripwire for that pattern. It does not stop a model from writing a wrong line that a golden does execute. It stops the quieter failure: scoring a rewrite the suite never called.

Where a spare model host fits

Generating candidate patches and rebuilding with --coverage is CPU work. Local laptops throttle. A shared box that already compiles the goldens is enough.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here. This article does not name models, quote a token quota, or report a latency number. The coverage grader above runs on any Linux host with GCC, CMake, and Python 3. If the eval already lives in that toolchain, the free server option is one place to park overnight coverage gates without changing the predicate.

What this envelope does not prove

Coverage contact is not functional correctness. A line can run and still be wrong.

Inlined templates, constexpr evaluation, and macros can make gcov line numbers disagree with the diff. Basename matching is a convenience and can collide. Header-only libraries need -fkeep-inline-functions or the hunk disappears.

Death tests and exec tests may not write .gcda files. Parallel ctest without GCOV_PREFIX can clobber counters. The grader must run after a single, serialized golden pass or use per-test prefixes.

A 100% hit rate on the hunk can still hide a removed check. Digest comparison and negative-compilation cases remain necessary. This gate only fills the never-called hole.

Who should skip this gate

Skip it when the patch is comments, CMake, or pure data files. gcov will not score those paths, and a zero scored_executable_lines result is expected.

Skip it when the team already runs mutation testing on the same goldens. Mutation is stronger and more expensive. The coverage gate is the cheaper filter.

Skip it when the binary cannot be rebuilt with --coverage, including some vendor toolchains and freestanding firmware without a host test double.

Do not use the zero-hit count as a model ranking. It is a boolean envelope on a single patch. Ranking models by coverage invites the model to insert dead code that a later pass covers with a trivial call.

A pinned command the CI can copy

#!/usr/bin/env bash
set -euo pipefail
./record_hunk.sh origin/main
cmake -S . -B build-cov -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage -fno-inline"
cmake --build build-cov -j"$(nproc)"
( cd build-cov && find . -name '*.gcda' -delete && ctest --output-on-failure )
( cd build-cov && find . -name '*.gcno' -print0 | xargs -0 -n1 gcov -b --json-format >/dev/null )
python3 cov_gate.py

Pin the compiler version next to this script. Coverage IR changes across GCC releases. A gate that flips because the image moved from GCC 12 to GCC 13 is a harness bug, not a model regression.

The Friday merge in the opening would have failed step 5. The goldens would still have been green. The report would have listed src/codec.cpp lines inside decode_frame as zero-hit. That is a small, boring, and sufficient reason to reject the eval.

Same Goldens, New Question: A Prompt-Pin Manifest for AI C++ Evals

Finley Li — Wed, 16 Sep 2026 03:48:53 +0000

A C++ library team merged a “clearer” system prompt on a Thursday. The golden cases stayed green. The compiler stayed quiet. A reviewer opened the next model patch on Monday and found using namespace std; in a public header, dropped const on three accessors, and a helper that threw from a destructor path the tests never entered.

Nothing in the scorecard had moved. The tests still asked last month’s question. The prompt no longer did.

Golden cases measure the answer. They do not measure whether the question is still the same question. Prompt edits are dependency changes. An eval that does not pin the prompt will report stability while the instruction surface drifts.

This article describes a small eval identity for AI C++ patches. The harness refuses to score a candidate unless a manifest of hashes matches the files on disk. The method stays useful if every product name is stripped out. A later section notes where a free remote runner can sit in the loop.

What the scorecard usually forgets

Most AI C++ harnesses version three things: the source under test, the compiler flags, and the expected I/O. That is necessary. It is not sufficient.

A prompt rewrite can keep every assert passing while changing which defects the model is even asked to avoid. Style rules vanish. Exception-safety language is softened. The model is told to prefer brevity. The goldens still check that add(2, 2) returns 4.

Silent question drift looks like a quality win. It is a spec change without a bump. Leaderboard rows from two prompt texts are then treated as one experiment. They are not.

Eval identity, not a vibe

Treat the eval as a tuple. If any element changes, the run is a different experiment.

Prompt text, including the system message and the user template.
Golden cases: inputs, expected outputs, and skip flags.
Grader: compiler invocation, sanitizers, and assertion driver.
Toolchain pins: compiler version string, language standard, and flag vector.
Candidate patch: the model output under test.

Hash items 1–4 into a committed manifest. Compare those hashes before any score is written. Fail closed on mismatch. Record the candidate hash only after the identity check passes.

The rest of this article implements that check with ordinary files. No database. No dashboard. The scripts below are a proposed tree, not a claim that they have been run against a private corpus.

Artifact: `eval_manifest.json`

Store the manifest next to the prompt, not in a wiki. Empty hashes are placeholders. A freeze script fills them. CI rejects a run whose live hashes do not match the committed pins.

{
  "schema": "eval-identity/v1",
  "suite": "ai-cpp-patches",
  "pins": {
    "prompt_sha256": "",
    "goldens_sha256": "",
    "grader_sha256": "",
    "toolchain_sha256": ""
  },
  "toolchain": {
    "cxx": "g++",
    "cxx_std": "c++20",
    "cxxflags": ["-O2", "-Wall", "-Wextra", "-Werror", "-fsanitize=address,undefined"]
  },
  "policy": {
    "on_mismatch": "fail_closed",
    "allow_prompt_bump": false
  }
}

allow_prompt_bump stays false in CI. A human freeze is the only way the prompt pin moves. That pause is the whole gate.

Layout on disk

Keep the suite boring and grepable. One prompt file. One golden directory. One grader tree. One captured toolchain dump.

eval/
  prompt.txt
  goldens/
    001_add.json
    002_overflow.json
    003_header_hygiene.json
  grader/
    run.sh
    catch_main.cpp
  toolchain.txt
  eval_manifest.json
  freeze.py
  check_identity.py
  score.py

toolchain.txt is a captured g++ -v dump plus the exact flag vector. Do not reconstruct it from memory during a run. Two machines with “the same g++” still diverge on patch level, default search paths, and sanitizer runtimes.

Step 1 — Freeze the identity

freeze.py hashes canonical bytes. The script does not pretty-print JSON before hashing goldens. It hashes the files as committed, in sorted path order, so a renamed case still moves the tree digest.

#!/usr/bin/env python3
"""Freeze eval identity. Proposed helper; run it on a real tree before trusting pins."""
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def sha256_tree(dirpath: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(dirpath.rglob("*")):
        if not p.is_file():
            continue
        rel = p.relative_to(dirpath).as_posix().encode()
        h.update(rel)
        h.update(b"\0")
        h.update(p.read_bytes())
        h.update(b"\0")
    return h.hexdigest()

def main() -> None:
    man_path = ROOT / "eval_manifest.json"
    man = json.loads(man_path.read_text())
    man["pins"]["prompt_sha256"] = sha256_file(ROOT / "prompt.txt")
    man["pins"]["goldens_sha256"] = sha256_tree(ROOT / "goldens")
    man["pins"]["grader_sha256"] = sha256_tree(ROOT / "grader")
    man["pins"]["toolchain_sha256"] = sha256_file(ROOT / "toolchain.txt")
    man_path.write_text(json.dumps(man, indent=2) + "\n")
    print("froze", man["pins"])

if __name__ == "__main__":
    main()

Run it only when a spec bump is intended.

python3 eval/freeze.py
git add eval/eval_manifest.json eval/prompt.txt eval/goldens eval/grader eval/toolchain.txt
git diff --cached eval/eval_manifest.json

The pin diff is the changelog. If the prompt hash moved and the goldens hash did not, the question changed and the tests did not. That pair is the reason the harness exists.

Step 2 — Fail closed before any model call

check_identity.py recomputes live hashes and compares. Scoring never starts on mismatch. Generation can wait. A wrong question is cheaper to catch before tokens are spent than after a green cell is published.

#!/usr/bin/env python3
from __future__ import annotations

import json
import sys
from pathlib import Path
from freeze import ROOT, sha256_file, sha256_tree

def live_pins() -> dict[str, str]:
    return {
        "prompt_sha256": sha256_file(ROOT / "prompt.txt"),
        "goldens_sha256": sha256_tree(ROOT / "goldens"),
        "grader_sha256": sha256_tree(ROOT / "grader"),
        "toolchain_sha256": sha256_file(ROOT / "toolchain.txt"),
    }

def main() -> int:
    man = json.loads((ROOT / "eval_manifest.json").read_text())
    expected = man["pins"]
    got = live_pins()
    bad = {
        k: {"expected": expected[k], "got": got[k]}
        for k in expected
        if expected[k] != got[k]
    }
    if bad:
        print("EVAL_IDENTITY_MISMATCH")
        print(json.dumps(bad, indent=2))
        return 2
    print("EVAL_IDENTITY_OK", json.dumps(got, sort_keys=True))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Wire it as a gate, not as a log line.

python3 eval/check_identity.py && python3 eval/score.py --patch candidate.diff

A typical mismatch looks like this after someone “just clarified” prompt.txt:

EVAL_IDENTITY_MISMATCH
{
  "prompt_sha256": {
    "expected": "9c1e0c6c0f3a7b1d8a2e4f55c0b91aa0e3c6d2f1a7b84c0d9e1f2a3b4c5d6e7f",
    "got":      "0b77a1d4e2c98f06b3aa4419d5c0e18f27d6b4c1a9e03f55c8d2a176b0e4c9aa"
  }
}

Those hex strings are examples of shape, not measurements from a private run. The important part is the exit code. CI treats 2 as a broken spec, not as a model failure.

Step 3 — Keep goldens that the new question can still fail

Prompt pins without hostile goldens only detect file drift. They do not detect a weaker instruction. Add at least one golden that encodes a rule the prompt claims to enforce. Header hygiene is a reliable example because tests almost never look at it.

goldens/003_header_hygiene.json:

{
  "id": "003_header_hygiene",
  "source": "box.hpp",
  "must_compile": true,
  "compile_unit": "const Box b{1}; auto x = b.value(); (void)x;",
  "forbid_regex": [
    "using namespace std;",
    "using namespace std::",
    "catch\\s*\\(\\s*\\.\\.\\.\\s*\\)"
  ]
}

A grader compiles the unit and scans the patch text. The regex layer is crude. It still catches the Thursday rewrite that stopped mentioning public headers. Put the scan in score.py so the grader tree hash moves when the check changes.

grader/run.sh stays the only compiler entry. That keeps the tree hash meaningful.

#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
python3 "$ROOT/check_identity.py"
CXX="${CXX:-g++}"
src="$1"
"$CXX" -std=c++20 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined \
  -c "$src" -o /tmp/candidate.o

Capture the live toolchain once per machine image.

{
  echo "# toolchain pin"
  g++ -v 2>&1
  echo "# flags"
  echo "-std=c++20 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined"
} > eval/toolchain.txt

Label the snippets as a proposed envelope. They are not a published benchmark and they are not a claim about any model’s pass rate.

Step 4 — Record the candidate only after identity holds

score.py should refuse to write last_result.json unless check_identity returned zero. Then hash the patch itself. Later spreadsheet rows become comparable only when the pin block matches.

# Proposed fragment, not a full product.
import hashlib, json, subprocess, sys
from pathlib import Path

patch_path = Path(sys.argv[1])
patch = patch_path.read_bytes()
subprocess.check_call([sys.executable, "eval/check_identity.py"])
pins = json.loads(Path("eval/eval_manifest.json").read_text())["pins"]
result = {
    "eval_manifest": pins,
    "patch_sha256": hashlib.sha256(patch).hexdigest(),
    "verdict": "unscored",
}
Path("eval/last_result.json").write_text(json.dumps(result, indent=2) + "\n")

Mixing rows from two prompt hashes is a methodology error. The file makes that error visible. A green cell under pin set A is not a data point under pin set B.

Decision table: what to bump

Change	Prompt pin	Goldens pin	Grader pin	Toolchain pin	Action
Wording tweak in `prompt.txt`	yes	no	no	no	Freeze, re-run the full suite, do not compare scores to the previous pin set
New golden for header hygiene	no	yes	maybe	no	Freeze, treat as spec expansion
Switch `-O0` to `-O2`	no	no	no	yes	Freeze; scores are a different experiment
ASan added to `run.sh`	no	no	yes	maybe	Freeze; old “green” is not transferable
Model sampled again, same files	no	no	no	no	Score and compare
Prompt and goldens both edited	yes	yes	no	no	Freeze once, write a note that two spec axes moved together

The dangerous row is the first one. Teams skip it because the tests still pass. The manifest exists to make that skip loud. Two axes moving in one commit is legal. It is also easy to misread later, so the note belongs in the same commit as the freeze.

Where a free remote runner fits

Local identity checks are cheap. Generating candidate patches is not. Operators who already iterate prompts against a C++ suite sometimes need a place to sample models without standing up a GPU box for every wording tweak.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that, as supplied by the operator of this account, offers free model access and a free server option. Those two availability claims are the only product facts used here. This article does not assert model names, token quotas, hardware, uptime, or permanence. Dashboard numbers go stale. Re-read the project page before planning capacity.

In this workflow the remote runner is optional. The identity files stay in git. A generation job may run off-laptop. The score step still happens only after check_identity.py passes on the same commit. If the free server is used to produce candidate.diff, copy that file back and hash it locally. Do not let a remote workspace silently edit prompt.txt.

Readers who want that runner can look up the public repository when they need it. The harness above does not depend on it.

Limitations

The manifest does not understand semantics. Two prompts can hash differently and mean the same thing. Two prompts can hash identically after a well-placed Unicode lookalike. Hashing is identity, not equivalence. Operators who need semantic diff still have to read the prompt.

Regex forbid-lists are bypassable. A model can split using namespace std; across a line continuation or hide a throw in a macro. A single compile unit does not prove ABI stability, thread safety, or that RSS stayed flat. Those are other envelopes. They do not replace a prompt pin, and a prompt pin does not replace them.

This approach is a poor fit for throwaway playground prompts, for suites without committed goldens, and for teams that want a single scalar “model quality” number across prompt versions. It is the wrong tool if the compiler cannot be pinned. Unpinned toolchains make the toolchain hash theater.

Do not use the freeze script as a ritual that rubber-stamps a weaker prompt. If the prompt hash moved, re-read the goldens. Add a case that would have failed under the old instruction. Then freeze.

What “green” is allowed to mean

After this harness is in place, a green cell means four files on disk still match a committed tuple, and the candidate survived that tuple’s grader. It does not mean the prompt is better. It does not mean last week’s leaderboard row is comparable.

Prompt edits become visible spec bumps. Silent question drift has to walk through freeze.py. That is a small gate. For C++ codegen evals, it is usually the gate that was missing.

The Placebo Patch: Measure Your C++ Eval Harness's Noise Floor Before You Trust a Regression

Finley Li — Mon, 14 Sep 2026 21:34:38 +0000

A four-person team running an AI patch loop on a C++ service added a merge gate last quarter. One rule: if any golden case changes verdict, the candidate patch is rejected. On a Tuesday morning the gate rejected a patch that renamed buf_len to buffer_length in a single function.

Nothing changed at runtime. The rename moved one assertion message, one stack trace line, and the interleaving of two log lines. The harness reported three regressions, and all three were artifacts.

The gate was measuring itself. What it lacked was a control arm, and a control arm is cheap to build.

The number nobody records

Every eval harness has a noise floor: the rate at which verdicts change when nothing semantic changes. That number is a property of a triple — harness, toolchain, machine — and it moves whenever any leg moves.

Teams usually discover it the expensive way. A regression is reported, a developer spends a day bisecting, and the conclusion is "flaky case, rerun it." Then the gate loses its authority, and real regressions get rerun to death alongside the imaginary ones.

A placebo corpus fixes this in about eighty lines of glue.

Step 1 — Build a placebo corpus

A placebo patch is a semantically neutral edit whose only job is to expose harness brittleness. Five classes cover most of the ground.

ID	Placebo patch	What it perturbs
P0	null patch (identical tree)	pure run-to-run variance
P1	comment-only insertion	nothing intentional
P2	local variable rename in one TU	debug info, assertion messages, symbol layout
P3	reorder `#include` lines in one file	declaration order, token positions
P4	`clang-format` applied to one function	line and column numbers
P5	add an unused `static` helper	symbol table, link order

P0 is the honest baseline. P1 through P5 should be invisible; when they are not, the case is reacting to shape rather than behavior.

Store them as unified diffs under placebo/patches/, named so they sort predictably.

placebo/
  patches/
    00-null.patch        # empty diff, applied as a no-op
    01-comment.patch
    02-rename.patch
    03-include-order.patch
    04-format.patch
    05-dead-static.patch
  out/                   # one JSON per (patch, run)

Writing P2 and P4 by hand across a 2,000-line translation unit is dull work with a sharp oracle attached: if a supposedly neutral edit accidentally changes behavior, the first matrix run exposes it, not a reviewer. That self-verifying property is what makes the variant generation a reasonable job for free model access.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness in this setup generates its neutral variants through MonkeyCode's free model access. Quota, throughput, and availability terms belong to the project's own documentation and should be read there rather than from any number quoted in an article.

Step 2 — Run the matrix

The harness needs one property: it must emit machine-readable verdicts, one entry per case. A --json flag on stdout is enough.

#!/usr/bin/env bash
# placebo/run_matrix.sh
set -euo pipefail

K="${K:-5}"                       # runs per patch
OUT=placebo/out
mkdir -p "$OUT"

for patch in placebo/patches/*.patch; do
  name=$(basename "$patch" .patch)
  for run in $(seq 1 "$K"); do
    if [ "$name" != "00-null" ]; then
      git apply "$patch"
    fi
    cmake --build build -j"$(nproc)" >/dev/null
    ./build/harness --json > "$OUT/$name.$run.json"
    git checkout -- .             # drop the placebo patch
  done
done

Two details decide whether the data means anything. Build with the same compiler, flags, and machine for the whole matrix, because a mixed matrix measures the toolchain. Keep K odd if the report is going to talk about a modal verdict.

Step 3 — Turn verdicts into a per-case admission report

The matrix is only raw material. The useful output is one row per golden case with two booleans: did it disagree with itself across runs, and did it flip under a neutral patch?

#!/usr/bin/env python3
"""noise_report.py - admission verdict per golden case."""
import collections, glob, json, pathlib

REF_PATCH = "00-null"

by_patch = collections.defaultdict(list)
for path in sorted(glob.glob("placebo/out/*.json")):
    record = json.loads(pathlib.Path(path).read_text())
    by_patch[record["patch"]].append(record)

cases = sorted(next(iter(by_patch.values()))[0]["cases"])
report = {}

for case in cases:
    reference = by_patch[REF_PATCH][0]["cases"][case]["status"]

    flake = any(
        len({r["cases"][case]["status"] for r in runs}) > 1
        for runs in by_patch.values()
    )
    placebo_sensitive = any(
        reference not in {r["cases"][case]["status"] for r in runs}
        for patch, runs in by_patch.items()
        if patch != REF_PATCH
    )

    report[case] = {
        "flake": flake,
        "placebo_sensitive": placebo_sensitive,
        "admission": ("gate" if not flake and not placebo_sensitive
                      else "advisory" if not flake
                      else "quarantine"),
    }

pathlib.Path("placebo/noise_report.json").write_text(
    json.dumps(report, indent=2, sort_keys=True))
excluded = [c for c, r in report.items() if r["admission"] != "gate"]
print(f"{len(excluded)}/{len(cases)} cases excluded from the merge gate")

Three outcomes fall out of the two booleans, and each one has a different owner.

run-to-run flake	placebo sensitive	admission	action
no	no	`gate`	eligible to block merges
no	yes	`advisory`	normalize the observable, or stop blocking on it
yes	either	`quarantine`	fix or delete on a deadline, never gate

The middle row is what pays for the exercise. A case that never flakes but flips when a function is reformatted is not measuring behavior; it is measuring line numbers.

Step 4 — Gate on the promotable set only

The gate script stays deliberately small. It reports a regression only when the case is in the gate set, and it prints everything it suppressed.

#!/usr/bin/env python3
"""gate.py CANDIDATE.json REFERENCE.json"""
import json, sys

noise = json.load(open("placebo/noise_report.json"))
candidate = json.load(open(sys.argv[1]))
reference = json.load(open(sys.argv[2]))

gateable = {c for c, r in noise.items() if r["admission"] == "gate"}
changed = {
    c for c in candidate["cases"]
    if candidate["cases"][c]["status"] != reference["cases"][c]["status"]
}

print(json.dumps({
    "regressions": sorted(changed & gateable),
    "suppressed_noisy": sorted(changed - gateable),
}, indent=2))

sys.exit(1 if changed & gateable else 0)

suppressed_noisy still needs to be visible. A case that changed verdict but is not gateable is exactly the information a human needs to decide whether to repair the case instead of the patch.

Step 5 — Re-measure instead of inheriting

The noise floor is not a constant, so the report has an expiry date. Re-run the matrix when the compiler version, the harness, or the CI machine class changes.

A five-patch by five-run matrix multiplies suite time by twenty-five. That makes it a nightly job rather than a per-pull-request job. The operator runs it on MonkeyCode's free server option, which is a defensible use of idle cycles: the control arm is slow, deterministic, and produces nothing user-visible.

If the suite is too slow even for that, cut K to three and say so in the report. A wide interval that is labeled honestly beats a narrow one that is invented.

Limitations

The control arm measures variance, not correctness. If every golden case encodes the same wrong expectation, all placebo runs agree with each other, and the gate stays confidently wrong.

Networked or timing-dependent cases belong outside the placebo arm. Five runs over a live HTTP dependency measure the network's mood, not the harness's stability. Keep the corpus on deterministic unit-level cases.

The gateable set can shrink. Track its size over time. A rough rule of thumb: if the promotable set falls below about sixty percent of the corpus, the corpus needs rewriting rather than more quarantine.

Who should skip this

A solo project with one fast, deterministic suite and no automated patch loop gains little here, because there is no gate to defend.

Anyone without a reproducible reference build should fix that first. The reference run is the baseline that every comparison in the report is drawn against.

Teams whose individual cases take minutes cannot run P x K matrices at all. They should start with P0 alone and one run per patch, just to find out whether the floor is measurable on their hardware.

Closing

A gate that fires on a variable rename costs more trust than it buys. Measuring the floor takes one script, one nightly job, and a decision table, and it converts "flaky case, rerun it" into a number somebody can act on. Nothing in this workflow requires a paid runner — the whole experiment above was run on MonkeyCode's free tier, which is also a low-stakes way to find out whether the control arm earns a permanent slot in CI.

Must-Not-Compile Is a Spec: A Negative-Compilation Envelope for AI C++ Patches

Finley Li — Sun, 13 Sep 2026 19:52:52 +0000

A payments adapter review landed on a Tuesday afternoon. The model had produced a three-file diff. The unit suite returned green in twelve seconds, and the reviewer merged it.

A week later a plugin author constructed TokenHandle from a raw integer. The original header had blocked that path with an explicit private constructor and a deleted converting constructor. The patch had moved the constructor to public so a test the model added in the same diff would compile.

Runtime assertions never saw the leak. The type had become a wider type while every EXPECT_* still passed. Positive tests reward compilation. They do not record the compilation that must keep failing.

The hole in treating build success as progress

Most AI C++ patch evals stop at three bits. The compiler exit code is zero. The test binary exits zero. Sometimes stdout is hashed. That triad is a spec for “does something run.” It is not a spec for “the public surface did not grow.”

Language-level contracts live in the failures. Deleted special members, explicit constructors, private factories, requires clauses, and static_assert checks exist so that certain translation units never become programs. An edit that turns those failures into successes is a behavior change. Graders that treat every successful compile as progress will score that edit as a win.

The envelope below treats selected compile failures as first-class oracles. It is a proposed, reproducible harness. It is not a published leaderboard and it does not report model scores.

What the envelope records

A negative-compilation envelope is a directory of tiny translation units plus a grader. Each unit includes the public header under test and then performs one operation the API is supposed to reject.

The grader keeps three buckets:

Must-build units. These compile, link, and run. They are the ordinary happy path.
Must-fail units. The compiler must return a non-zero status. A diagnostic substring may be pinned.
Must-still-fail after the patch. If a must-fail unit starts compiling, the patch is a surface regression even when runtime tests stay green.

The third bucket is the point of the harness. AI patches frequently unstick a test by weakening a type.

Verdict table

must-build after patch	must-fail after patch	Verdict
compiles and tests pass	still fails to compile	accept
compiles and tests pass	now compiles	surface regression
no longer compiles	still fails	patch broke the build
runtime assertion fails	still fails	functional regression
compiles and tests pass	fails, but diagnostic text drifted	toolchain mismatch, not a model win

A minimal library under test

The example is a move-only handle. Copy is deleted. Integer construction is not part of the public surface. The snippets are labeled examples, not extracted production code.

// include/token_handle.hpp
#pragma once
#include <cstdint>
#include <utility>

class TokenHandle {
public:
  static TokenHandle from_trusted(std::uint64_t id) noexcept {
    return TokenHandle(id);
  }

  TokenHandle(const TokenHandle&) = delete;
  TokenHandle& operator=(const TokenHandle&) = delete;

  TokenHandle(TokenHandle&& other) noexcept : id_(other.id_) {
    other.id_ = 0;
  }

  TokenHandle& operator=(TokenHandle&& other) noexcept {
    if (this != &other) {
      id_ = other.id_;
      other.id_ = 0;
    }
    return *this;
  }

  std::uint64_t id() const noexcept { return id_; }

private:
  explicit TokenHandle(std::uint64_t id) noexcept : id_(id) {}
  std::uint64_t id_;
};

A positive unit that must keep compiling and running:

// cases/pos_move_ok.cpp
#include "token_handle.hpp"
#include <cassert>

int main() {
  auto a = TokenHandle::from_trusted(42);
  auto b = std::move(a);
  assert(b.id() == 42);
  return 0;
}

Two negative units that must keep failing. These files are not broken tests. They are the contract.

// cases/neg_copy.cpp
#include "token_handle.hpp"

int main() {
  auto a = TokenHandle::from_trusted(1);
  TokenHandle b = a; // copy must remain deleted
  (void)b;
}

// cases/neg_from_int.cpp
#include "token_handle.hpp"

int main() {
  TokenHandle h = 7; // converting construction must remain impossible
  (void)h;
}

Manifest and grader

Pin the compiler, the language mode, and the include path. A JSON manifest keeps the envelope reviewable without extra YAML dependencies.

{
  "compiler": ["g++", "-std=c++20", "-Wall", "-Wextra", "-Werror"],
  "include": "include",
  "must_build": ["cases/pos_move_ok.cpp"],
  "must_fail": [
    {"path": "cases/neg_copy.cpp", "diagnostic": "deleted"},
    {"path": "cases/neg_from_int.cpp", "diagnostic": ""}
  ]
}

Proposed grader in Python 3. It shells out to the real compiler. It does not parse C++.

#!/usr/bin/env python3
"""Negative-compilation envelope. Proposed example, not a published score."""
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
BIN = Path("/tmp/negcomp_a.out")


def compile_unit(compiler, include, src):
    cmd = compiler + [f"-I{include}", str(src), "-o", str(BIN)]
    return subprocess.run(cmd, capture_output=True, text=True)


def main():
    spec = json.loads((ROOT / "envelope.json").read_text())
    compiler = spec["compiler"]
    include = str(ROOT / spec["include"])
    failures = []

    for rel in spec["must_build"]:
        proc = compile_unit(compiler, include, ROOT / rel)
        if proc.returncode != 0:
            failures.append("must_build failed: " + rel + "\n" + proc.stderr)
            continue
        run = subprocess.run([str(BIN)], capture_output=True, text=True)
        if run.returncode != 0:
            failures.append("runtime failed: " + rel)

    for item in spec["must_fail"]:
        proc = compile_unit(compiler, include, ROOT / item["path"])
        if proc.returncode == 0:
            failures.append("SURFACE REGRESSION: " + item["path"] + " now compiles")
            continue
        needle = item.get("diagnostic") or ""
        blob = (proc.stderr or "") + (proc.stdout or "")
        if needle and needle not in blob:
            failures.append(
                "must_fail, diagnostic missed " + repr(needle) + ": " + item["path"]
            )

    if failures:
        print("\n".join(failures))
        return 1
    print("envelope: must-build green, must-fail still failing")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run the grader before and after a candidate patch:

python3 grade_envelope.py
git apply /tmp/model.patch
python3 grade_envelope.py; echo "grade_exit=$?"
git checkout -- include/token_handle.hpp

A model that publishes the constructor to satisfy neg_from_int.cpp will flip that unit from fail to success. The second run prints SURFACE REGRESSION. Exit code zero on the unit tests is then irrelevant.

Numbered workflow for an AI C++ patch eval

Freeze envelope.json in the same commit as the headers it describes. Do not let the model edit the must-fail list in the same turn as the production sources.
Generate the patch in an isolated worktree. Keep envelope files read-only to the agent, or reject diffs that touch cases/neg_*.cpp.
Apply the patch. Rebuild must-build units. Run them.
Recompile every must-fail unit. Require a non-zero compiler status. If a diagnostic substring is pinned, require it.
Accept the patch only when all three hold: must-build still builds, tests still pass, must-fail still fails.
Record g++ --version or clang++ --version next to the envelope. Diagnostic text is not portable across major versions.

Step 2 is the lock. Oracle suites fail when the model rewrites the oracle. Negative units fail the same way if they are writable.

Where generation and grading split

Generating the candidate patch and compiling the envelope are separate jobs. The first is a language-model call. The second is a compiler farm.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free-tier model access can sit on the generate side of that split, and its free server option can sit on the grade side so the must-fail matrix does not occupy a laptop that is already compiling the product. Those are availability options, not quotas, model names, or latency claims. Teams that already pin a CI image should keep grading inside that image. The envelope does not depend on a particular vendor.

Maintainers who already score AI C++ patches can drop envelope.json onto an idle runner and compare pre-patch and post-patch compiler statuses. That is the only product-shaped suggestion in this write-up.

Compiler notes that break naive graders

GCC and Clang do not emit the same words for a deleted constructor. Pinning diagnostic to "deleted" can fail on a compiler that says use of deleted function versus one that says attempting to reference a deleted function. Status-only checks are more portable and weaker.

CMake try_compile is an alternative to the Python loop. The inversion is easy to misread in review.

try_compile(NEG_COPY_BUILT
  ${CMAKE_BINARY_DIR}/neg_copy
  SOURCES ${CMAKE_SOURCE_DIR}/cases/neg_copy.cpp
  CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${CMAKE_SOURCE_DIR}/include"
)
if(NEG_COPY_BUILT)
  message(FATAL_ERROR "neg_copy compiled; public surface widened")
endif()

try_compile returns true on success. The envelope treats true as failure. The Python grader makes that inversion obvious in one if proc.returncode == 0.

Concepts and SFINAE change the picture again. A constraint failure may be a substitution failure inside a template, not a hard error at top-level main. Negative units should call the constrained function in a context that forces a hard error. Otherwise the unit compiles and the envelope lies.

Limitations

The envelope does not prove API stability. It proves that a finite list of snippets still fail. A model can widen the surface along an unlisted axis: a new converting operator, a friend declaration, or a defaulted copy after a refactor into a trivially copyable layout.

Diagnostic substring matching is brittle across compiler upgrades. Status-only matching is coarse. Both need a pinned toolchain.

Compile-fail tests do not catch runtime undefined behavior that still compiles. They do not replace sanitizers, ABI dumps, or human review.

Header changes that only affect inlining can still be correct while flipping a must-fail unit if the unit was over-specified. Treat false positives as envelope bugs until a human reads the diagnostic.

Who should not use this

Do not adopt the envelope for patches that never touch headers or type definitions. A leaf .cpp bugfix has no compile-fail contract to protect.

Do not treat it as a security control. A private constructor is not a sandbox.

Do not run a single-compiler manifest as the only gate on a codebase that ships three toolchains. Split envelopes per compiler, or check status only.

Skip it when the team cannot keep cases/neg_*.cpp out of the model’s writable set. A model that deletes the negative units will score perfectly.

Closing

Runtime green is a weak statement about C++. The language spends a large fraction of its design budget on programs that must not exist. An eval that never asks the compiler to refuse those programs will systematically prefer models that widen types until the tests compile.

Keep the must-fail list small, versioned, and unwritable to the agent. Grade the patch against that list with the same compiler the project ships. The merge that only looks at ctest will keep promoting public constructors that were never in the spec.

MonkeyCode provides free models that can run this workflow.

Asserts Passed, RSS Doubled: An rusage Envelope for AI C++ Patches

Finley Li — Sat, 12 Sep 2026 17:51:49 +0000

The following incident is a composite of a common failure mode, not a claim about a named production system. A daemon team merged a one-line cache suggested by a coding model. Every unit test stayed green. The next on-call graph told a different story: resident set size climbed through the night, and a previously quiet process started swapping.

The patch replaced a linear scan with an unbounded unordered_map. Correctness tests used fixtures of a few dozen keys. Nothing in the pipeline asked how much memory or CPU a correct answer was allowed to spend.

Functional goldens remain necessary. They are not a resource spec.

Cost is part of the contract

AI C++ patches fail in noisy ways and in quiet ways. Missing headers and invented helpers break the build. Resource regressions usually do not. The binary still links. Asserts still pass. The process simply gets heavier on every call.

An eval harness that only checks exit codes will score the heavy patch as a win. A second artifact has to sit beside the tests: a committed resource envelope. The envelope is JSON next to the fixtures. It records ceilings, not averages. If a candidate exceeds a ceiling, the patch fails even when every assert passes.

This article is a worked example. It is not production telemetry. Numbers in the sample envelope are illustrative ceilings for a tiny driver. They are not measurements from a deployed service.

What the envelope records

The file stays small on purpose. Three fields catch a surprising number of helpful-looking model edits on Linux.

max_rss_kb — high-water resident set from child rusage (ru_maxrss, kilobytes on Linux).
max_user_us — user CPU time in microseconds.
timeout_s — wall-clock cap; a hang becomes a reject before RSS is even read.

Optional later fields include max_minflt and max_inblock. A field is worth adding only after a real incident names it. The envelope is a tripwire. Tripwires are allowed to be coarse.

Ceilings have to be recorded on the same class of host that will grade candidates. Copying a laptop envelope onto a shared eval box is a factory for false failures. Host class is part of the spec, the same way -O2 is part of the spec.

Worked example: the cache that never evicted

The sample library looks up scores by id. The baseline scans a small vector. Tests pass. A coding model then fixes the scan by filling a process-lifetime unordered_map with no eviction. Lookups still return the right integer. RSS does not stay still.

Header under test

#pragma once
#include <cstdint>
#include <unordered_map>
#include <vector>

struct ScoreRow {
  std::uint32_t id;
  int score;
};

inline int lookup_scan(const std::vector<ScoreRow>& rows, std::uint32_t id) {
  for (const auto& row : rows) {
    if (row.id == id) return row.score;
  }
  return -1;
}

// Anti-pattern used to demonstrate the envelope.
// Models emit this when a prompt says "make it faster" and never says "bound the table."
inline int lookup_unbounded_cache(const std::vector<ScoreRow>& rows,
                                  std::uint32_t id) {
  static std::unordered_map<std::uint32_t, int> cache;
  auto it = cache.find(id);
  if (it != cache.end()) return it->second;
  int score = lookup_scan(rows, id);
  cache.emplace(id, score);
  return score;
}

The unbounded cache is deliberate. It is the silent case: right answers, growing weight.

Driver the harness actually times

#include "score_lookup.hpp"
#include <cstdint>
#include <iostream>
#include <string>
#include <vector>

int main(int argc, char** argv) {
  const bool use_cache = (argc > 1 && std::string(argv[1]) == "cache");
  std::vector<ScoreRow> rows;
  rows.reserve(256);
  for (std::uint32_t i = 0; i < 256; ++i) {
    rows.push_back({i, static_cast<int>(i % 17)});
  }
  long long acc = 0;
  for (int round = 0; round < 40000; ++round) {
    std::uint32_t id = static_cast<std::uint32_t>((round * 17u) % 100000u);
    acc += use_cache ? lookup_unbounded_cache(rows, id)
                     : lookup_scan(rows, id);
  }
  std::cout << acc << "\n";
  return 0;
}

Both modes print one integer and exit 0. Functional equality is the trap. The envelope is the distinction.

Committed ceilings

{
  "cmd": ["./driver"],
  "timeout_s": 5,
  "max_rss_kb": 8192,
  "max_user_us": 800000,
  "notes": "Illustrative Linux ceilings for this driver. Re-record on the eval host."
}

Rebuild the envelope when the host, the compiler, or the fixture size changes. Treat that rebuild as a spec change. The grader should not raise ceilings on its own.

The rusage grader

The grader lives outside the candidate on purpose. A patch that prints RSS: ok is not evidence. Python 3 plus the resource module is enough on Linux. The parent process must not wait on other children before the timed run, or RUSAGE_CHILDREN will mix histories.

Label the script as a template. It is not certified on every distro. GNU /usr/bin/time -f '%x %U %M' is a valid alternative when Python is unwelcome. The committed artifact should still be envelope.json, not a screenshot of a terminal.

#!/usr/bin/env python3
"""Linux rusage envelope grader for C++ patch candidates. Worked example only."""
from __future__ import annotations

import argparse
import json
import resource
import subprocess
import sys
from pathlib import Path


def measure(cmd: list[str], timeout_s: float) -> dict:
    try:
        completed = subprocess.run(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            timeout=timeout_s,
            check=False,
        )
        timed_out = False
        returncode = int(completed.returncode)
        stdout = completed.stdout
        stderr = completed.stderr
    except subprocess.TimeoutExpired as exc:
        timed_out = True
        returncode = 124
        stdout = exc.stdout or b""
        stderr = exc.stderr or b""
    ru = resource.getrusage(resource.RUSAGE_CHILDREN)
    return {
        "timed_out": timed_out,
        "returncode": returncode,
        "stdout": stdout.decode("utf-8", errors="replace"),
        "stderr": stderr.decode("utf-8", errors="replace"),
        "max_rss_kb": int(ru.ru_maxrss),
        "user_us": int(ru.ru_utime * 1_000_000),
    }


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cmd", nargs="+", required=True)
    parser.add_argument("--envelope", type=Path, required=True)
    parser.add_argument("--record", action="store_true")
    parser.add_argument("--force", action="store_true")
    parser.add_argument("--grade", action="store_true")
    parser.add_argument("--expect-stdout", type=Path)
    args = parser.parse_args()
    if args.record == args.grade:
        print("exactly one of --record or --grade is required", file=sys.stderr)
        return 2

    timeout_s = 5.0
    if args.grade:
        env = json.loads(args.envelope.read_text())
        timeout_s = float(env.get("timeout_s", 5))

    result = measure(args.cmd, timeout_s)

    if args.record:
        if args.envelope.exists() and not args.force:
            print("refusing to overwrite envelope without --force", file=sys.stderr)
            return 2
        # Slack is explicit in the file. Hidden slack inside the grader is a bug.
        payload = {
            "cmd": args.cmd,
            "timeout_s": timeout_s,
            "max_rss_kb": int(result["max_rss_kb"] * 1.3) + 1024,
            "max_user_us": int(result["user_us"] * 1.5) + 50_000,
            "notes": "ceilings = one baseline run plus committed slack; re-record on host change",
        }
        args.envelope.write_text(json.dumps(payload, indent=2) + "\n")
        print(json.dumps({"mode": "record", "measured": {
            "max_rss_kb": result["max_rss_kb"],
            "user_us": result["user_us"],
            "returncode": result["returncode"],
        }, "wrote": str(args.envelope)}))
        return 0

    env = json.loads(args.envelope.read_text())
    reasons = []
    if result["timed_out"]:
        reasons.append("timeout")
    if result["returncode"] != 0:
        reasons.append(f"exit_{result['returncode']}")
    if result["max_rss_kb"] > int(env["max_rss_kb"]):
        reasons.append("rss")
    if result["user_us"] > int(env["max_user_us"]):
        reasons.append("user_cpu")
    if args.expect_stdout and result["stdout"] != args.expect_stdout.read_text():
        reasons.append("stdout")
    verdict = "reject" if reasons else "accept"
    print(json.dumps({
        "mode": "grade",
        "verdict": verdict,
        "reasons": reasons,
        "measured": {
            "returncode": result["returncode"],
            "max_rss_kb": result["max_rss_kb"],
            "user_us": result["user_us"],
            "timed_out": result["timed_out"],
        },
    }, indent=2))
    return 1 if reasons else 0


if __name__ == "__main__":
    raise SystemExit(main())

Linux reports ru_maxrss in kilobytes. macOS reports bytes. This template is Linux-shaped on purpose. Mixing units without a platform tag will either reject every patch or accept every leak.

Numbered workflow

Run the loop the same way on every candidate. Drift in the loop is how silent cost bugs get a passing grade.

Pin the toolchain. Record g++ --version next to the envelope. A different libstdc++ can move RSS without any source change.
Build the known-good binary at -O2. Optimization level is part of the spec. Debug heaps lie about both RSS and user time.
Capture a baseline with --record. The record path should refuse to overwrite unless --force is passed.
Apply the candidate patch in a worktree. Rebuild with the same flags. Sanitizers stay out of this particular loop; they change RSS on purpose.
Grade the candidate with --grade and --expect-stdout. Read the JSON verdict line, not the driver's stdout, for the cost decision.
Reject when functional output mismatches or when any ceiling breaks. Accept only when both oracles agree.
If the cost increase is the actual design, raise envelope.json in the same commit as the patch. Split commits hide the trade.

A one-line compile plus the two harness modes looks like this:

set -euo pipefail
g++ --version
g++ -std=c++17 -O2 -Wall -Wextra -o driver driver.cpp
./driver > golden_stdout.txt
python3 rusage_harness.py --record --force --cmd ./driver --envelope envelope.json
python3 rusage_harness.py --grade --cmd ./driver cache \
  --envelope envelope.json --expect-stdout golden_stdout.txt

The interesting outcome is exit status 1 with "reasons": ["rss"] while stdout still matches. That is the patch that looks like competence in a unit-test UI.

Where generation and grading can share a host

Local laptops add noise. Browser tabs move max RSS. Thermal throttling moves user CPU. An eval that is supposed to catch silent weight needs a boring machine.

Teams that already generate C++ patches from models can run this envelope next to the compiler. MonkeyCode is an open-source project with free model access and a free server option, which is enough to keep both generation and the rusage loop off a developer workstation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The envelope still has to live in the repository. Free inference does not define max_rss_kb. The server is a stable place to run the grader, not a substitute for committed ceilings. Model names, token quotas, and hardware details are out of scope here because they change and they are not what the envelope measures.

How the silent case shows up

The scan driver and the cache driver can share stdout. The harness then prints two different reasons to fail.

Observation	Functional oracle	Envelope oracle	Action
Same stdout, RSS and CPU under ceilings	pass	pass	accept
Different stdout, any cost	fail	ignored	reject
Same stdout, RSS over ceiling	pass	fail	reject as cost regression
Timeout	fail	fail	reject as hang
Compile error	fail	n/a	reject

The third row is the point of the extra artifact. Models also emit optimizations that raise CPU while lowering RSS, or the reverse. A single-field envelope will miss half of those. RSS and user CPU start together. They should not be averaged into one score. Averages hide which contract broke.

Limitations

ru_maxrss is a high-water mark, not a leak proof. A patch can allocate, free, and still pass if the peak stays under the ceiling. The envelope will not replace AddressSanitizer or a heap profiler.

Shared eval hosts are noisy. A noisy neighbor can raise minor faults and sometimes RSS. Pinning a cgroup, or committing a small slack percentage inside envelope.json, is honest. Hidden slack inside the Python script is a grader bug.

The driver above uses a static map, which is easy to see in review. Real regressions hide in extra copies of std::string, in std::regex constructed per call, in logging that formats on a hot path. The envelope does not name the guilty line. It only refuses to ship the binary.

Timeouts are wall-clock. A loaded host turns a legal patch into a hang. timeout_s needs the same honesty as RSS, and the machine should stay otherwise idle during grade. After TimeoutExpired, child rusage may be incomplete. Treat timeout as a hard reject rather than a partial RSS reading.

This loop does not prove asymptotic complexity. Forty thousand iterations can still miss an O(n²) path that production will hit at millions. Pair the envelope with at least one fixture that resembles production cardinality. Tiny goldens created the original incident.

Who should skip this approach

The technique does not replace perf or microbenchmark libraries when the question is a 2% inner-loop change. The tripwire is too coarse for that job.

It does not transfer to Windows without a different oracle. The script above is not a portable abstraction.

It does not belong on patches that are explicitly space-time tradeoffs unless the envelope update rides along in the same review. A grader that always loses to an intentional cache is theatre.

It should not run under extra sanitizers and then compare against a non-sanitized envelope. That comparison is meaningless.

Skip the technique when the binary's memory is dominated by a huge mmap the patch cannot affect. The ceiling will never move. The signal is zero.

Closing

Unit tests answer whether the answer was right. They do not answer whether the process was allowed to become larger while getting that answer. Commit both oracles. Grade both on every model patch.

The sample files above are small enough to drop beside an existing C++ eval. A stable host matters more than a clever dashboard. If patch generation already has spare capacity, envelope.json is the cheaper artifact to try on that same host first.

Release Builds Are a Different Spec: A Flag-Matrix Harness for AI C++ Patches

Finley Li — Fri, 11 Sep 2026 15:32:24 +0000

A maintainer merged an AI-written bounds check into a small C++ parser. Debug CI compiled with -O0, ran the unit tests, and reported success. The guard was a single assert. The production job used -O2 -DNDEBUG, so the guard disappeared, and a short buffer turned a size_t into an over-read.

That failure is not a missing test case. It is a missing compile profile. The model satisfied the one command the prompt displayed. Release flags were never part of the spec the grader enforced.

Public debates about AI coding skill still score patches on whether tests pass. Compiler flags are the part of the score those threads skip. A patch that is green under one profile and wrong under another is a silent regression, even when every assertion in the Debug binary fires.

Why a single compile lies

AI C++ patches optimize for the build line in the prompt. If that line is g++ -std=c++17 -O0 -g, the model never sees NDEBUG, inlining, tautological-compare folding, or signed-overflow assumptions. Tests attached to that command become a second prompt. They do not become a contract.

Three splits show up constantly in generated patches:

Validation lives only in assert, so -DNDEBUG deletes the check.
Signed overflow or strict aliasing is undefined, and -O2 exploits it.
The Debug run is sanitizer-clean only because the sanitizer was never linked.

A flag-matrix harness treats those profiles as first-class oracles. The same sources, the same tests, four compiles. Disagreement is a failed eval, not a flake.

The artifact: a compile-profile ledger

The harness below is deliberately small. It grades one subject function across Debug, Release, an NDEBUG-only build, and an ASan/UBSan build. A candidate patch must pass every profile that the project claims to ship. Label the snippets as a local recipe, not as production CI until the team pins compiler versions.

1. Pin a subject that can fail two ways

Keep the function tiny so the matrix stays cheap. The first subject uses assert as a bounds check. The second relies on signed arithmetic that Debug may tolerate.

// subject_assert.cpp — assert-only guard (fails under -DNDEBUG)
#include <cassert>
#include <cstddef>
#include <cstdint>

std::uint32_t prefix_sum(const std::uint32_t* p, std::size_t n, std::size_t i) {
    assert(p != nullptr && i < n);   // gone in Release
    std::uint32_t acc = 0;
    for (std::size_t k = 0; k <= i; ++k) acc += p[k];
    return acc;
}

// subject_overflow.cpp — signed overflow the optimizer may fold
int scale_index(int base, int stride, int i) {
    return base + stride * i;        // UB if the product overflows
}

A golden test for the first function must include an out-of-range index. Under Debug the process aborts. Under -DNDEBUG it may return a value or crash later. Both outcomes are eval signal. Do not rewrite the test to skip the bad index. That rewrite hides the profile split.

2. Declare profiles as data, not comments

Store the matrix next to the tests. Comments in a README drift. A file the runner reads does not.

# profiles.txt — one compile recipe per line: name|cxxflags
debug|-std=c++17 -O0 -g
release|-std=c++17 -O2 -DNDEBUG
ndebug_o0|-std=c++17 -O0 -DNDEBUG
san|-std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer

The third row is the one most AI graders omit. It keeps -O0 so the binary stays easy to debug, but it still strips asserts. If a patch only “works” because assert is alive, this row fails while debug passes. That pair is the ledger’s most useful disagreement.

3. Drive the matrix from a shell runner

The runner compiles each profile into its own binary, executes the same test driver, and writes a row per profile. Exit code, sanitizer noise, and a stable stdout digest all belong in the row. Do not fold them into a single boolean until the end.

#!/usr/bin/env bash
# run_matrix.sh — proposed local harness, not a hosted service
set -euo pipefail
CXX=${CXX:-g++}
SRC=${1:?usage: run_matrix.sh subject.cpp testdriver.cpp}
TEST=${2:?}
ROOT=$(mktemp -d)
trap 'rm -rf "$ROOT"' EXIT

pass=0
fail=0
while IFS='|' read -r name flags; do
  [[ -z "${name:-}" || "$name" == \#* ]] && continue
  out="$ROOT/$name"
  # shellcheck disable=SC2086
  if ! $CXX $flags -o "$out" "$SRC" "$TEST" 2>"$ROOT/$name.err"; then
    echo "$name COMPILE_FAIL"
    fail=$((fail+1))
    continue
  fi
  set +e
  "$out" >"$ROOT/$name.out" 2>"$ROOT/$name.san"
  rc=$?
  set -e
  digest=$(cksum "$ROOT/$name.out" | awk '{print $1}')
  san=$(wc -c <"$ROOT/$name.san")
  echo "$name rc=$rc digest=$digest san_bytes=$san"
  if [[ $rc -ne 0 || $san -ne 0 ]]; then
    fail=$((fail+1))
  else
    pass=$((pass+1))
  fi
done < profiles.txt

echo "matrix pass=$pass fail=$fail"
[[ $fail -eq 0 ]]

Run it the same way for the baseline tree and for the patched tree. The eval is the diff of those two ledgers, not the patched tree alone. A patch that turns a Release crash into a Debug-only assert is not an improvement. It moved the failure into a profile the default CI never runs.

4. Require unanimous pass, then diff digests

Unanimous exit-code zero is necessary and not sufficient. Digests must match across profiles that are supposed to be functionally identical. debug and ndebug_o0 should agree on stdout for every defined input. They should disagree on out-of-range inputs if and only if the project documents abort-vs-unchecked as intentional. Most library code should not.

# decision table (eval outcome)
# debug  release  ndebug_o0  san     verdict
# pass   pass     pass       pass    accept
# pass   fail     fail       pass    assert-only guard; reject
# pass   pass     pass       fail    sanitizer-only bug; reject
# pass   fail     pass       fail    optimizer/UB; reject
# fail   fail     fail       fail    tests too weak or patch broken; reject

Print the table from the two ledgers. Do not keep it in a slide. A grader that cannot emit the row cannot catch the split.

5. Add one negative golden that Debug hides

The matrix is useless if every test input is well-defined. Include at least one case the function must reject: null pointer, i == n, a product that overflows int. Under debug the process may trap. Under release the same input must not be treated as success. Record expected status per profile in a side file so the runner does not invent a policy.

# expect.txt — input_id|profile|want_rc
overflow_max|debug|1
overflow_max|release|1
overflow_max|ndebug_o0|1
overflow_max|san|1
oob_index|debug|1
oob_index|release|1
oob_index|ndebug_o0|1
oob_index|san|1

If the project truly wants Release to skip checks, say so in expect.txt. Silence is not a policy. AI patches will fill silence with assert.

Where free remote generation fits

Generating candidate patches and grading them are different jobs. The matrix above needs a local compiler, sanitizer runtimes, and the project’s own headers. It does not need a GPU. Candidate generation does.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which some teams use only to produce patch candidates when they do not want to host a model themselves. The flag-matrix harness still runs on the maintainer’s machine. Treat published token and hardware claims as things to verify on the project’s current docs before depending on them; this article does not assign quotas, model names, or uptime.

A practical split is: sample several patches remotely, drop each diff into a worktree, and require run_matrix.sh to print fail=0 plus matching digests. If the remote side is unavailable, the grader still works. If the grader is skipped, remote tokens only buy more green Debug builds.

Limitations

The matrix does not prove absence of undefined behavior. It raises the cost of the usual lies: assert-only safety, optimizer-visible overflow, and unsanitized heaps. It also multiplies compile time. Four profiles on a large translation unit is a real bill. Cache object files per profile or the harness becomes something people turn off.

Sanitizer rows need matching libraries and will not catch every data race or every uninitialized read. MSVC, libstdc++, and libc++ disagree on what is defined. Pin the compiler. Do not compare a GCC Debug digest to a Clang Release digest and call the mismatch a model failure.

Stdout digests fail for tests that print timestamps, pointer values, or iteration order of unordered containers. Stabilize output or hash a structured result file instead of raw stdout. Timing is not a digest. Do not turn this harness into a benchmark.

Who should not use this

Teams that ship a single -O0 binary and never enable NDEBUG gain little. Firmware trees that forbid RTTI, exceptions, and sanitizers need a reduced matrix, not a copy-paste of profiles.txt. If the subject is a header-only template soup that takes minutes to instantiate, run the matrix on a extracted TU that includes only the patched symbols.

This approach is also the wrong tool for style-only diffs, comment rewrites, and CMake cleanup. Those changes do not split across -O2. Spend the four compiles on functions that touch arithmetic, indexing, lifetime, or error paths.

What to keep in the eval, not the prompt

Prompts rot. Profiles in a file next to the tests do not, if the runner is the merge gate. Put the matrix in the same job that already compiles the AI patch. Keep the prompt free of flag essays. The model will still write assert. The ledger will still fail ndebug_o0. That is the point of the extra compile.

A Release build is a different specification from the Debug command the model saw. Grade both. Then treat disagreement as a rejected patch, not as an interesting footnote in a green CI log.

Exit Code Zero Is Not a Spec: A Digest Harness for AI C++ Patches

Finley Li — Wed, 09 Sep 2026 15:46:49 +0000

A payments team merged a model-written C++ patch on a Friday. Every parse_amount test returned zero. Monday's batch importer split records on the substring WARN, which the new code printed to stderr for every rounded cent.

The tests had never hashed stderr. The agent loop had no reason to. Coding agents now retry until a visible command succeeds. The loop is not malicious. It is incomplete.

This article proposes a digest harness that treats extra process channels as part of the spec. The method stays useful without any hosted model. It is a grader, not a demo.

Why a pass/fail bit is the wrong oracle

Unit tests encode the checks a human remembered to write. Agent retries encode whatever makes those checks green. Anything outside that set can drift: diagnostic streams, errno, elapsed-time class, sanitizer verdicts, even which paths the binary opens.

A digest does not replace tests. It notices when tests stayed green while the rest of the process changed. That gap shows up constantly once a model is allowed to edit, build, and rerun until CI is quiet.

The observable tuple

For each golden case the harness records a tuple, then hashes it.

Process exit code.
SHA-256 of stdout bytes.
SHA-256 of stderr bytes.
Sanitizer class: clean, asan, ubsan, timeout, or crash.
Wall-time bucket, not raw milliseconds.

Raw timings thrash across machines. Buckets such as lt_50ms, lt_500ms, lt_5s, and gt_5s keep the digest stable enough to compare. Treat those cutoffs as a proposal. Pick buckets that match the service SLA, not these defaults.

Store one JSON object per case. The file is the lock. A later candidate is accepted only when every field matches, not when ctest is green.

{
  "case": "parse_amount.rounding_up",
  "exit": 0,
  "stdout_sha256": "e3b0c44298fc1c149afbf4c8996fb924...",
  "stderr_sha256": "d4c3b2a1907e6f5d4c3b2a1907e6f5d4...",
  "sanitizer": "clean",
  "time_bucket": "lt_50ms"
}

A fixture the tests will not defend

The following C++ fixture is a worked example. It is not production billing code. The unit test checks the integer result and ignores stderr on purpose, which is the failure mode the digest is meant to catch.

// parse_amount.cpp — example fixture, not a library
#include <charconv>
#include <iostream>
#include <string_view>

// Returns cents. Writes a diagnostic on rounding.
int parse_amount(std::string_view s) {
    double v = 0.0;
    auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v);
    if (ec != std::errc{}) return -1;
    int cents = static_cast<int>(v * 100.0 + 0.5);
    if (v * 100.0 != static_cast<double>(cents)) {
        std::cerr << "WARN: rounding " << s << "\n";
    }
    return cents;
}

int main(int argc, char** argv) {
    if (argc < 2) return 2;
    int c = parse_amount(argv[1]);
    if (c < 0) return 1;
    std::cout << c << "\n";
    return 0;
}

A typical unit test only asserts parse_amount("1.005") == 101. An agent can keep that equality and still change the diagnostic, the rounding mode, or the crash behavior under -fsanitize=undefined. The digest is the extra oracle.

Build the grader in numbered steps

Pin the toolchain first. Unpinned compilers make digests lie.

1. Record a baseline binary under two configs

Compile once without sanitizers and once with undefined-behavior sanitizing. Keep both artifacts. Do not mix their digests.

cxx=${CXX:-g++}
std=${CXXSTD:-c++20}

"$cxx" -std="$std" -O2 -o parse_amount parse_amount.cpp
"$cxx" -std="$std" -O1 -fsanitize=undefined -fno-omit-frame-pointer \
    -o parse_amount.ubsan parse_amount.cpp

If the local compiler rejects -fsanitize=undefined, stop. A missing sanitizer is not a clean class. Record unavailable and refuse to compare against a clean baseline from another machine.

2. Drive cases from a plain text list

Golden inputs stay tiny and boring on purpose. The point is coverage of channels, not coverage of finance.

# cases.txt — one argv payload per line
1.00
1.005
0
-3.2
999999999999
not-a-number

3. Capture the tuple without trusting the shell

The script below is a proposal. Run it as a local tool. It does not claim to have been executed against a public corpus.

#!/usr/bin/env python3
"""digest_run.py — capture a behavior digest for one argv."""
import hashlib, json, os, subprocess, sys, time

BUCKETS = [(0.05, "lt_50ms"), (0.5, "lt_500ms"), (5.0, "lt_5s")]

def bucket(seconds: float) -> str:
    for limit, name in BUCKETS:
        if seconds < limit:
            return name
    return "gt_5s"

def digest_of(bin_path: str, arg: str, timeout: float) -> dict:
    env = os.environ.copy()
    env["UBSAN_OPTIONS"] = "print_stacktrace=1:halt_on_error=1"
    t0 = time.monotonic()
    try:
        p = subprocess.run(
            [bin_path, arg],
            capture_output=True,
            timeout=timeout,
            env=env,
        )
        elapsed = time.monotonic() - t0
        san = "clean"
        blob = (p.stderr or b"") + (p.stdout or b"")
        if b"runtime error:" in blob or b"UBSAN" in blob:
            san = "ubsan"
        if p.returncode < 0:
            san = "crash"
        return {
            "case": arg,
            "exit": p.returncode,
            "stdout_sha256": hashlib.sha256(p.stdout).hexdigest(),
            "stderr_sha256": hashlib.sha256(p.stderr).hexdigest(),
            "sanitizer": san,
            "time_bucket": bucket(elapsed),
        }
    except subprocess.TimeoutExpired:
        return {
            "case": arg,
            "exit": None,
            "stdout_sha256": None,
            "stderr_sha256": None,
            "sanitizer": "timeout",
            "time_bucket": "gt_5s",
        }

def main() -> int:
    bin_path, arg, timeout = sys.argv[1], sys.argv[2], float(sys.argv[3])
    json.dump(digest_of(bin_path, arg, timeout), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

4. Freeze the baseline, then grade candidates as a diff

Write one JSON file per case under gold/. After a model emits a patch, rebuild both binaries and emit cand/. Comparison is structural. A single mismatched field fails the patch even when the process exit code is still zero.

#!/usr/bin/env python3
"""digest_diff.py — fail on any field drift. Example grader, not a CI product."""
import json, pathlib, sys

FIELDS = ("exit", "stdout_sha256", "stderr_sha256", "sanitizer", "time_bucket")

def load(dirpath: str) -> dict:
    out = {}
    for p in sorted(pathlib.Path(dirpath).glob("*.json")):
        obj = json.loads(p.read_text())
        out[obj["case"]] = obj
    return out

def main() -> int:
    gold, cand = load(sys.argv[1]), load(sys.argv[2])
    failed = 0
    if gold.keys() != cand.keys():
        print("case set drifted", file=sys.stderr)
        return 2
    for case in gold:
        for field in FIELDS:
            if gold[case][field] != cand[case][field]:
                print(f"FAIL {case} {field}: {gold[case][field]!r} -> {cand[case][field]!r}")
                failed += 1
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

python3 digest_diff.py gold cand
# exit 0: every channel matched
# exit 1: at least one silent drift
# exit 2: the case set itself changed

A decision table for the mismatch

Use the table during review. Do not auto-merge on a story about flakiness.

Digest field	Typical agent shortcut	Review action
`exit` still 0, `stderr_sha256` changes	New logging, leftover debug, or a `WARN` that becomes a parser	Require the stream to be part of the test or strip it
`stdout_sha256` changes with tests still green	Formatting drift, extra newline, locale, or a different rounding path	Freeze expected bytes or reject the patch
`sanitizer` moves from `clean` to `ubsan`	Signed overflow that `-O2` tests never hit	Reject; do not “quiet” UBSAN_OPTIONS
`time_bucket` jumps one class	Sleep-based race “fix”, extra copies, or a remote FS stall	Re-run three times; fail if the bucket stays shifted
`sanitizer` is `timeout`	Infinite retry, lock, or pathological input	Fail closed; do not raise the timeout to make it green

The interesting row is the first one. Tests stay green. The product still breaks. That is the whole point of hashing stderr.

Where a free model tier and a free server fit

Local laptops are a poor farm for sanitizer builds. They thermal-throttle. They also leak prompt history into shell logs. A hosted compile box with a pinned compiler is the boring fix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free-tier model access and a free server option. Those two availability claims are the only product facts used here. This article does not name models, quote a token ceiling, or publish hardware or timing numbers, because those details change and would go stale in a tutorial.

The workflow stays the same on any box that can run g++ and Python 3. Generate a candidate patch from the free-tier model access. Build both binaries on the free server so sanitizer flags match the baseline. Emit cand/ there. Diff against gold/ before a human looks at the diff hunks. If the digest fails, the patch is not “almost good.” It changed an observable the tests never named.

Limitations

Digests are brittle in the ways tests are not. Locale, libstdc++ debug iterators, and ASLR-backed printouts will churn hashes. Redirected fds that include timestamps will churn them too. Time buckets still false-positive on a noisy neighbor, which is why the table requires a rerun rather than a story.

The harness also cannot see bugs that never become process output. A wrong cache key that still prints the same cents will hash equal. So will a leak that no test frees. AddressSanitizer on a third binary can be added later. It is a different digest, not a field on this one.

Nondeterministic tests are out of scope. If the fixture reads /dev/urandom, the wall clock, or a live socket, freeze those seams first. Hashing noise is not a grader.

Who should not use this

Skip the digest harness when the project has no pinned compiler, when golden cases require a GPU, or when the binary is an interactive TUI. Skip it when the team already snapshots stdout, stderr, and sanitizer logs in CI. Skip it for patches that are supposed to change diagnostics; update gold/ in the same commit and say so in the log.

Do not use a free shared server for secrets, customer traces, or proprietary headers. Digests of public fixtures are the intended workload. Proprietary corpora belong on a machine the team owns.

Agent loops will keep optimizing the command they can see. Give them a command that sees more than exit 0. Then keep the human review for the cases the digest cannot name.

Green Tests, Drifted Symbols: A Witness Harness for AI C++ Patches

Finley Li — Tue, 08 Sep 2026 03:29:29 +0000

A release engineer froze a C++ CSV splitter at tag v0.4.2, then accepted an AI-written patch that claimed to speed up quoted-field parsing. Catch2 reported 41 passed, 0 failed. The shared library still loaded. Seven days later the distro packaging job broke: nm -D libsplit.so no longer listed split_row_alloc, because the model had marked the helper static inline in a header. Tests never called that symbol through the dynamic ABI. The suite stayed green. The binary drifted.

That failure mode shows up whenever generated C++ lands faster than reviewers can read every diff. Unit tests check the paths someone remembered to write. They do not check exported symbols, section sizes, sanitizer findings under a second optimization level, or the exact bytes a CLI writes to stderr on a bad flag. An eval harness that only asserts ctest therefore grades the demonstration, not the artifact.

This article proposes a witness harness. After the familiar compile-and-test loop, the grader records a small, committed snapshot of the binary and of the program's I/O contract. The next model patch must match that snapshot, or the eval fails closed. The method is toolchain-dependent and incomplete. It is also cheap enough to run on a laptop, and it does not require the model to be honest about what it changed.

What the witness actually records

A witness is not another golden unit test. It is a fingerprint of side effects the suite usually ignores. The proposed tape stores four files per fixture:

argv.json — argument vector and selected environment keys, including LC_ALL.
stdin.bin — exact bytes fed to the process, including empty files.
expect.json — exit code, stdout SHA-256, stderr SHA-256, timeout.
symbols.nm — sorted, demangled defined symbols from the linked target, plus one size line for text/data/bss.

If any of those files is missing, the harness does not skip. It fails. Missing evidence is a fail, not a pass. That single rule stops the silent path where a new fixture is added, the model “passes,” and nobody notices the witness was never recorded.

Cheap generation makes that path more common, not less. When C++ patches are easy to obtain, technical debt moves from missing tests into binaries that still satisfy the tests. The tape is there to make the binary argue back.

Repository layout

The following tree is a proposal, not a shipped product. Keep the production sources and the eval corpus in the same git repo so a patch cannot forget the tape.

eval/
  fixtures/
    csv_quote/
      argv.json
      stdin.bin
      expect.json
      symbols.nm
    bad_flag/
      argv.json
      stdin.bin
      expect.json
      symbols.nm
  witness.py
  matrix.cmake
src/
  split.cpp
  split.hpp
CMakeLists.txt

matrix.cmake encodes the compile grid. One configuration is not an eval. AI C++ patches routinely survive -O0 and explode under -O2 and -fsanitize=address. The grid below is deliberately small so a remote builder can finish it without a private GPU farm.

# matrix.cmake — proposed compile grid
set(WITNESS_CONFIGS
  "dbg|-O0 -g -Wall -Wextra -Werror"
  "rel|-O2 -DNDEBUG -Wall -Werror"
  "asan|-O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer"
)

A three-cell matrix will not replace vendor CI. It will catch the patch that introduced a signed overflow only visible at -O2. That is the usual way a fluent model output becomes a production crash.

Step 1 — Freeze the I/O contract

Write expect.json by running the current binary, not the model's story about the binary. The procedure below is labeled as a proposal.

# Proposed capture, run from a clean build/
export LC_ALL=C
./split --strict < ../eval/fixtures/csv_quote/stdin.bin \
  > /tmp/out.bin 2> /tmp/err.bin
echo $?
python3 - <<'PY'
import hashlib, json, pathlib
def sha(p):
    return hashlib.sha256(pathlib.Path(p).read_bytes()).hexdigest()
print(json.dumps({
    "exit": 0,
    "stdout_sha256": sha("/tmp/out.bin"),
    "stderr_sha256": sha("/tmp/err.bin"),
    "timeout_s": 5
}, indent=2))
PY

Commit those hashes. Do not commit raw stdout if the fixture might contain customer rows. Hashes are enough for regression and safer if the corpus later runs on a shared builder. Pin LC_ALL=C in argv.json as well, or a locale change will look like a model regression.

Step 2 — Freeze exported symbols

nm output is noisy. Pipe it through a filter that keeps only T and W symbols, drops compiler-generated guards, and sorts. The filter is the contract. Change the filter and every witness will churn, which is preferable to silently accepting a new export.

# Proposed symbol freeze for a C++ shared object or CLI
nm -C --defined-only ./libsplit.so \
  | awk '$2 ~ /^[TW]$/ { print $2, $3 }' \
  | grep -v '_GLOBAL' \
  | sort > ../eval/fixtures/csv_quote/symbols.nm
size ./libsplit.so | awk 'NR==2 { print "size", $1, $2, $3 }' \
  >> ../eval/fixtures/csv_quote/symbols.nm

Point nm at the .so when the packaging surface is a library. Point it at the CLI when the artifact is a statically linked tool. A header-only inline that used to be an exported T symbol becoming an inlined phantom is exactly the packaging break from the opening story. The size line is coarse. It still catches the patch that quietly pulled in another translation unit.

Step 3 — Grade a candidate patch fail-closed

witness.py is a proposed grader. It applies a unified diff, builds each matrix cell, runs each fixture, and compares witnesses. Any exception, missing file, or mismatch returns non-zero. There is no warn status. Reviewers can still read the log. Automation should not.

#!/usr/bin/env python3
"""Proposed fail-closed witness grader for AI C++ patches. Illustrative."""
from __future__ import annotations
import hashlib, json, os, subprocess, sys, tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
EVAL = ROOT / "eval" / "fixtures"
TIMEOUT = 30

class Fail(Exception):
    pass

def run(cmd, cwd, env=None, timeout=TIMEOUT, stdin=None):
    return subprocess.run(
        cmd, cwd=cwd, env=env, input=stdin,
        stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout,
    )

def sha256(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()

def build(cell_flags: str, work: Path) -> Path:
    src = work / "build"
    src.mkdir(exist_ok=True)
    cmake = run(
        ["cmake", "-S", str(work), "-B", str(src),
         f"-DCMAKE_CXX_FLAGS={cell_flags}"],
        cwd=work,
    )
    if cmake.returncode != 0:
        raise Fail(f"cmake failed: {cmake.stderr.decode()[-800:]}")
    ninja = run(["cmake", "--build", str(src), "-j"], cwd=work)
    if ninja.returncode != 0:
        raise Fail(f"build failed: {ninja.stderr.decode()[-800:]}")
    bin_path = src / "split"
    if not bin_path.exists():
        raise Fail("binary missing after build")
    return bin_path

def symbols_of(bin_path: Path) -> str:
    nm = run(["nm", "-C", "--defined-only", str(bin_path)], cwd=bin_path.parent)
    if nm.returncode != 0:
        raise Fail("nm failed")
    lines = []
    for line in nm.stdout.decode(errors="replace").splitlines():
        parts = line.split()
        if len(parts) >= 3 and parts[1] in {"T", "W"}:
            lines.append(f"{parts[1]} {parts[2]}")
    lines.sort()
    sz = run(["size", str(bin_path)], cwd=bin_path.parent)
    if sz.returncode == 0:
        rows = sz.stdout.decode().splitlines()
        if len(rows) >= 2:
            cols = rows[1].split()
            lines.append(f"size {cols[0]} {cols[1]} {cols[2]}")
    return "\n".join(lines) + "\n"

def grade_fixture(bin_path: Path, fx: Path) -> None:
    argv = json.loads((fx / "argv.json").read_text())
    expect = json.loads((fx / "expect.json").read_text())
    stdin = (fx / "stdin.bin").read_bytes()
    gold_nm = (fx / "symbols.nm").read_text()
    cmd = [str(bin_path), *argv.get("args", [])]
    env = os.environ.copy()
    env.update(argv.get("env", {}))
    env.setdefault("LC_ALL", "C")
    try:
        p = run(
            cmd, cwd=bin_path.parent, env=env,
            timeout=expect.get("timeout_s", 5), stdin=stdin,
        )
    except subprocess.TimeoutExpired as exc:
        raise Fail(f"{fx.name}: timeout") from exc
    if p.returncode != expect["exit"]:
        raise Fail(f"{fx.name}: exit {p.returncode} != {expect['exit']}")
    if sha256(p.stdout) != expect["stdout_sha256"]:
        raise Fail(f"{fx.name}: stdout hash mismatch")
    if sha256(p.stderr) != expect["stderr_sha256"]:
        raise Fail(f"{fx.name}: stderr hash mismatch")
    got = symbols_of(bin_path)
    if got != gold_nm:
        raise Fail(f"{fx.name}: symbol witness mismatch")

def main() -> int:
    patch = Path(sys.argv[1]) if len(sys.argv) > 1 else None
    cells = [
        "-O0 -g -Wall -Wextra -Werror",
        "-O2 -DNDEBUG -Wall -Werror",
        "-O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer",
    ]
    fixtures = sorted(p for p in EVAL.iterdir() if p.is_dir())
    if not fixtures:
        print("FAIL closed: no fixtures", file=sys.stderr)
        return 2
    with tempfile.TemporaryDirectory() as td:
        work = Path(td) / "src"
        subprocess.check_call(["git", "clone", "--local", str(ROOT), str(work)])
        if patch:
            applied = run(["git", "apply", str(patch.resolve())], cwd=work)
            if applied.returncode != 0:
                print("FAIL closed: patch does not apply", file=sys.stderr)
                return 2
        for flags in cells:
            try:
                binary = build(flags, work)
                for fx in fixtures:
                    for required in ("argv.json", "stdin.bin", "expect.json", "symbols.nm"):
                        if not (fx / required).exists():
                            raise Fail(f"{fx.name}: missing {required}")
                    grade_fixture(binary, fx)
            except Fail as exc:
                print(f"FAIL closed [{flags}]: {exc}", file=sys.stderr)
                return 1
    print("PASS: compile matrix and witnesses matched")
    return 0

if __name__ == "__main__":
    sys.exit(main())

The script clones the repo into a temp directory so a failing patch cannot leave object files in the working tree. That matters when the next eval reuses the same checkout, including on a shared builder. Sanitizer flags stay inside one matrix cell. Mixing ASan into the release cell would make size witnesses flap for reasons that have nothing to do with the patch.

Step 4 — Read the verdict table, not the model's summary

Models narrate. The harness should not. Map every exit path to a single token so a log scraper can count regressions without parsing English.

Token	Meaning	Typical cause
`PATCH_REJECT`	`git apply` failed	Wrong paths, already-applied hunk
`CMAKE_FAIL`	configure error	Invented options, missing C++ standard
`BUILD_FAIL`	compile or link error	Undeclared symbols, ODR splits
`SAN_FAIL`	ASan/UBSan abort	Use-after-free, signed overflow
`IO_FAIL`	exit/stdout/stderr mismatch	CLI drift, extra logging
`SYM_FAIL`	`nm`/`size` mismatch	Hidden ABI change
`TIMEOUT`	wall clock exceeded	Accidental quadratic parse
`PASS`	all cells and fixtures matched	Still not a ship decision

PASS means the tape did not move. It does not mean the patch is desirable. Size may stay within the witness because the size line is coarse. Reviewers still read the diff. The point of the table is to stop a chatty model from turning a SYM_FAIL into a paragraph about “equivalent refactoring.”

Where a free coding server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free-tier model access and a free server option. Those two availability claims matter to this workflow in a narrow way. The model is a patch factory. The server is an optional place to run witness.py when a laptop cannot hold three sanitizer builds at once. The grader remains the authority.

A generated patch that explains itself well and then changes split_row_alloc from exported to inline still exits SYM_FAIL. Hosted tokens do not grade C++. Compilers, sanitizers, and nm do. Treat any advertised allowance as a budget that can change, not as a capacity plan and not as a benchmark. No model names, queue times, or hardware claims are made here because they go stale.

A practical loop looks like this:

Store the witness tape in git next to the sources.
Ask the coding model for a patch against a single fixture name, not against “make it faster.”
Save the unified diff to candidate.patch.
Run python3 eval/witness.py candidate.patch.
If the result is not PASS, send the fail token and the last 40 log lines back into the next prompt. Do not paste the whole binary.

That loop is slow on purpose. Silent regressions are cheap to ship and expensive to notice. Spending a compile matrix per candidate is the point, not a defect.

Limitations

nm and size vary across binutils, LLVM, and MinGW. Witnesses are not portable between distros. Commit the toolchain version next to symbols.nm or the tape will flap on every image bump. Sanitizers miss data races that only TSan would see. TSan is omitted here because it disagrees with many allocators and would turn the grid into a flake factory.

SHA-256 of stdout is brittle against locale, date stamps, and trailing newlines. Fixtures must pin LC_ALL=C and refuse timestamps in CLI output. The harness does not prove functional equivalence for GUI programs, embedded firmware, or signal-heavy servers. It also does not prove that a larger binary is wrong. It only proves the binary is different.

Free-tier model access and a free server have unspecified queue time, hardware, and retention. Do not put proprietary fixtures on a shared host. Do not use this article’s commands as a performance comparison between products. No timings are claimed, and none should be inferred from a green PASS line.

Who should not use this approach

Teams with a dedicated ABI review bot and a full -fsanitize matrix in CI already have a stronger tape. Kernel modules, plugins loaded by a stable C API, and libraries that version symbols with .symver need a real ABI checker such as libabigail, not a sorted nm dump. Organizations that cannot open the eval corpus should not send stdin fixtures to any hosted coding server, free or not.

Skip the method when the artifact is not a CLI or a shared library. A notebook, a header-only template soup, or a CMake-only repo will produce empty nm output and a false sense of safety. Skip it for one-off prototypes where the binary is allowed to churn and the only consumer is the author.

The witness harness exists because cheap C++ generation turned “the tests are green” into a weak sentence. Green tests and a drifted symbol table can coexist. Record the symbol table. Fail closed when it moves. Readers who already keep C++ fixtures on disk can run that same loop against a free-tier coding server instead of provisioning a GPU host for the first filter.

MonkeyCode provides free models that can run this workflow.

Mutation-Score the Grader: A Prompt Lockfile for AI C++ Patch Evals

Finley Li — Sun, 06 Sep 2026 21:02:19 +0000

A mid-size systems team merged an AI-written patch into a lock-free queue. Unit tests stayed green. Two weeks later a prompt edit asked the model to prefer simpler atomics. The new patch compiled. The golden case still passed, because it never ran two producers. Production dropped items under load. The harness did not fail. It had never been mutation-scored, and the prompt had never been pinned.

Cheap C++ patches create a second class of debt. Golden cases rot. Graders accept weaker oracles. Prompt text drifts in a chat window with no hash. The rest of this article describes a small, reproducible eval harness that treats the prompt as a locked dependency and mutation-tests the grader itself.

What this harness is for

The workflow targets AI-generated C++ patches that look locally correct. It does not replace code review. It catches three silent failures: prompt drift, oracle rot, and patches that satisfy only the tests the team already showed the model.

A demo test repeats the happy path the model already saw. A contract case states what must remain true after a patch the model has not been coached to pass. The difference is the whole method.

Artifact: a prompt lockfile plus a mutation pass

The artifact has four files.

prompt.lock.json — hashes of the system prompt, the user template, and the eval corpus.
cases/ — golden cases with inputs, forbidden diffs, and grader commands.
mutate_grader.py — a script that weakens oracles on purpose and expects the harness to fail.
run_eval.sh — a thin runner that compiles, links sanitizers, and records grades.

Each file is small enough to keep in the same repository as the C++ library under test. The lockfile is the product. The mutation pass is how the team learns the product is not theater.

Step 1: Pin the prompt like a lockfile

Prompts change faster than headers. A one-line edit can alter which includes the model emits. The lockfile records cryptographic hashes, not the prompt prose, so review diffs stay short.

{
  "schema": "cpp-eval-lock/v1",
  "prompt": {
    "system_sha256": "replace-with-sha256-of-system.txt",
    "user_template_sha256": "replace-with-sha256-of-user.md.tmpl",
    "style_rules_sha256": "replace-with-sha256-of-invariants.md"
  },
  "corpus": {
    "cases_sha256": "replace-with-sha256-of-cases-tree",
    "grader_sha256": "replace-with-sha256-of-mutate_grader.py"
  },
  "language": "c++17",
  "sanitizers": ["address", "undefined"]
}

A pre-commit hook recomputes the hashes. A mismatch is a failed eval, even if every test is still green. That is the point. Green tests with a drifted prompt are a silent regression of the harness, not of the library.

#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import hashlib, json, pathlib, sys
lock = json.loads(pathlib.Path("prompt.lock.json").read_text())
def sha(p):
    return hashlib.sha256(pathlib.Path(p).read_bytes()).hexdigest()
checks = {
    "system_sha256": sha("system.txt"),
    "user_template_sha256": sha("user.md.tmpl"),
    "style_rules_sha256": sha("invariants.md"),
}
bad = [k for k, v in checks.items() if lock["prompt"][k] != v]
if bad:
    print("lock mismatch:", ", ".join(bad))
    sys.exit(2)
print("prompt lock ok")
PY

The snippet is a proposal. Teams should replace the placeholder hashes before using it as a gate. Unsigned prose in a chat log is not a pin.

Step 2: Write golden cases as contracts, not demos

Each case records three oracles: compile flags, forbidden diff substrings, and a behavior command. Forbidden diffs catch deletions the model likes when a prompt says "simplify." Behavior still matters. A patch that keeps the atomics and loses items must fail the command oracle.

{
  "id": "queue-mpmc-no-loss",
  "source_files": ["src/queue.hpp", "src/queue.cpp"],
  "forbidden_diff": [
    "std::memory_order_relaxed on head or tail",
    "removal of mutex or atomic around published index"
  ],
  "compile": {
    "std": "c++17",
    "flags": ["-Wall", "-Werror", "-fsanitize=address,undefined"]
  },
  "behavior": {
    "cmd": ["./queue_mpmc_test", "--producers", "2", "--consumers", "2", "--n", "10000"],
    "expect_exit": 0,
    "expect_stdout_contains": ["lost=0"]
  }
}

A minimal C++ golden driver looks like the following. It is an example driver, not a proven lock-free queue.

// cases/queue_mpmc_test.cpp — example driver, not production queue code
#include "queue.hpp"
#include <atomic>
#include <iostream>
#include <thread>
#include <vector>

int main() {
  const int producers = 2, consumers = 2, n = 10000;
  Queue<int> q;
  std::atomic<int> produced{0}, consumed{0};
  std::vector<std::thread> ts;
  for (int i = 0; i < producers; ++i) {
    ts.emplace_back([&] {
      for (int k = 0; k < n; ++k) {
        q.push(k);
        produced.fetch_add(1, std::memory_order_relaxed);
      }
    });
  }
  for (int i = 0; i < consumers; ++i) {
    ts.emplace_back([&] {
      int v;
      while (consumed.load(std::memory_order_relaxed) < producers * n) {
        if (q.try_pop(v))
          consumed.fetch_add(1, std::memory_order_relaxed);
      }
    });
  }
  for (auto& t : ts) t.join();
  const int lost = produced.load() - consumed.load();
  std::cout << "lost=" << lost << "\n";
  return lost == 0 ? 0 : 1;
}

Single-thread tests would have kept the original silent failure green. Contention is part of the oracle. A case that never contends is a demo wearing a contract badge.

Step 3: Grade in layers, then stop early

Numbered graders keep the failure mode readable.

Lock grade. Prompt hashes match prompt.lock.json.
Compile grade. -Wall -Werror plus ASan and UBSan.
Diff grade. Forbidden substrings do not appear in the unified diff.
Behavior grade. The golden driver exits 0 with the expected tokens.

A runner can be a shell script. Keep it boring.

#!/usr/bin/env bash
set -euo pipefail
case_id="${1:?case id}"
patch="${2:?unified diff}"

./check_lock.sh
git apply --check "$patch"
git apply "$patch"

g++ -std=c++17 -Wall -Werror -fsanitize=address,undefined \
  -o queue_mpmc_test src/queue.cpp cases/queue_mpmc_test.cpp

python3 grade_diff.py --case "$case_id" --diff "$patch"
./queue_mpmc_test --producers 2 --consumers 2 --n 10000 | tee /tmp/out.txt
grep -q 'lost=0' /tmp/out.txt

Early stop matters. A compile failure should not be recorded as a behavior pass. Mixed grades are how silent regressions hide in dashboards. One CSV line per patch is enough.

Step 4: Mutation-score the grader

If the expected stdout is deleted and the harness still passes, the oracle is decoration. Mutation scoring applies hostile edits to the case definition, not to the C++ library. The harness must fail each mutant.

# mutate_grader.py — proposal: mutants target the oracle, not the product
import copy, json, pathlib, sys

def load(path):
    return json.loads(pathlib.Path(path).read_text())

def mutants(case):
    out = []
    c = copy.deepcopy(case)
    c["behavior"]["expect_exit"] = 1
    out.append(("invert-exit", c))
    c = copy.deepcopy(case)
    c["behavior"]["cmd"] = ["./queue_mpmc_test", "--producers", "1",
                            "--consumers", "1", "--n", "1"]
    out.append(("drop-contention", c))
    c = copy.deepcopy(case)
    c["forbidden_diff"] = []
    out.append(("drop-forbidden", c))
    c = copy.deepcopy(case)
    c["compile"]["flags"] = ["-O0"]
    out.append(("drop-sanitizers", c))
    return out

def harness_fails(case) -> bool:
    # Labeled stub: wire this to run_eval.sh against the mutant case.
    return False

def main():
    case = load(sys.argv[1])
    killed = 0
    rows = mutants(case)
    for name, mutant in rows:
        failed = harness_fails(mutant)
        print(f"{name}: {'killed' if failed else 'SURVIVED'}")
        killed += int(failed)
    print(f"mutation score: {killed}/{len(rows)}")
    sys.exit(0 if killed == len(rows) else 3)

if __name__ == "__main__":
    main()

A surviving mutant means the grader would accept a quieter case. That is how the original queue patch escaped. The single-thread golden case was a surviving mutant of a contention oracle.

Teams should refuse to expand the corpus until the mutation score is full. Extra green cases with dead oracles increase confidence theater. The score measures the harness. It does not measure the library.

Step 5: Run the loop where the patch is actually produced

The lockfile and the mutation pass are local files. They do not require a particular vendor. They do require that each generated patch is saved as a diff, then graded offline.

When a team uses MonkeyCode, the same user.md.tmpl should be the only user prompt sent, and run_eval.sh plus mutate_grader.py should be the commands that actually grade the diff. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator-described free-tier model access and a free server option are enough to exercise that loop. They are not a production SLA, a quota, a hardware claim, or a benchmark.

A practical cadence:

Freeze prompt files and refresh prompt.lock.json.
Request a patch for one failing library test, using the locked template.
Save the unified diff.
Run run_eval.sh.
Run mutate_grader.py against the case that just graded the patch.
Record lock hash, patch hash, grades, and mutation score in one CSV line.

date,lock,patch,lock_grade,compile,diff,behavior,mutants_killed,mutants_total
2026-09-07,a1b2,c3d4,pass,pass,fail,pass,3,4

The sample row is illustrative. A diff fail with a behavior pass is the interesting class: the model restored function and deleted an invariant the tests did not encode. CSV is enough. A dashboard is optional.

Decision table: when to treat green as a pass

Signal	Treat green as pass	Action
Lock mismatch, tests pass	No	Restore prompt files or re-hash deliberately
Mutation score incomplete	No	Repair oracles before adding cases
Compile pass, sanitizer flags dropped	No	Restore ASan and UBSan in the case
Behavior pass, forbidden diff hit	No	Treat as invariant regression
All grades pass, mutation score full	Conditionally	Review remaining design risk

"Conditionally" is intentional. The harness does not prove lock-freedom, linearizability, or API taste. It proves the oracles still bite.

Limitations and who should skip this

The approach is weak for GUI code, for generated protobufs, and for patches whose correctness is visual. It is also weak when the golden driver itself is wrong. Mutation scoring detects quiet oracles. It does not detect a confidently wrong expected value.

Sanitizer-backed C++ tests need a toolchain that actually links libasan. Windows MSVC setups will not paste the sample flags unchanged. The example queue driver can hang if try_pop never returns and the stop condition is sloppy. Timeouts belong in the runner. They are omitted above for brevity.

Teams that already have a mature property-based test suite may find the JSON cases redundant. Teams that cannot pin prompts because every request is a fresh chat with no files should not pretend they have this harness. Do not use this workflow as a substitute for a memory-model review on lock-free structures. Do not publish mutation scores as marketing metrics. A 4/4 kill rate on four crude mutants is not a safety case.

Keep the CSV boring

Cases the model never saw are a separate control. This article is about graders that fail to fail. Pin the prompt. Mutate the oracle. Keep the CSV boring.

Teams that already store diffs from a coding assistant can drop prompt.lock.json and mutate_grader.py beside those diffs and reuse the same runner on free-tier model access or a free server option. That is the whole loop.

MonkeyCode provides free models that can run this workflow.

Hidden Oracles Catch AI C++ Patches That Only Pass the Tests You Showed

Finley Li — Sat, 05 Sep 2026 19:34:33 +0000

A queue maintainer handed a coding model a broken bounded integer ring and three unit tests. The prompt showed capacities 8, 16, and 32. The returned patch compiled, linked, and turned every visible assertion green. Two days later a production producer used capacity 10 and a head index near 2^32. The wrap path the model had specialized to power-of-two sizes dropped an element with no exception and no log line.

Visible tests are a hint. They are not a specification. A model that can read the prompt can memorize the hint. The rest of this article treats that failure as a grading problem, not a vibes problem, and walks through a two-layer harness a C++ team can run on every generated patch.

Why a green demo still ships silent damage

Coding models optimize for the text in front of them. If the only executable contract in the prompt is tests/visible.cpp, a plausible strategy is to special-case those inputs. The compiler will not object. A human skim of a 20-line diff often will not object either.

That is a different bug class from a patch that fails to build. It is also different from a patch that deletes a mutex. Overfit code still looks careful. It copies the house style. It adds comments that restate the three tests. The defect lives in the cases nobody pasted into the chat.

Agent-style coding loops make the pattern cheaper to repeat. Each retry can grope toward the visible file. Without a second oracle the loop calls that progress. Technical debt then arrives as a passing suite, not as a red CI job.

Two layers, one patch, three exit codes

The harness below keeps two test binaries that share one implementation.

Visible tests may be quoted in the prompt. They document the bug the model is asked to fix.
Hidden oracles never appear in the prompt. They vary capacity, fill level, wrap, and sanitizer-visible undefined behavior.
The grader applies a unified diff, builds both binaries, and maps outcomes to a small fail taxonomy.

A patch is accepted only when both binaries pass under AddressSanitizer and UndefinedBehaviorSanitizer. A patch that greens the demo and reds the oracle is scored OVERFIT, not PASS. That single label is the point of the artifact.

The layout is boring on purpose.

eval_hidden_oracle/
  include/bounded_queue.hpp
  src/bounded_queue.cpp
  tests/visible.cpp
  oracles/hidden.cpp
  prompts/fix_queue.md
  patches/
  scripts/grade.sh
  manifest.json

manifest.json binds a prompt file to the two binaries so a later prompt edit cannot silently drop the hidden target.

{
  "id": "bounded-queue-v3",
  "prompt": "prompts/fix_queue.md",
  "sources": ["include/bounded_queue.hpp", "src/bounded_queue.cpp"],
  "visible": "tests/visible.cpp",
  "hidden": "oracles/hidden.cpp",
  "cxxflags": ["-std=c++17", "-fsanitize=address,undefined", "-fno-omit-frame-pointer"]
}

The worked example below is a fixture, not a production benchmark. No pass-rate is claimed.

The bug the prompt is allowed to describe

The header is small enough to paste. The defect is a classic full/empty collision plus a power-of-two wrap.

#pragma once
#include <cstddef>
#include <cstdint>
#include <optional>

class BoundedQueue {
public:
  explicit BoundedQueue(std::size_t cap);
  bool push(std::int32_t v);
  std::optional<std::int32_t> pop();
  std::size_t size() const;
  std::size_t capacity() const { return cap_; }

private:
  std::size_t cap_;
  std::size_t head_ = 0;
  std::size_t tail_ = 0;
  std::size_t count_ = 0;
  std::int32_t* buf_;
};

A broken implementation that still satisfies naive tests might mask indices with cap_ - 1 even when cap_ is not a power of two, and might treat head_ == tail_ as empty while also using that condition for full. Visible tests that only push three items into size 8 will not see it.

bool BoundedQueue::push(std::int32_t v) {
  if (count_ == cap_) return false;
  buf_[tail_ & (cap_ - 1)] = v; // wrong when cap_ is not 2^n
  tail_++;
  count_++;
  return true;
}

Step 1 — Write visible tests the prompt may include

Keep this file honest and small. It should fail on the broken code. It should not enumerate the interesting capacities.

#include "bounded_queue.hpp"
#include <cassert>

int main() {
  BoundedQueue q(8);
  assert(q.push(1));
  assert(q.push(2));
  assert(q.push(3));
  assert(q.size() == 3);
  assert(q.pop() == 1);
  assert(q.pop() == 2);
  assert(q.size() == 1);
  return 0;
}

The prompt then states the symptom in engineering language: push reports success but a later pop misses values when the queue is reused. It must not paste oracles/hidden.cpp.

Step 2 — Hide property checks the model does not get to read

The oracle file is the actual grader. It walks capacities that are not powers of two, fills to the last slot, wraps head and tail past 2^16, and checks FIFO order. It also asks ASan to notice an out-of-bounds index.

#include "bounded_queue.hpp"
#include <cassert>
#include <cstdint>
#include <vector>

static void roundtrip(std::size_t cap, int cycles) {
  BoundedQueue q(cap);
  std::vector<std::int32_t> want;
  for (int i = 0; i < static_cast<int>(cap); ++i) {
    assert(q.push(i));
    want.push_back(i);
  }
  assert(!q.push(999));
  for (int c = 0; c < cycles; ++c) {
    for (std::int32_t v : want) {
      auto got = q.pop();
      assert(got.has_value());
      assert(*got == v);
      assert(q.push(v));
    }
  }
  assert(q.size() == cap);
}

int main() {
  for (std::size_t cap : {1u, 2u, 3u, 5u, 7u, 10u, 15u}) {
    roundtrip(cap, 4);
  }
  BoundedQueue long_run(10);
  for (int i = 0; i < 10000; ++i) {
    assert(long_run.push(i));
    auto v = long_run.pop();
    assert(v.has_value());
    assert(*v == i);
  }
  return 0;
}

Those cases are cheap. They are also exactly the inputs a model will not hard-code if it never saw them. If a later author copies this file into the prompt, the harness is compromised and the OVERFIT label stops meaning anything.

Step 3 — Grade the diff, not the chat transcript

scripts/grade.sh is the reproducible core. It copies sources to a work tree, applies one unified diff, builds two binaries, and prints a single token the rest of a pipeline can parse.

#!/usr/bin/env bash
set -euo pipefail
ROOT=$(cd "$(dirname "$0")/.." && pwd)
PATCH=${1:?usage: grade.sh patches/foo.diff}
WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT

cp -R "$ROOT/include" "$ROOT/src" "$ROOT/tests" "$ROOT/oracles" "$WORK/"
(cd "$WORK" && patch -p1 < "$PATCH")

CXXFLAGS=(-std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -Iinclude)

visible_status=0
hidden_status=0
g++ "${CXXFLAGS[@]}" -o "$WORK/visible" src/bounded_queue.cpp tests/visible.cpp \
  && "$WORK/visible" || visible_status=$?
g++ "${CXXFLAGS[@]}" -o "$WORK/hidden" src/bounded_queue.cpp oracles/hidden.cpp \
  && "$WORK/hidden" || hidden_status=$?

if [[ $visible_status -ne 0 && $hidden_status -ne 0 ]]; then echo FAIL_BOTH; exit 2; fi
if [[ $visible_status -ne 0 ]]; then echo FAIL_VISIBLE; exit 3; fi
if [[ $hidden_status -ne 0 ]]; then echo OVERFIT; exit 4; fi
echo PASS

Run it on a known-bad patch first. A harness that cannot fail is not a harness.

chmod +x scripts/grade.sh
# labeled example: apply a recorded overfit diff
./scripts/grade.sh patches/power_of_two_mask.diff
# expected line: OVERFIT

Keep a second recorded diff that actually uses count_ and % cap_ (or a growable index into a std::vector). That file should print PASS. If both diffs print PASS, the hidden oracle is too weak.

Step 4 — Record fail modes instead of a single score

A numeric “quality” score hides the decision. The table is the artifact the team can argue about.

Visible	Hidden	Label	Typical cause
fail	fail	`FAIL_BOTH`	patch missed the bug, or broke compilation
fail	pass	`FAIL_VISIBLE`	rare; oracle weaker than the demo, fix the oracle
pass	fail	`OVERFIT`	special-cased prompt tests, wrong wrap, dropped invariant
pass	pass	`PASS`	candidate for human review, not an auto-merge

FAIL_VISIBLE with a green hidden binary is a harness bug. Stop generating patches and repair the oracle. OVERFIT is the label this article exists to make cheap to produce. PASS still needs a reviewer. Sanitizers catch a slice of undefined behavior. They do not catch a wrong API shape.

Step 5 — Spend free codegen trials on the loop, not on the grader

The grader is local g++ plus two binaries. The model is the only part that needs a remote coding endpoint. Repeated retries are where token budget actually matters, because each failed OVERFIT should feed a new attempt with the visible tests unchanged and the hidden file still hidden.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding project that, per the operator, offers free model access on the order of 10 million tokens and a free server option. Those two availability claims are the only product facts used here. No model names, hardware, quota clocks, or accuracy numbers are attached, because those change and this article does not re-measure them.

A practical split looks like this. The prompt and visible tests go to the coding server. The patch comes back as a diff. grade.sh runs on the maintainer’s machine or in ordinary CI. If the label is OVERFIT, the next prompt may say the visible tests must keep passing and that wrap must work for non-power-of-two capacities, without ever attaching oracles/hidden.cpp. The hidden file remains the judge, not the lesson.

Readers who already keep a C++ golden-case folder can point that same grader at patches from a free coding server instead of pasting diffs by hand.

What this method refuses to claim

Hidden oracles are still finite. A queue that passes capacities {1,2,3,5,7,10,15} can fail at 64 KiB or under a concurrent producer the fixture never starts. The harness does not replace thread-safety review, ABI review, or a design discussion about whether a ring buffer was the right object.

Do not leak the oracle. Once hidden cases appear in the prompt, the model can overfit the second file and the taxonomy collapses to “both tests were in the chat.” Treat oracle leakage like test-gold leakage in a programming contest.

Do not run untrusted generated patches unsandboxed on a laptop that mounts secrets. ASan is not a jail. Apply diffs inside a throwaway directory or container. The script above uses mktemp; that is a start, not a security boundary for hostile input.

Do not auto-merge on PASS. The label means “the cases we bothered to hide did not fail.” Cheap code still accumulates debt when the oracle lags the product.

Who should skip this harness

Skip it when the change is a comment-only edit or a build-system tweak with no runtime. Skip it when there is no compiler toolchain and no way to run sanitizers. Skip it when the specification is interactive UI with no executable property. Skip it when the team will paste the hidden file into every prompt “to help the model.” At that point the second layer is theatre.

Teams that already grade only with the compiler are not done. Compile-clean overfit is the case this fixture is built to name. Teams that already maintain a large property-based suite can swap oracles/hidden.cpp for that suite and keep the same exit taxonomy. The structure travels. The ring buffer does not have to.

The maintainer in the opening scene did not need a larger model. They needed a test the model was not allowed to read. Once that file exists, every later prompt change, server change, or retry policy can be scored with the same three tokens: FAIL_BOTH, OVERFIT, PASS. The demo can stay green. The oracle still gets a vote.

Undeclared Includes, Invented Helpers: An Assumption Ledger for AI C++ Patches

Finley Li — Fri, 04 Sep 2026 17:22:27 +0000

A maintainer merged an AI-written C++ patch because the golden unit test still returned the cached size. The function compiled. The assertion passed. The diff also planted a process-wide static std::mutex that no test ever locked from a second thread. Two days later a request handler blocked on that mutex during shutdown. The golden case had scored the return value. It had never scored what the model assumed it was allowed to invent.

Behavioral oracles are necessary. They are not sufficient. Agent-style coding loops now emit plausible patches that satisfy the demo and still smuggle headers, helpers, widened lock scopes, and dropped const into the tree. Those undeclared assumptions are a distinct failure class. This article treats them as first-class eval failures, not as style nits.

The workflow below is a prompt and eval harness. It keeps golden cases for behavior. It adds a machine-readable assumption ledger for everything the model is not allowed to introduce. The grader is small enough to run on a laptop or on a free coding server. It does not claim to prove correctness. It claims to fail loud when the patch quietly rewrites the contract.

The failure the tests never saw

Consider a size cache that must stay lock-free on the read path. A typical prompt asks the model to fix a stale-read bug. A typical golden test checks that two calls return the same size_t. A typical model “fixes” the race by adding a static mutex and locking both the write and the read. The test stays green. The original invariant is gone.

The same pattern shows up as an invented ensure_ready() helper, a new #include <thread>, a file-scope std::atomic_flag, or a removed noexcept. Each of those can be a legitimate design change. None of them should land because a codegen loop guessed. The ledger makes the allowed surface explicit before the prompt is sent.

Ledger format before the prompt

Store the contract next to the golden case, not inside the chat. YAML is enough for a first harness. JSON works if the rest of the pipeline already speaks it.

# cases/size_cache/assumptions.yaml
case_id: size_cache_stale_read
language: cpp
allowed_headers:
  - cstddef
  - cstdint
forbidden_headers:
  - mutex
  - thread
  - shared_mutex
allowed_new_symbols: []
forbidden_constructs:
  - static_mutex
  - file_scope_atomic
  - sleep_for
must_preserve:
  - const on SizeCache::size
  - noexcept on SizeCache::size
  - no new non-const static in translation unit
behavior_oracle: ./cases/size_cache/test_size_cache

The prompt then cites the ledger instead of restating it in prose. Models paraphrase constraints. Files do not. If a later prompt edit drops a sentence, the grader still reads the same YAML.

Label the following as an example harness, not as a published benchmark. It was not executed against a vendor leaderboard for this article.

Artifact: a three-stage grader

The harness has three stages on purpose. Compile and run still matter. Assumption scoring sits between parse and execute so a green test cannot hide a contract breach.

Parse the unified diff into added includes, added symbols, and stripped qualifiers.
Compare those additions against assumptions.yaml.
Build and run the behavior oracle only if the ledger is clean.

# grade_assumptions.py — example harness, not a scored study
from __future__ import annotations

import re
import subprocess
import sys
from pathlib import Path

import yaml

INCLUDE_RE = re.compile(r"^\+\s*#include\s*[<"]([^>"]+)[>"]", re.M)
STATIC_MUTEX_RE = re.compile(r"^\+.*static\s+std::mutex\b", re.M)
CONST_SIZE_RE = re.compile(r"\bsize\s*\(\s*\)\s*const\s*noexcept")


def load_ledger(path: Path) -> dict:
    return yaml.safe_load(path.read_text())


def added_headers(diff: str) -> set[str]:
    names = set()
    for match in INCLUDE_RE.finditer(diff):
        names.add(Path(match.group(1)).name)
    return names


def grade(diff: str, ledger: dict) -> list[str]:
    failures = []
    headers = added_headers(diff)
    forbidden = set(ledger.get("forbidden_headers", []))
    sneak = sorted(headers & forbidden)
    if sneak:
        failures.append(f"undeclared headers: {sneak}")
    if "static_mutex" in ledger.get("forbidden_constructs", []) and STATIC_MUTEX_RE.search(diff):
        failures.append("undeclared static mutex")
    if "const on SizeCache::size" in ledger.get("must_preserve", []):
        if "size() const noexcept" in diff and not CONST_SIZE_RE.search(diff):
            # stripped in a changed signature line
            if re.search(r"^[-].*size\s*\(.*const", diff, re.M) and re.search(
                r"^\+.*size\s*\(", diff, re.M
            ):
                failures.append("dropped const/noexcept on SizeCache::size")
    return failures


def run_oracle(cmd: str) -> int:
    return subprocess.call(cmd, shell=True)


def main() -> int:
    diff = Path(sys.argv[1]).read_text()
    ledger = load_ledger(Path(sys.argv[2]))
    failures = grade(diff, ledger)
    if failures:
        print("ASSUMPTION_FAIL")
        for item in failures:
            print(f"- {item}")
        return 2
    code = run_oracle(ledger["behavior_oracle"])
    if code != 0:
        print("BEHAVIOR_FAIL")
        return code
    print("PASS")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

A regex grader will miss macros and clever typedefs. That is an accepted limitation, recorded later. The point of the first version is a deterministic, reviewable fail code: 2 for assumption leakage, non-zero from the oracle for behavioral breakage. Silent green is the outcome this harness is built to destroy.

Golden case that should fail the mutex “fix”

Keep the production-shaped snippet tiny. The ledger is the interesting part.

// cases/size_cache/size_cache.hpp
#pragma once
#include <cstddef>

class SizeCache {
  std::size_t cached_{0};
public:
  void store(std::size_t n) { cached_ = n; }
  std::size_t size() const noexcept { return cached_; }
};

// cases/size_cache/test_size_cache.cpp
#include "size_cache.hpp"
#include <cassert>

int main() {
  SizeCache c;
  c.store(4);
  assert(c.size() == 4);
  assert(c.size() == 4);
  return 0;
}

Feed the model a patch prompt that states the stale-read symptom and points at assumptions.yaml. Save the model diff to out.patch. Then run the grader before anyone opens a pull request.

python3 grade_assumptions.py out.patch cases/size_cache/assumptions.yaml

Expected outcomes for this example, if the model adds #include <mutex> and a static lock: exit status 2 and ASSUMPTION_FAIL. Expected outcome if the model only updates store under an existing, already-declared discipline and the assert still holds: PASS. Do not treat those two strings as measured model accuracy. They are the harness contract.

Prompt packaging that the grader can audit

Put the prompt in the same directory. Version it. Diff it. The eval loop should hash prompt, ledger, and source together so a “better” model run is not actually a quieter constraint.

You are patching SizeCache in size_cache.hpp.
Honor cases/size_cache/assumptions.yaml exactly.
Do not add headers or symbols unless they appear in allowed_headers
or allowed_new_symbols.
Do not introduce static synchronization.
Keep SizeCache::size const and noexcept.
Return a unified diff and nothing else.

When an agent loop is allowed to call tools, the same ledger becomes a tool-side check. The agent can compile. It can run test_size_cache. It still fails if it “helpfully” includes <mutex>. That is the whole design: the extra degree of freedom agents enjoy is also an extra degree of assumption.

Where a free coding server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Running this loop by hand on one patch teaches the format. Running it on every prompt revision needs a place that can apply diffs, compile C++, and keep the YAML next to the source. MonkeyCode is an open-source coding-server project that, as supplied for this article, offers free model access on the order of 10 million tokens and a free server option. Those two availability claims are the only product facts used here. No model names, hardware profiles, latency numbers, or permanence guarantees are attached.

The harness does not depend on that server. A local clang++ and the Python grader already reproduce the method. The free tier is relevant when the same golden case must be sent through several prompt revisions without burning a paid quota on the evaluation traffic. Token budget then belongs to the eval loop, not to a demo chat.

A single, optional next step for readers who already own a C++ oracle is to point this ledger at whatever coding server they can compile on, including MonkeyCode’s free server if that is the stack under test.

Numbered workflow for a prompt change

Use the same sequence every time a system prompt, a tool list, or a “be careful with threads” paragraph changes. Prompt edits are code edits. They ship silent regressions if only the happy-path test is rerun.

Freeze assumptions.yaml before editing the prompt. If the allowed surface must grow, change the ledger in the same commit as the prompt.
Keep one golden case that the old prompt already passed, including a case that must stay ASSUMPTION_FAIL when a mutex is introduced.
Generate a patch with the new prompt. Save the raw diff. Do not hand-edit it before grading.
Run grade_assumptions.py. Treat exit 2 as a product regression, not as a style comment.
Run the behavior oracle only after the ledger is clean.
Record prompt hash, ledger hash, and exit class (PASS / ASSUMPTION_FAIL / BEHAVIOR_FAIL) in a log the next revision can diff.

Step 2 is easy to skip. It is the step that catches a new prompt that “helps” by locking. A suite of only happy paths will celebrate that patch.

What this does not catch

A header-name check does not understand #include via a wrapper header. A regex does not understand using Lock = std::mutex. Dropped const on a template specialization in another file will sail through. Data races that never touch a forbidden construct still need ThreadSanitizer or a real lock-order review. The ledger is a contract filter. It is not a replacement for sanitizers, code review, or a memory model.

The behavior oracle in the example is a single-threaded assert. That is intentional for the article. Production caches need concurrent tests. Those tests still fail to notice an undeclared mutex if they never contend. Run both layers.

Do not cite this harness as evidence that one host, model, or free tier is more accurate than another. No timing table is included because none was measured for this write-up.

Who should not use this approach

Teams without a compile-and-run oracle should not pretend the ledger is an eval. A diff linter alone will rubber-stamp a patch that does not compile. Security-sensitive code should not accept a green ledger as authorization to skip human review. Beginners who cannot state must_preserve in one page of YAML will encode the wrong contract and then automate it.

The method also wastes time on throwaway snippets with no invariants. If the file is a 20-line experiment and every header is in play, write a unit test and stop. Assumption ledgers pay off when the surrounding code already has ownership, const, and lock discipline that a helpful agent will otherwise “improve.”

Closing the loop

Golden tests answer whether the patch still returns 4. The assumption ledger answers whether the patch was allowed to invent a mutex to get there. Agent coding loops fail the second question more often than the first. Score both. Keep the YAML next to the prompt. Fail the build on undeclared includes before the unit test gets a chance to lie.

Zero-Infra CI Gate: Auto-Verify AI-Generated C++ Patches

Finley Li — Fri, 04 Sep 2026 13:07:35 +0000

A zero-infrastructure CI gate verifies AI-generated C++ patches by sending each pull-request diff to a free AI server, applying the returned unified diff on an isolated worktree, and running your existing C++ tests—with no extra servers. I use this pattern to reject patches that fail to compile or break tests before they consume a maintainer’s review slot.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier and free server options are operator-supplied claims. Token quotas, model lists, and server limits change over time, so verify the current terms before wiring this into your CI.

Why a CI Gate Beats Manual Review of AI C++ Patches

AI-generated patches land in my repositories faster than I can review them. A single maintainer might review about ten diffs per hour. A CI runner can compile and test far more in the same window. That gap is the bottleneck, and it grows every time a contributor pastes model output into a pull request.

Automation does not replace human review. It filters the obvious failures first. I compare the two approaches like this:

Manual-only review — a person reads every diff, including ones that do not compile. Throughput stays near ten diffs per hour.
CI gate, then review — the runner applies the AI patch, builds, and runs ctest. Only patches that apply, compile, and pass existing tests reach a person.

A patch that does not compile, or that breaks an existing test, should never wait in a review queue. Only the survivors reach a person.

The GitHub Actions documentation makes this pattern straightforward: I define a workflow once, and it runs on every relevant pull request. No new VMs, no extra infrastructure—just a YAML file in the repository. Compared with standing up a dedicated review bot or a private runner fleet, this approach reuses the GitHub-hosted minutes I already have and the CMake/ctest suite I already maintain.

The trade-off is intentionally narrow. The gate answers one question: does this AI patch apply cleanly, compile, and pass the tests I already trust? It does not answer whether the fix is semantically correct under load or against untested edge cases.

How the Four-Stage Pipeline Works

I keep the workflow to four stages so each failure is obvious in the Actions log:

Trigger — fire on opened and synchronize pull-request events that touch source or test paths.
Generate — send the diff to the MonkeyCode server with a narrow prompt that asks for a minimal unified-diff fix, not an explanation.
Apply — apply the returned patch to a clean worktree, never the runner’s primary checkout.
Verify — run the project’s existing test suite and post the result back to the pull request.

Isolation is the design decision I will not drop. The AI never touches the real working tree. Every candidate patch lands in a temporary directory, so a malformed diff cannot corrupt the checkout the workflow still needs for reporting.

Two cheap guards sit around that isolation:

Path filter — a documentation-only change skips the AI round-trip, which saves both tokens and runner minutes. If your tree uses include/ or lib/ instead of src/, change the globs rather than removing the filter.
Unified diff, not a full rewrite — the payload is smaller than a whole-file replacement, and git apply --check can reject a bad patch before CMake starts.

If the generate step returns prose instead of a patch, I want the job to fail at JSON extraction, not at a mysterious empty build.

Step-by-Step GitHub Actions Implementation

The workflow is about sixty lines of YAML. I copy it into .github/workflows/ai-patch-verification.yml and then swap paths and build commands. The sequence below is the same one I run.

Trigger only on C++ source and tests

name: ai-patch-verification

on:
  pull_request:
    types: [opened, synchronize]
    paths:
      - 'src/**'
      - 'tests/**'

types covers a new PR and later pushes to the same branch, so an updated AI patch gets re-checked. paths is the cheapest skip: a README change does not need this job.

Request a minimal fix from the free server

Checkout with full history so git diff origin/main...HEAD is accurate. Write the PR diff to a temp file, then POST it with a prompt that forbids prose.

jobs:
  verify-ai-patch:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get diff
        id: diff
        run: |
          git fetch origin main
          git diff origin/main...HEAD > /tmp/changes.patch
          echo "patch_size=$(wc -c < /tmp/changes.patch)" >> "$GITHUB_OUTPUT"

      - name: Request fix from MonkeyCode server
        id: ai
        env:
          MC_API_KEY: ${{ secrets.MONKEYCODE_API_KEY }}
        run: |
          jq -n --arg diff "$(cat /tmp/changes.patch)" \
            '{prompt: "The attached diff contains a bug. Return only a unified diff that fixes it. Do not explain.", diff: $diff}' \
            | curl -s -X POST \
              -H "Authorization: Bearer $MC_API_KEY" \
              -H "Content-Type: application/json" \
              -d @- > /tmp/ai-response.json
          jq -r '.patch' /tmp/ai-response.json > /tmp/fix.patch
          echo "fix_size=$(wc -c < /tmp/fix.patch)" >> "$GITHUB_OUTPUT"

Store MONKEYCODE_API_KEY as a repository secret. The jq extract fails the job early if the JSON has no patch field—better than applying an empty file and wondering why CMake did nothing. If your default branch is not main, change both the fetch and the diff bases.

Apply the patch in an isolated worktree

The git worktree documentation is why I prefer a second checkout over git apply in the primary tree: the reporting job still has a clean repo if the patch is garbage.

      - name: Apply patch to clean worktree
        run: |
          git worktree add /tmp/clean-checkout origin/main
          cd /tmp/clean-checkout
          git apply --check /tmp/fix.patch
          git apply /tmp/fix.patch

      - name: Run test suite
        working-directory: /tmp/clean-checkout
        run: |
          cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
          cmake --build build -j2
          ctest --test-dir build --output-on-failure

git apply --check is the guard. A patch that fails this check is rejected without touching the build. I use -j2 on GitHub-hosted runners to stay inside typical two-core machines; bump it if you use larger runners.

If the project is not CMake, replace those three commands with your real build—ninja, make test, or a wrapper script. The isolation pattern stays the same.

Report the verdict on the pull request

The GitHub CLI documentation covers gh pr comment. I post on every outcome so the PR timeline shows a pass or a fail without opening the log first.

      - name: Post result to PR
        if: always()
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          if [ "${{ job.status }}" = "success" ]; then
            gh pr comment "${{ github.event.pull_request.number }}" \
              --body "AI patch verification passed. The fix compiles and all tests pass."
          else
            gh pr comment "${{ github.event.pull_request.number }}" \
              --body "AI patch verification failed. See the workflow log for details."
          fi

if: always() posts even when tests fail. A green check is not a correctness proof; it is a signal that the patch is worth a human’s time.

Limitations and When I Would Not Use This

This pipeline verifies that a patch compiles and passes existing tests. It does not prove the fix is correct. A data race can pass a single ctest run and still crash under load. Teams that need stronger guarantees should add sanitizers and stress tests to the same workflow, not treat the AI response as a review.

The free tier is not a replacement for a dedicated CI budget. The token allowance covers a limited number of patches per period. Consider a project that receives twenty AI-assisted pull requests per week. Each request sends a diff of roughly five kilobytes and receives a patch of similar size. The token cost per request is small, but the cumulative count adds up. I log token usage from the API response and compare it against the allowance at the end of each month.

Teams with proprietary code should check whether the free server keeps their data inside their network. The self-hosted option exists for that reason, but it shifts the operational burden onto the team.

I skip this gate when any of the following is true:

The team cannot send diffs to a third-party server.
We need a specific model version pinned for reproducibility.
Patch volume exceeds the free allowance and a mid-sprint pause is unacceptable.

What to Do Next

Copy the workflow into .github/workflows/ai-patch-verification.yml, adjust the paths globs and the CMake/ctest commands, and add MONKEYCODE_API_KEY as a repository secret. Then run this smoke check:

Open a small, intentionally broken pull request that touches src/ or tests/.
Confirm the isolated worktree rejects a bad patch.
Push a clean patch to the same branch and confirm the job passes and comments on the PR.

If you hit a snag, or you want a follow-up on adding sanitizers and stress tests in the same job, leave a comment below. I read them all.

MonkeyCode provides free models that can run this workflow.

DEV Community: Finley Li

Zero Hits on the Hunk: A Coverage Gate for AI C++ Patch Evals

Blind goldens are a different bug class

Decision table before any flags

Artifact: a coverage envelope

1. Record the hunk, not the whole tree

2. Rebuild with instrumentation

3. Run the existing goldens, nothing extra

4. Map hits onto the patch

5. Fail closed, then keep the report

How this relates to models that outgrow their tests

Where a spare model host fits

What this envelope does not prove

Who should skip this gate

A pinned command the CI can copy

Same Goldens, New Question: A Prompt-Pin Manifest for AI C++ Evals

What the scorecard usually forgets

Eval identity, not a vibe

Artifact: eval_manifest.json

Layout on disk

Step 1 — Freeze the identity

Step 2 — Fail closed before any model call

Step 3 — Keep goldens that the new question can still fail

Step 4 — Record the candidate only after identity holds

Decision table: what to bump

Where a free remote runner fits

Limitations

What “green” is allowed to mean

The Placebo Patch: Measure Your C++ Eval Harness's Noise Floor Before You Trust a Regression

The number nobody records

Step 1 — Build a placebo corpus

Step 2 — Run the matrix

Step 3 — Turn verdicts into a per-case admission report

Step 4 — Gate on the promotable set only

Step 5 — Re-measure instead of inheriting

Limitations

Who should skip this

Closing

Must-Not-Compile Is a Spec: A Negative-Compilation Envelope for AI C++ Patches

The hole in treating build success as progress

What the envelope records

Verdict table

A minimal library under test

Manifest and grader

Numbered workflow for an AI C++ patch eval

Where generation and grading split

Compiler notes that break naive graders

Limitations

Who should not use this

Closing

Asserts Passed, RSS Doubled: An rusage Envelope for AI C++ Patches

Cost is part of the contract

What the envelope records

Worked example: the cache that never evicted

Header under test

Driver the harness actually times

Committed ceilings

The rusage grader

Numbered workflow

Where generation and grading can share a host

How the silent case shows up

Limitations

Who should skip this approach

Closing

Release Builds Are a Different Spec: A Flag-Matrix Harness for AI C++ Patches

Why a single compile lies

The artifact: a compile-profile ledger

1. Pin a subject that can fail two ways

2. Declare profiles as data, not comments

3. Drive the matrix from a shell runner

4. Require unanimous pass, then diff digests

5. Add one negative golden that Debug hides

Where free remote generation fits

Limitations

Who should not use this

What to keep in the eval, not the prompt

Exit Code Zero Is Not a Spec: A Digest Harness for AI C++ Patches

Why a pass/fail bit is the wrong oracle

The observable tuple

A fixture the tests will not defend

Artifact: `eval_manifest.json`