Coverage-Coupled Grading: Flag AI C++ Patch Lines That No Golden Case Ever Ran

A patch landed in our C++ service last quarter with a green board: 412 golden cases passing, no flakes, no sanitizer noise. Three days later a nightly batch job wrote garbage into a column that had never been read by a test. The changed line was real, the tests were real, and they simply never met.

That failure mode is the one I now design against: every golden case passes because no case ever executes the line the model touched. Coverage is not correctness. But coverage that is joined to the diff is a cheap, honest alarm — and it catches a class of silent regression that pass/fail grading cannot see.

This article is the harness I use for that join. It assumes you already grade AI-written C++ patches with golden cases and want one more gate before a patch is accepted.

1. The problem statement, stated precisely

A golden case gives you a boolean. A patch gives you a set of changed lines. The dangerous region is the intersection of "changed" with "never executed by any case" — unverified behaviour that still reports success.

The gate is therefore not "coverage must go up." It is narrower and much harder to game:

For every line an AI patch adds or modifies, at least one golden case must execute that line — or a human must sign off on the gap by name.

Two properties matter. It is per-line, so a large refactor cannot be waved through by an unrelated coverage increase. And it is per-case, so you learn which case owns the line, which is the first question anyone asks during review.

2. Build an instrumented binary once

Clang source-based coverage is the least painful option for per-case attribution. Build a second binary tree; never grade against the release build (optimisation and inlining move the line map underneath you).

cmake -S . -B build-cov \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_CXX_FLAGS="-fprofile-instr-generate -fcoverage-mapping -O0 -g" \
  -DCMAKE_EXE_LINKER_FLAGS="-fprofile-instr-generate"
cmake --build build-cov -j"$(nproc)"

One binary, one build, reused for every case. Rebuilding per case is where people give up on this idea.

3. Run each golden case in isolation

Per-case attribution needs one profile per case. The %p in the filename keeps forked children from overwriting each other.

rm -rf prof && mkdir -p prof
while read -r case_id; do
  LLVM_PROFILE_FILE="prof/case-${case_id}-pid%p.profraw" \
    ./build-cov/tests --case "$case_id" || echo "FAIL $case_id" >&2
done < cases.txt

Keep cases.txt under version control — it is part of the grader, not a scratch file. The harness lives as scripts, not prose checklists, because prose drifts and scripts get diffed.

4. Join coverage to the diff

The script below is the whole gate. It parses git diff -U0 for changed line numbers in the new file, exports per-case executed lines from llvm-cov, and emits three buckets: covered, uncovered, unmapped.

#!/usr/bin/env python3
# covgate.py — flag changed lines that no golden case executed.
# Usage: python3 covgate.py <repo-root> <instrumented-test-binary>
import json, re, subprocess, sys, pathlib, collections

ROOT = pathlib.Path(sys.argv[1])
BIN  = sys.argv[2]                      # e.g. build-cov/tests
PROF = ROOT / "prof"
BASE = "HEAD"                           # base rev of the model's patch
exec_sh = lambda cmd: subprocess.run(
    cmd, cwd=ROOT, text=True, capture_output=True, check=True).stdout

RAW = re.compile(r"^case-(.+)-pid\d+\.profraw$")

def per_case_profiles():
    cases = collections.defaultdict(list)
    for raw in sorted(PROF.glob("case-*-pid*.profraw")):
        m = RAW.match(raw.name)
        if m:
            cases[m.group(1)].append(raw)
    return cases

def executed_lines(raws, tag):
    merged = PROF / f"merged-{tag}.profdata"
    exec_sh(["llvm-profdata", "merge", "-sparse",
             *[str(r) for r in raws], "-o", str(merged)])
    doc = json.loads(exec_sh(["llvm-cov", "export", BIN,
                              "-instr-profile", str(merged)]))
    lines = collections.defaultdict(set)
    for unit in doc["data"]:
        for f in unit["files"]:
            for seg in f["segments"]:
                line, _col, count, has_count = seg[0], seg[1], seg[2], seg[3]
                if has_count and count > 0:
                    lines[f["filename"]].add(line)
    return lines

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def changed_lines():
    diff = exec_sh(["git", "diff", "-U0", "--no-color", BASE,
                    "--", "*.cpp", "*.cc", "*.h", "*.hpp"])
    out, cur = collections.defaultdict(set), None
    for ln in diff.splitlines():
        if ln.startswith("+++ b/"):
            cur = ln[6:]
        m = HUNK.match(ln)
        if m and cur:
            start, n = int(m.group(1)), int(m.group(2) or 1)
            out[cur].update(range(start, start + n))
    return out

norm = lambda p: p.replace("\\", "/")

def main():
    cases = per_case_profiles()
    if not cases:
        sys.exit("no .profraw files — run step 3 first")

    union = collections.defaultdict(set)
    for cid, raws in cases.items():
        for f, ls in executed_lines(raws, cid).items():
            union[norm(f)] |= ls

    report = {"covered": [], "uncovered": [], "unmapped": []}
    for rel, lines in changed_lines().items():
        keys = [k for k in union if k.endswith("/" + norm(rel))]
        if not keys:
            report["unmapped"].append({"file": rel, "lines": sorted(lines)})
            continue
        hit = set().union(*(union[k] for k in keys))
        for line in sorted(lines):
            bucket = "covered" if line in hit else "uncovered"
            report[bucket].append({"file": rel, "line": line})

    json.dump(report, sys.stdout, indent=2)
    sys.exit(1 if report["uncovered"] else 0)

if __name__ == "__main__":
    main()

Exit code 1 means "at least one changed line is unexecuted." That single bit is what you wire into the grader.

5. Read the three buckets with different eyes

Not every bucket means the same thing, and treating them uniformly is how this gate gets switched off in week two.

Bucket	Meaning	Default action
covered	Some case executed the line	Accept; log which case for review context
uncovered	Changed, executable, never run	Reject, or require a named human waiver
unmapped	Changed file absent from coverage data	Investigate: header-only code, uninstrumented target, or a file the binary never links

The unmapped bucket is the one people skip and later regret. A header that is compiled into a different translation unit than the test binary's will look unmapped forever; if you never look, the gate silently has a hole exactly where template-heavy code lives.

6. Where free model access and a free server actually fit

The gate above only works if you run it on many candidate patches, because a gate that blocks one patch a week is theatre. Fan-out is the cost driver: k candidates per failing case, each needing a build, a case sweep, and a coverage export.

I use MonkeyCode's free model access for the wide, disposable part of that fan-out — the candidates I expect to reject — and reserve paid capacity for the narrow set that survives. The operator states a free allowance of roughly 10 million tokens and a free server option as of this writing; confirm the current terms on the project's own page before you build a budget around them, since allowances change.

The free server matters for a less obvious reason: coverage exports are I/O-heavy and bursty, and they do not belong on the same box as latency-sensitive CI. A separate small host running the sweep on a schedule keeps the main pipeline honest. The operator describes MonkeyCode as an open-source coding-agent project; check the repository for the licence and current feature set rather than trusting a blog post, including this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

7. Limitations, stated before someone finds them for me

• Coverage is not assertion. A line executed and never checked is still unverified. This gate catches "never ran," not "ran and proved nothing."
• Debug builds lie about hot paths. -O0 changes inlining and template instantiation; a line covered in build-cov may not exist in the release binary. Keep a separate release-split check rather than merging the two concerns.
• segments is an approximation. Region-based line attribution can mark a line executed when only part of its expression ran. Treat the output as a strong hint plus an audit trail, not a proof.
• Diff base is a policy choice. HEAD works when the patch is uncommitted; otherwise pin the merge base explicitly and record it in the report.
• Header instrumentation explodes build time. On a heavy template codebase expect the instrumented build to dominate everything else in this workflow.

Do not adopt this if you have no isolated per-case runner, no instrumented build target, or a test suite where most cases share one giant fixture — you will get uncovered noise on lines that are genuinely exercised, and the gate will lose its credibility in a week.

8. What to do next

The order matters more than the tooling. First get per-case profiles, then get the diff join, then add the gate to the grader only after you have hand-checked twenty reports against reality. A gate nobody has audited is just another number that goes green.

If you want to try the fan-out half cheaply, run this harness against a handful of candidate patches from MonkeyCode's free tier on a spare host, and compare the uncovered list to what a human reviewer actually flagged. That comparison — not the pass rate — is the number worth tracking.