Zero Hits on the Hunk: A Coverage Gate for AI C++ Patch Evals

A C++ service accepted an AI patch on a Friday afternoon. The golden suite stayed green. The reviewer saw a plausible rewrite of decode_frame() and merged.

Monday's on-call page was a truncated header from a partner. decode_frame() had never run under the goldens. Those cases only constructed payloads that skipped the new branch. The harness had measured a function the patch did not touch.

This failure mode is not a flaky test. It is a blind one. When models outgrow the cases used to score them, a green golden file can mean the suite never called the hunk.

Blind goldens are a different bug class

Output digests catch silent changes in printed bytes. Resource envelopes catch RSS spikes. Flag matrices catch -O2 surprises. None of those checks ask whether any golden executed the lines the model rewrote.

Coverage is a weak proxy for correctness. It is a strong proxy for contact. A hunk with zero hits was not evaluated. Treating that result as a pass is a category error.

The rest of this article is a small, reproducible envelope. It fails the eval when patched executable lines report zero hits under the existing goldens. The method is a harness design, not a published benchmark.

Decision table before any flags

The gate is a predicate on two facts the CI already has: the golden result, and hit counts on the diff.

Golden result	Scored hunk lines	Envelope
red	any	reject; the suite already failed
green	every listed line count > 0	coverage accept only
green	some listed line counts = 0	reject
green	no executable patched lines scored	reject; suite and patch did not overlap

A coverage accept is not a merge. Digest checks, sanitizers, and must-not-compile cases still apply. This table only fills the hole where the suite never touched the rewrite.

Artifact: a coverage envelope

The envelope has five steps. Each fragment can be copied into a CI job. Replace the sample paths with the repository under test.

1. Record the hunk, not the whole tree

Coverage of untouched files is noise. The gate should score the patch.

#!/usr/bin/env bash
# record_hunk.sh — example workflow, not a measured production run
set -euo pipefail
BASE="${1:-origin/main}"
git diff -U0 "$BASE"...HEAD -- '*.cpp' '*.cc' '*.cxx' '*.h' '*.hpp' \
  > /tmp/patch.diff

python3 - <<'PY'
import re, json
hunks = {}
path = None
for line in open("/tmp/patch.diff"):
    if line.startswith("+++ b/"):
        path = line[6:].strip()
        hunks.setdefault(path, [])
        continue
    m = re.match(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@", line)
    if m and path:
        start = int(m.group(1))
        length = int(m.group(2) or "1")
        hunks[path].extend(range(start, start + length))
json.dump(hunks, open("/tmp/hunk_lines.json", "w"), indent=2)
print("wrote /tmp/hunk_lines.json")
PY

The JSON map is the only input the grader trusts for what changed. Untracked generated files stay out of the map unless the operator adds them on purpose.

2. Rebuild with instrumentation

Compile the test binary the same way the goldens already compile it, plus coverage flags. Mixing -O2 and --coverage is allowed, but the flag set should stay pinned. A coverage build is a different translation. Do not compare its timings against a release binary.

cmake -S . -B build-cov \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage -fno-inline -fno-elide-constructors"
cmake --build build-cov -j"$(nproc)"

-fno-inline keeps hunk lines from vanishing into a caller. That flag is a measurement choice. It is not a claim that inlining is unsafe in production.

3. Run the existing goldens, nothing extra

Do not add tests in this step. The point is to measure the suite that already claims to accept the patch.

cd build-cov
find . -name '*.gcda' -delete
ctest --output-on-failure

If the goldens fail, the coverage gate does not run. A red suite is already a reject. The envelope only speaks when the goldens are green.

4. Map hits onto the patch

GCC can emit gcov JSON. Parse that, not the text reports. GCC 9 and later accept --json-format.

# from the build tree, after goldens
while IFS= read -r gcno; do
  gcov -b -c --json-format "$gcno" >/dev/null
done < <(find . -name '*.gcno')

A compact grader then joins hunk_lines.json with the gcov JSON.

# cov_gate.py — example grader, not an executed leaderboard
from __future__ import annotations

import json
import sys
from pathlib import Path

def load_gcov(dir: Path) -> dict[str, dict[int, int]]:
    hits: dict[str, dict[int, int]] = {}
    for p in dir.rglob("*.gcov.json"):
        data = json.loads(p.read_text())
        for f in data.get("files", []):
            src = f["file"]
            line_hits = hits.setdefault(src, {})
            for ln in f.get("lines", []):
                line_hits[int(ln["line_number"])] = int(ln.get("count", 0))
    return hits

def basename_match(hunk_path: str, gcov_path: str) -> bool:
    return Path(hunk_path).name == Path(gcov_path).name

def main() -> int:
    hunks = json.loads(Path("/tmp/hunk_lines.json").read_text())
    hits = load_gcov(Path("."))
    missed = []
    scored = 0
    for path, lines in hunks.items():
        file_hits = {}
        for gpath, lh in hits.items():
            if basename_match(path, gpath) or gpath.endswith(path):
                file_hits = lh
                break
        for n in lines:
            # skip lines gcov never lists (comments, braces-only)
            if n not in file_hits:
                continue
            scored += 1
            if file_hits[n] == 0:
                missed.append(f"{path}:{n}")
    report = {
        "scored_executable_lines": scored,
        "zero_hit_lines": missed[:50],
        "zero_hit_count": len(missed),
    }
    Path("/tmp/cov_gate.json").write_text(json.dumps(report, indent=2))
    if scored == 0:
        print("cov_gate: no executable patched lines were scored", file=sys.stderr)
        return 2
    if missed:
        print(f"cov_gate: {len(missed)} patched lines had zero hits", file=sys.stderr)
        return 1
    print("cov_gate: every scored patched line was hit")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Exit code 2 means the suite and the patch did not overlap at all. Exit code 1 means some patched executable lines stayed cold. Both are rejects under this envelope.

5. Fail closed, then keep the report

Store /tmp/cov_gate.json next to the golden digest. A later prompt pin or model swap can compare zero-hit counts the same way it compares hashes. The coverage gate is a predicate, not a leaderboard.

A toy program makes the trap obvious.

// src/codec.cpp
int decode_body(const char* p, int n) { return n > 0 ? p[0] : -1; }

int decode_frame(const char* p, int n) {
    if (n < 2) return -1;
    return (static_cast<unsigned char>(p[0]) << 8) |
           static_cast<unsigned char>(p[1]);
}

// tests/golden_codec.cpp
#include <cassert>
int decode_body(const char*, int);
int decode_frame(const char*, int);
int main() {
    const char body[] = "x";
    assert(decode_body(body, 1) == 'x');
    return 0;
}

An AI patch that "simplifies" decode_frame still leaves this golden green. The coverage gate reports zero hits on that function and rejects the eval. That is the entire point.

How this relates to models that outgrow their tests

Public debate in mid-September 2026 returned to a familiar claim: coding models already beat most developers, or that passing a test suite is the same as engineering. Those slogans are not a spec. A green golden file is evidence that the cases ran. It is not evidence that the cases touched the patch.

When a model learns the suite, it can keep outputs stable while moving behavior into untested functions. Coverage-on-the-hunk is a cheap tripwire for that pattern. It does not stop a model from writing a wrong line that a golden does execute. It stops the quieter failure: scoring a rewrite the suite never called.

Where a spare model host fits

Generating candidate patches and rebuilding with --coverage is CPU work. Local laptops throttle. A shared box that already compiles the goldens is enough.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here. This article does not name models, quote a token quota, or report a latency number. The coverage grader above runs on any Linux host with GCC, CMake, and Python 3. If the eval already lives in that toolchain, the free server option is one place to park overnight coverage gates without changing the predicate.

What this envelope does not prove

Coverage contact is not functional correctness. A line can run and still be wrong.

Inlined templates, constexpr evaluation, and macros can make gcov line numbers disagree with the diff. Basename matching is a convenience and can collide. Header-only libraries need -fkeep-inline-functions or the hunk disappears.

Death tests and exec tests may not write .gcda files. Parallel ctest without GCOV_PREFIX can clobber counters. The grader must run after a single, serialized golden pass or use per-test prefixes.

A 100% hit rate on the hunk can still hide a removed check. Digest comparison and negative-compilation cases remain necessary. This gate only fills the never-called hole.

Who should skip this gate

Skip it when the patch is comments, CMake, or pure data files. gcov will not score those paths, and a zero scored_executable_lines result is expected.

Skip it when the team already runs mutation testing on the same goldens. Mutation is stronger and more expensive. The coverage gate is the cheaper filter.

Skip it when the binary cannot be rebuilt with --coverage, including some vendor toolchains and freestanding firmware without a host test double.

Do not use the zero-hit count as a model ranking. It is a boolean envelope on a single patch. Ranking models by coverage invites the model to insert dead code that a later pass covers with a trivial call.

A pinned command the CI can copy

#!/usr/bin/env bash
set -euo pipefail
./record_hunk.sh origin/main
cmake -S . -B build-cov -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_FLAGS="--coverage -fno-inline"
cmake --build build-cov -j"$(nproc)"
( cd build-cov && find . -name '*.gcda' -delete && ctest --output-on-failure )
( cd build-cov && find . -name '*.gcno' -print0 | xargs -0 -n1 gcov -b --json-format >/dev/null )
python3 cov_gate.py

Pin the compiler version next to this script. Coverage IR changes across GCC releases. A gate that flips because the image moved from GCC 12 to GCC 13 is a harness bug, not a model regression.

The Friday merge in the opening would have failed step 5. The goldens would still have been green. The report would have listed src/codec.cpp lines inside decode_frame as zero-hit. That is a small, boring, and sufficient reason to reject the eval.