Green Tests, Drifted Symbols: A Witness Harness for AI C++ Patches

A release engineer froze a C++ CSV splitter at tag v0.4.2, then accepted an AI-written patch that claimed to speed up quoted-field parsing. Catch2 reported 41 passed, 0 failed. The shared library still loaded. Seven days later the distro packaging job broke: nm -D libsplit.so no longer listed split_row_alloc, because the model had marked the helper static inline in a header. Tests never called that symbol through the dynamic ABI. The suite stayed green. The binary drifted.

That failure mode shows up whenever generated C++ lands faster than reviewers can read every diff. Unit tests check the paths someone remembered to write. They do not check exported symbols, section sizes, sanitizer findings under a second optimization level, or the exact bytes a CLI writes to stderr on a bad flag. An eval harness that only asserts ctest therefore grades the demonstration, not the artifact.

This article proposes a witness harness. After the familiar compile-and-test loop, the grader records a small, committed snapshot of the binary and of the program's I/O contract. The next model patch must match that snapshot, or the eval fails closed. The method is toolchain-dependent and incomplete. It is also cheap enough to run on a laptop, and it does not require the model to be honest about what it changed.

What the witness actually records

A witness is not another golden unit test. It is a fingerprint of side effects the suite usually ignores. The proposed tape stores four files per fixture:

1. argv.json — argument vector and selected environment keys, including LC_ALL.
2. stdin.bin — exact bytes fed to the process, including empty files.
3. expect.json — exit code, stdout SHA-256, stderr SHA-256, timeout.
4. symbols.nm — sorted, demangled defined symbols from the linked target, plus one size line for text/data/bss.

If any of those files is missing, the harness does not skip. It fails. Missing evidence is a fail, not a pass. That single rule stops the silent path where a new fixture is added, the model “passes,” and nobody notices the witness was never recorded.

Cheap generation makes that path more common, not less. When C++ patches are easy to obtain, technical debt moves from missing tests into binaries that still satisfy the tests. The tape is there to make the binary argue back.

Repository layout

The following tree is a proposal, not a shipped product. Keep the production sources and the eval corpus in the same git repo so a patch cannot forget the tape.

eval/
  fixtures/
    csv_quote/
      argv.json
      stdin.bin
      expect.json
      symbols.nm
    bad_flag/
      argv.json
      stdin.bin
      expect.json
      symbols.nm
  witness.py
  matrix.cmake
src/
  split.cpp
  split.hpp
CMakeLists.txt

matrix.cmake encodes the compile grid. One configuration is not an eval. AI C++ patches routinely survive -O0 and explode under -O2 and -fsanitize=address. The grid below is deliberately small so a remote builder can finish it without a private GPU farm.

# matrix.cmake — proposed compile grid
set(WITNESS_CONFIGS
  "dbg|-O0 -g -Wall -Wextra -Werror"
  "rel|-O2 -DNDEBUG -Wall -Werror"
  "asan|-O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer"
)

A three-cell matrix will not replace vendor CI. It will catch the patch that introduced a signed overflow only visible at -O2. That is the usual way a fluent model output becomes a production crash.

Step 1 — Freeze the I/O contract

Write expect.json by running the current binary, not the model's story about the binary. The procedure below is labeled as a proposal.

# Proposed capture, run from a clean build/
export LC_ALL=C
./split --strict < ../eval/fixtures/csv_quote/stdin.bin \
  > /tmp/out.bin 2> /tmp/err.bin
echo $?
python3 - <<'PY'
import hashlib, json, pathlib
def sha(p):
    return hashlib.sha256(pathlib.Path(p).read_bytes()).hexdigest()
print(json.dumps({
    "exit": 0,
    "stdout_sha256": sha("/tmp/out.bin"),
    "stderr_sha256": sha("/tmp/err.bin"),
    "timeout_s": 5
}, indent=2))
PY

Commit those hashes. Do not commit raw stdout if the fixture might contain customer rows. Hashes are enough for regression and safer if the corpus later runs on a shared builder. Pin LC_ALL=C in argv.json as well, or a locale change will look like a model regression.

Step 2 — Freeze exported symbols

nm output is noisy. Pipe it through a filter that keeps only T and W symbols, drops compiler-generated guards, and sorts. The filter is the contract. Change the filter and every witness will churn, which is preferable to silently accepting a new export.

# Proposed symbol freeze for a C++ shared object or CLI
nm -C --defined-only ./libsplit.so \
  | awk '$2 ~ /^[TW]$/ { print $2, $3 }' \
  | grep -v '_GLOBAL' \
  | sort > ../eval/fixtures/csv_quote/symbols.nm
size ./libsplit.so | awk 'NR==2 { print "size", $1, $2, $3 }' \
  >> ../eval/fixtures/csv_quote/symbols.nm

Point nm at the .so when the packaging surface is a library. Point it at the CLI when the artifact is a statically linked tool. A header-only inline that used to be an exported T symbol becoming an inlined phantom is exactly the packaging break from the opening story. The size line is coarse. It still catches the patch that quietly pulled in another translation unit.

Step 3 — Grade a candidate patch fail-closed

witness.py is a proposed grader. It applies a unified diff, builds each matrix cell, runs each fixture, and compares witnesses. Any exception, missing file, or mismatch returns non-zero. There is no warn status. Reviewers can still read the log. Automation should not.

#!/usr/bin/env python3
"""Proposed fail-closed witness grader for AI C++ patches. Illustrative."""
from __future__ import annotations
import hashlib, json, os, subprocess, sys, tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
EVAL = ROOT / "eval" / "fixtures"
TIMEOUT = 30

class Fail(Exception):
    pass

def run(cmd, cwd, env=None, timeout=TIMEOUT, stdin=None):
    return subprocess.run(
        cmd, cwd=cwd, env=env, input=stdin,
        stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout,
    )

def sha256(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()

def build(cell_flags: str, work: Path) -> Path:
    src = work / "build"
    src.mkdir(exist_ok=True)
    cmake = run(
        ["cmake", "-S", str(work), "-B", str(src),
         f"-DCMAKE_CXX_FLAGS={cell_flags}"],
        cwd=work,
    )
    if cmake.returncode != 0:
        raise Fail(f"cmake failed: {cmake.stderr.decode()[-800:]}")
    ninja = run(["cmake", "--build", str(src), "-j"], cwd=work)
    if ninja.returncode != 0:
        raise Fail(f"build failed: {ninja.stderr.decode()[-800:]}")
    bin_path = src / "split"
    if not bin_path.exists():
        raise Fail("binary missing after build")
    return bin_path

def symbols_of(bin_path: Path) -> str:
    nm = run(["nm", "-C", "--defined-only", str(bin_path)], cwd=bin_path.parent)
    if nm.returncode != 0:
        raise Fail("nm failed")
    lines = []
    for line in nm.stdout.decode(errors="replace").splitlines():
        parts = line.split()
        if len(parts) >= 3 and parts[1] in {"T", "W"}:
            lines.append(f"{parts[1]} {parts[2]}")
    lines.sort()
    sz = run(["size", str(bin_path)], cwd=bin_path.parent)
    if sz.returncode == 0:
        rows = sz.stdout.decode().splitlines()
        if len(rows) >= 2:
            cols = rows[1].split()
            lines.append(f"size {cols[0]} {cols[1]} {cols[2]}")
    return "\n".join(lines) + "\n"

def grade_fixture(bin_path: Path, fx: Path) -> None:
    argv = json.loads((fx / "argv.json").read_text())
    expect = json.loads((fx / "expect.json").read_text())
    stdin = (fx / "stdin.bin").read_bytes()
    gold_nm = (fx / "symbols.nm").read_text()
    cmd = [str(bin_path), *argv.get("args", [])]
    env = os.environ.copy()
    env.update(argv.get("env", {}))
    env.setdefault("LC_ALL", "C")
    try:
        p = run(
            cmd, cwd=bin_path.parent, env=env,
            timeout=expect.get("timeout_s", 5), stdin=stdin,
        )
    except subprocess.TimeoutExpired as exc:
        raise Fail(f"{fx.name}: timeout") from exc
    if p.returncode != expect["exit"]:
        raise Fail(f"{fx.name}: exit {p.returncode} != {expect['exit']}")
    if sha256(p.stdout) != expect["stdout_sha256"]:
        raise Fail(f"{fx.name}: stdout hash mismatch")
    if sha256(p.stderr) != expect["stderr_sha256"]:
        raise Fail(f"{fx.name}: stderr hash mismatch")
    got = symbols_of(bin_path)
    if got != gold_nm:
        raise Fail(f"{fx.name}: symbol witness mismatch")

def main() -> int:
    patch = Path(sys.argv[1]) if len(sys.argv) > 1 else None
    cells = [
        "-O0 -g -Wall -Wextra -Werror",
        "-O2 -DNDEBUG -Wall -Werror",
        "-O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer",
    ]
    fixtures = sorted(p for p in EVAL.iterdir() if p.is_dir())
    if not fixtures:
        print("FAIL closed: no fixtures", file=sys.stderr)
        return 2
    with tempfile.TemporaryDirectory() as td:
        work = Path(td) / "src"
        subprocess.check_call(["git", "clone", "--local", str(ROOT), str(work)])
        if patch:
            applied = run(["git", "apply", str(patch.resolve())], cwd=work)
            if applied.returncode != 0:
                print("FAIL closed: patch does not apply", file=sys.stderr)
                return 2
        for flags in cells:
            try:
                binary = build(flags, work)
                for fx in fixtures:
                    for required in ("argv.json", "stdin.bin", "expect.json", "symbols.nm"):
                        if not (fx / required).exists():
                            raise Fail(f"{fx.name}: missing {required}")
                    grade_fixture(binary, fx)
            except Fail as exc:
                print(f"FAIL closed [{flags}]: {exc}", file=sys.stderr)
                return 1
    print("PASS: compile matrix and witnesses matched")
    return 0

if __name__ == "__main__":
    sys.exit(main())

The script clones the repo into a temp directory so a failing patch cannot leave object files in the working tree. That matters when the next eval reuses the same checkout, including on a shared builder. Sanitizer flags stay inside one matrix cell. Mixing ASan into the release cell would make size witnesses flap for reasons that have nothing to do with the patch.

Step 4 — Read the verdict table, not the model's summary

Models narrate. The harness should not. Map every exit path to a single token so a log scraper can count regressions without parsing English.

Token	Meaning	Typical cause
PATCH_REJECT	git apply failed	Wrong paths, already-applied hunk
CMAKE_FAIL	configure error	Invented options, missing C++ standard
BUILD_FAIL	compile or link error	Undeclared symbols, ODR splits
SAN_FAIL	ASan/UBSan abort	Use-after-free, signed overflow
IO_FAIL	exit/stdout/stderr mismatch	CLI drift, extra logging
SYM_FAIL	nm/size mismatch	Hidden ABI change
TIMEOUT	wall clock exceeded	Accidental quadratic parse
PASS	all cells and fixtures matched	Still not a ship decision

PASS means the tape did not move. It does not mean the patch is desirable. Size may stay within the witness because the size line is coarse. Reviewers still read the diff. The point of the table is to stop a chatty model from turning a SYM_FAIL into a paragraph about “equivalent refactoring.”

Where a free coding server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free-tier model access and a free server option. Those two availability claims matter to this workflow in a narrow way. The model is a patch factory. The server is an optional place to run witness.py when a laptop cannot hold three sanitizer builds at once. The grader remains the authority.

A generated patch that explains itself well and then changes split_row_alloc from exported to inline still exits SYM_FAIL. Hosted tokens do not grade C++. Compilers, sanitizers, and nm do. Treat any advertised allowance as a budget that can change, not as a capacity plan and not as a benchmark. No model names, queue times, or hardware claims are made here because they go stale.

A practical loop looks like this:

1. Store the witness tape in git next to the sources.
2. Ask the coding model for a patch against a single fixture name, not against “make it faster.”
3. Save the unified diff to candidate.patch.
4. Run python3 eval/witness.py candidate.patch.
5. If the result is not PASS, send the fail token and the last 40 log lines back into the next prompt. Do not paste the whole binary.

That loop is slow on purpose. Silent regressions are cheap to ship and expensive to notice. Spending a compile matrix per candidate is the point, not a defect.

Limitations

nm and size vary across binutils, LLVM, and MinGW. Witnesses are not portable between distros. Commit the toolchain version next to symbols.nm or the tape will flap on every image bump. Sanitizers miss data races that only TSan would see. TSan is omitted here because it disagrees with many allocators and would turn the grid into a flake factory.

SHA-256 of stdout is brittle against locale, date stamps, and trailing newlines. Fixtures must pin LC_ALL=C and refuse timestamps in CLI output. The harness does not prove functional equivalence for GUI programs, embedded firmware, or signal-heavy servers. It also does not prove that a larger binary is wrong. It only proves the binary is different.

Free-tier model access and a free server have unspecified queue time, hardware, and retention. Do not put proprietary fixtures on a shared host. Do not use this article’s commands as a performance comparison between products. No timings are claimed, and none should be inferred from a green PASS line.

Who should not use this approach

Teams with a dedicated ABI review bot and a full -fsanitize matrix in CI already have a stronger tape. Kernel modules, plugins loaded by a stable C API, and libraries that version symbols with .symver need a real ABI checker such as libabigail, not a sorted nm dump. Organizations that cannot open the eval corpus should not send stdin fixtures to any hosted coding server, free or not.

Skip the method when the artifact is not a CLI or a shared library. A notebook, a header-only template soup, or a CMake-only repo will produce empty nm output and a false sense of safety. Skip it for one-off prototypes where the binary is allowed to churn and the only consumer is the author.

The witness harness exists because cheap C++ generation turned “the tests are green” into a weak sentence. Green tests and a drifted symbol table can coexist. Record the symbol table. Fail closed when it moves. Readers who already keep C++ fixtures on disk can run that same loop against a free-tier coding server instead of provisioning a GPU host for the first filter.

MonkeyCode provides free models that can run this workflow.