Exit Code Zero Is Not a Spec: A Digest Harness for AI C++ Patches

A payments team merged a model-written C++ patch on a Friday. Every parse_amount test returned zero. Monday's batch importer split records on the substring WARN, which the new code printed to stderr for every rounded cent.

The tests had never hashed stderr. The agent loop had no reason to. Coding agents now retry until a visible command succeeds. The loop is not malicious. It is incomplete.

This article proposes a digest harness that treats extra process channels as part of the spec. The method stays useful without any hosted model. It is a grader, not a demo.

Why a pass/fail bit is the wrong oracle

Unit tests encode the checks a human remembered to write. Agent retries encode whatever makes those checks green. Anything outside that set can drift: diagnostic streams, errno, elapsed-time class, sanitizer verdicts, even which paths the binary opens.

A digest does not replace tests. It notices when tests stayed green while the rest of the process changed. That gap shows up constantly once a model is allowed to edit, build, and rerun until CI is quiet.

The observable tuple

For each golden case the harness records a tuple, then hashes it.

1. Process exit code.
2. SHA-256 of stdout bytes.
3. SHA-256 of stderr bytes.
4. Sanitizer class: clean, asan, ubsan, timeout, or crash.
5. Wall-time bucket, not raw milliseconds.

Raw timings thrash across machines. Buckets such as lt_50ms, lt_500ms, lt_5s, and gt_5s keep the digest stable enough to compare. Treat those cutoffs as a proposal. Pick buckets that match the service SLA, not these defaults.

Store one JSON object per case. The file is the lock. A later candidate is accepted only when every field matches, not when ctest is green.

{
  "case": "parse_amount.rounding_up",
  "exit": 0,
  "stdout_sha256": "e3b0c44298fc1c149afbf4c8996fb924...",
  "stderr_sha256": "d4c3b2a1907e6f5d4c3b2a1907e6f5d4...",
  "sanitizer": "clean",
  "time_bucket": "lt_50ms"
}

A fixture the tests will not defend

The following C++ fixture is a worked example. It is not production billing code. The unit test checks the integer result and ignores stderr on purpose, which is the failure mode the digest is meant to catch.

// parse_amount.cpp — example fixture, not a library
#include <charconv>
#include <iostream>
#include <string_view>

// Returns cents. Writes a diagnostic on rounding.
int parse_amount(std::string_view s) {
    double v = 0.0;
    auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v);
    if (ec != std::errc{}) return -1;
    int cents = static_cast<int>(v * 100.0 + 0.5);
    if (v * 100.0 != static_cast<double>(cents)) {
        std::cerr << "WARN: rounding " << s << "\n";
    }
    return cents;
}

int main(int argc, char** argv) {
    if (argc < 2) return 2;
    int c = parse_amount(argv[1]);
    if (c < 0) return 1;
    std::cout << c << "\n";
    return 0;
}

A typical unit test only asserts parse_amount("1.005") == 101. An agent can keep that equality and still change the diagnostic, the rounding mode, or the crash behavior under -fsanitize=undefined. The digest is the extra oracle.

Build the grader in numbered steps

Pin the toolchain first. Unpinned compilers make digests lie.

1. Record a baseline binary under two configs

Compile once without sanitizers and once with undefined-behavior sanitizing. Keep both artifacts. Do not mix their digests.

cxx=${CXX:-g++}
std=${CXXSTD:-c++20}

"$cxx" -std="$std" -O2 -o parse_amount parse_amount.cpp
"$cxx" -std="$std" -O1 -fsanitize=undefined -fno-omit-frame-pointer \
    -o parse_amount.ubsan parse_amount.cpp

If the local compiler rejects -fsanitize=undefined, stop. A missing sanitizer is not a clean class. Record unavailable and refuse to compare against a clean baseline from another machine.

2. Drive cases from a plain text list

Golden inputs stay tiny and boring on purpose. The point is coverage of channels, not coverage of finance.

# cases.txt — one argv payload per line
1.00
1.005
0
-3.2
999999999999
not-a-number

3. Capture the tuple without trusting the shell

The script below is a proposal. Run it as a local tool. It does not claim to have been executed against a public corpus.

#!/usr/bin/env python3
"""digest_run.py — capture a behavior digest for one argv."""
import hashlib, json, os, subprocess, sys, time

BUCKETS = [(0.05, "lt_50ms"), (0.5, "lt_500ms"), (5.0, "lt_5s")]

def bucket(seconds: float) -> str:
    for limit, name in BUCKETS:
        if seconds < limit:
            return name
    return "gt_5s"

def digest_of(bin_path: str, arg: str, timeout: float) -> dict:
    env = os.environ.copy()
    env["UBSAN_OPTIONS"] = "print_stacktrace=1:halt_on_error=1"
    t0 = time.monotonic()
    try:
        p = subprocess.run(
            [bin_path, arg],
            capture_output=True,
            timeout=timeout,
            env=env,
        )
        elapsed = time.monotonic() - t0
        san = "clean"
        blob = (p.stderr or b"") + (p.stdout or b"")
        if b"runtime error:" in blob or b"UBSAN" in blob:
            san = "ubsan"
        if p.returncode < 0:
            san = "crash"
        return {
            "case": arg,
            "exit": p.returncode,
            "stdout_sha256": hashlib.sha256(p.stdout).hexdigest(),
            "stderr_sha256": hashlib.sha256(p.stderr).hexdigest(),
            "sanitizer": san,
            "time_bucket": bucket(elapsed),
        }
    except subprocess.TimeoutExpired:
        return {
            "case": arg,
            "exit": None,
            "stdout_sha256": None,
            "stderr_sha256": None,
            "sanitizer": "timeout",
            "time_bucket": "gt_5s",
        }

def main() -> int:
    bin_path, arg, timeout = sys.argv[1], sys.argv[2], float(sys.argv[3])
    json.dump(digest_of(bin_path, arg, timeout), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

4. Freeze the baseline, then grade candidates as a diff

Write one JSON file per case under gold/. After a model emits a patch, rebuild both binaries and emit cand/. Comparison is structural. A single mismatched field fails the patch even when the process exit code is still zero.

#!/usr/bin/env python3
"""digest_diff.py — fail on any field drift. Example grader, not a CI product."""
import json, pathlib, sys

FIELDS = ("exit", "stdout_sha256", "stderr_sha256", "sanitizer", "time_bucket")

def load(dirpath: str) -> dict:
    out = {}
    for p in sorted(pathlib.Path(dirpath).glob("*.json")):
        obj = json.loads(p.read_text())
        out[obj["case"]] = obj
    return out

def main() -> int:
    gold, cand = load(sys.argv[1]), load(sys.argv[2])
    failed = 0
    if gold.keys() != cand.keys():
        print("case set drifted", file=sys.stderr)
        return 2
    for case in gold:
        for field in FIELDS:
            if gold[case][field] != cand[case][field]:
                print(f"FAIL {case} {field}: {gold[case][field]!r} -> {cand[case][field]!r}")
                failed += 1
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

python3 digest_diff.py gold cand
# exit 0: every channel matched
# exit 1: at least one silent drift
# exit 2: the case set itself changed

A decision table for the mismatch

Use the table during review. Do not auto-merge on a story about flakiness.

Digest field	Typical agent shortcut	Review action
exit still 0, stderr_sha256 changes	New logging, leftover debug, or a WARN that becomes a parser	Require the stream to be part of the test or strip it
stdout_sha256 changes with tests still green	Formatting drift, extra newline, locale, or a different rounding path	Freeze expected bytes or reject the patch
sanitizer moves from clean to ubsan	Signed overflow that -O2 tests never hit	Reject; do not “quiet” UBSAN_OPTIONS
time_bucket jumps one class	Sleep-based race “fix”, extra copies, or a remote FS stall	Re-run three times; fail if the bucket stays shifted
sanitizer is timeout	Infinite retry, lock, or pathological input	Fail closed; do not raise the timeout to make it green

The interesting row is the first one. Tests stay green. The product still breaks. That is the whole point of hashing stderr.

Where a free model tier and a free server fit

Local laptops are a poor farm for sanitizer builds. They thermal-throttle. They also leak prompt history into shell logs. A hosted compile box with a pinned compiler is the boring fix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free-tier model access and a free server option. Those two availability claims are the only product facts used here. This article does not name models, quote a token ceiling, or publish hardware or timing numbers, because those details change and would go stale in a tutorial.

The workflow stays the same on any box that can run g++ and Python 3. Generate a candidate patch from the free-tier model access. Build both binaries on the free server so sanitizer flags match the baseline. Emit cand/ there. Diff against gold/ before a human looks at the diff hunks. If the digest fails, the patch is not “almost good.” It changed an observable the tests never named.

Limitations

Digests are brittle in the ways tests are not. Locale, libstdc++ debug iterators, and ASLR-backed printouts will churn hashes. Redirected fds that include timestamps will churn them too. Time buckets still false-positive on a noisy neighbor, which is why the table requires a rerun rather than a story.

The harness also cannot see bugs that never become process output. A wrong cache key that still prints the same cents will hash equal. So will a leak that no test frees. AddressSanitizer on a third binary can be added later. It is a different digest, not a field on this one.

Nondeterministic tests are out of scope. If the fixture reads /dev/urandom, the wall clock, or a live socket, freeze those seams first. Hashing noise is not a grader.

Who should not use this

Skip the digest harness when the project has no pinned compiler, when golden cases require a GPU, or when the binary is an interactive TUI. Skip it when the team already snapshots stdout, stderr, and sanitizer logs in CI. Skip it for patches that are supposed to change diagnostics; update gold/ in the same commit and say so in the log.

Do not use a free shared server for secrets, customer traces, or proprietary headers. Digests of public fixtures are the intended workload. Proprietary corpora belong on a machine the team owns.

Agent loops will keep optimizing the command they can see. Give them a command that sees more than exit 0. Then keep the human review for the cases the digest cannot name.