Release Builds Are a Different Spec: A Flag-Matrix Harness for AI C++ Patches

A maintainer merged an AI-written bounds check into a small C++ parser. Debug CI compiled with -O0, ran the unit tests, and reported success. The guard was a single assert. The production job used -O2 -DNDEBUG, so the guard disappeared, and a short buffer turned a size_t into an over-read.

That failure is not a missing test case. It is a missing compile profile. The model satisfied the one command the prompt displayed. Release flags were never part of the spec the grader enforced.

Public debates about AI coding skill still score patches on whether tests pass. Compiler flags are the part of the score those threads skip. A patch that is green under one profile and wrong under another is a silent regression, even when every assertion in the Debug binary fires.

Why a single compile lies

AI C++ patches optimize for the build line in the prompt. If that line is g++ -std=c++17 -O0 -g, the model never sees NDEBUG, inlining, tautological-compare folding, or signed-overflow assumptions. Tests attached to that command become a second prompt. They do not become a contract.

Three splits show up constantly in generated patches:

1. Validation lives only in assert, so -DNDEBUG deletes the check.
2. Signed overflow or strict aliasing is undefined, and -O2 exploits it.
3. The Debug run is sanitizer-clean only because the sanitizer was never linked.

A flag-matrix harness treats those profiles as first-class oracles. The same sources, the same tests, four compiles. Disagreement is a failed eval, not a flake.

The artifact: a compile-profile ledger

The harness below is deliberately small. It grades one subject function across Debug, Release, an NDEBUG-only build, and an ASan/UBSan build. A candidate patch must pass every profile that the project claims to ship. Label the snippets as a local recipe, not as production CI until the team pins compiler versions.

1. Pin a subject that can fail two ways

Keep the function tiny so the matrix stays cheap. The first subject uses assert as a bounds check. The second relies on signed arithmetic that Debug may tolerate.

// subject_assert.cpp — assert-only guard (fails under -DNDEBUG)
#include <cassert>
#include <cstddef>
#include <cstdint>

std::uint32_t prefix_sum(const std::uint32_t* p, std::size_t n, std::size_t i) {
    assert(p != nullptr && i < n);   // gone in Release
    std::uint32_t acc = 0;
    for (std::size_t k = 0; k <= i; ++k) acc += p[k];
    return acc;
}

// subject_overflow.cpp — signed overflow the optimizer may fold
int scale_index(int base, int stride, int i) {
    return base + stride * i;        // UB if the product overflows
}

A golden test for the first function must include an out-of-range index. Under Debug the process aborts. Under -DNDEBUG it may return a value or crash later. Both outcomes are eval signal. Do not rewrite the test to skip the bad index. That rewrite hides the profile split.

2. Declare profiles as data, not comments

Store the matrix next to the tests. Comments in a README drift. A file the runner reads does not.

# profiles.txt — one compile recipe per line: name|cxxflags
debug|-std=c++17 -O0 -g
release|-std=c++17 -O2 -DNDEBUG
ndebug_o0|-std=c++17 -O0 -DNDEBUG
san|-std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer

The third row is the one most AI graders omit. It keeps -O0 so the binary stays easy to debug, but it still strips asserts. If a patch only “works” because assert is alive, this row fails while debug passes. That pair is the ledger’s most useful disagreement.

3. Drive the matrix from a shell runner

The runner compiles each profile into its own binary, executes the same test driver, and writes a row per profile. Exit code, sanitizer noise, and a stable stdout digest all belong in the row. Do not fold them into a single boolean until the end.

#!/usr/bin/env bash
# run_matrix.sh — proposed local harness, not a hosted service
set -euo pipefail
CXX=${CXX:-g++}
SRC=${1:?usage: run_matrix.sh subject.cpp testdriver.cpp}
TEST=${2:?}
ROOT=$(mktemp -d)
trap 'rm -rf "$ROOT"' EXIT

pass=0
fail=0
while IFS='|' read -r name flags; do
  [[ -z "${name:-}" || "$name" == \#* ]] && continue
  out="$ROOT/$name"
  # shellcheck disable=SC2086
  if ! $CXX $flags -o "$out" "$SRC" "$TEST" 2>"$ROOT/$name.err"; then
    echo "$name COMPILE_FAIL"
    fail=$((fail+1))
    continue
  fi
  set +e
  "$out" >"$ROOT/$name.out" 2>"$ROOT/$name.san"
  rc=$?
  set -e
  digest=$(cksum "$ROOT/$name.out" | awk '{print $1}')
  san=$(wc -c <"$ROOT/$name.san")
  echo "$name rc=$rc digest=$digest san_bytes=$san"
  if [[ $rc -ne 0 || $san -ne 0 ]]; then
    fail=$((fail+1))
  else
    pass=$((pass+1))
  fi
done < profiles.txt

echo "matrix pass=$pass fail=$fail"
[[ $fail -eq 0 ]]

Run it the same way for the baseline tree and for the patched tree. The eval is the diff of those two ledgers, not the patched tree alone. A patch that turns a Release crash into a Debug-only assert is not an improvement. It moved the failure into a profile the default CI never runs.

4. Require unanimous pass, then diff digests

Unanimous exit-code zero is necessary and not sufficient. Digests must match across profiles that are supposed to be functionally identical. debug and ndebug_o0 should agree on stdout for every defined input. They should disagree on out-of-range inputs if and only if the project documents abort-vs-unchecked as intentional. Most library code should not.

# decision table (eval outcome)
# debug  release  ndebug_o0  san     verdict
# pass   pass     pass       pass    accept
# pass   fail     fail       pass    assert-only guard; reject
# pass   pass     pass       fail    sanitizer-only bug; reject
# pass   fail     pass       fail    optimizer/UB; reject
# fail   fail     fail       fail    tests too weak or patch broken; reject

Print the table from the two ledgers. Do not keep it in a slide. A grader that cannot emit the row cannot catch the split.

5. Add one negative golden that Debug hides

The matrix is useless if every test input is well-defined. Include at least one case the function must reject: null pointer, i == n, a product that overflows int. Under debug the process may trap. Under release the same input must not be treated as success. Record expected status per profile in a side file so the runner does not invent a policy.

# expect.txt — input_id|profile|want_rc
overflow_max|debug|1
overflow_max|release|1
overflow_max|ndebug_o0|1
overflow_max|san|1
oob_index|debug|1
oob_index|release|1
oob_index|ndebug_o0|1
oob_index|san|1

If the project truly wants Release to skip checks, say so in expect.txt. Silence is not a policy. AI patches will fill silence with assert.

Where free remote generation fits

Generating candidate patches and grading them are different jobs. The matrix above needs a local compiler, sanitizer runtimes, and the project’s own headers. It does not need a GPU. Candidate generation does.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which some teams use only to produce patch candidates when they do not want to host a model themselves. The flag-matrix harness still runs on the maintainer’s machine. Treat published token and hardware claims as things to verify on the project’s current docs before depending on them; this article does not assign quotas, model names, or uptime.

A practical split is: sample several patches remotely, drop each diff into a worktree, and require run_matrix.sh to print fail=0 plus matching digests. If the remote side is unavailable, the grader still works. If the grader is skipped, remote tokens only buy more green Debug builds.

Limitations

The matrix does not prove absence of undefined behavior. It raises the cost of the usual lies: assert-only safety, optimizer-visible overflow, and unsanitized heaps. It also multiplies compile time. Four profiles on a large translation unit is a real bill. Cache object files per profile or the harness becomes something people turn off.

Sanitizer rows need matching libraries and will not catch every data race or every uninitialized read. MSVC, libstdc++, and libc++ disagree on what is defined. Pin the compiler. Do not compare a GCC Debug digest to a Clang Release digest and call the mismatch a model failure.

Stdout digests fail for tests that print timestamps, pointer values, or iteration order of unordered containers. Stabilize output or hash a structured result file instead of raw stdout. Timing is not a digest. Do not turn this harness into a benchmark.

Who should not use this

Teams that ship a single -O0 binary and never enable NDEBUG gain little. Firmware trees that forbid RTTI, exceptions, and sanitizers need a reduced matrix, not a copy-paste of profiles.txt. If the subject is a header-only template soup that takes minutes to instantiate, run the matrix on a extracted TU that includes only the patched symbols.

This approach is also the wrong tool for style-only diffs, comment rewrites, and CMake cleanup. Those changes do not split across -O2. Spend the four compiles on functions that touch arithmetic, indexing, lifetime, or error paths.

What to keep in the eval, not the prompt

Prompts rot. Profiles in a file next to the tests do not, if the runner is the merge gate. Put the matrix in the same job that already compiles the AI patch. Keep the prompt free of flag essays. The model will still write assert. The ledger will still fail ndebug_o0. That is the point of the extra compile.

A Release build is a different specification from the Debug command the model saw. Grade both. Then treat disagreement as a rejected patch, not as an interesting footnote in a green CI log.