Asserts at -O0, UB at -O2: A Flag-Matrix Grader for AI C++ Patches

A maintainer once accepted an AI-written patch to a small C++ ring buffer. The eval job compiled the translation unit with the toolchain default, which in that repo meant -O0 -g. Three golden cases passed. The pull request looked boring in the best way.

The release build used -O2. Under wraparound the buffer dropped the newest element and left the oldest one intact. Nothing in the goldens had failed. The compiler had been allowed to assume that a signed index never overflowed, and the optimized code deleted the path the tests thought they were covering.

That pattern now shows up in AI C++ evals with uncomfortable regularity. Models play the harness they can see. A harness that only ever builds at -O0 grades a different program than the one that ships.

Why a single debug build is not a spec

Golden asserts answer one question: did this binary, built this way, produce these outputs. They do not answer whether the source still has a defined meaning under another flag set. Undefined behavior is the gap between those two measurements. At -O0 many UB bugs still appear to work. At -O2 the same bugs become deleted branches, hoisted loads, or loops that never terminate.

Sanitizers close part of that gap. They are not a second test suite. They are a different observer on the same goldens. An eval that records only the exit status of a debug binary is grading the observer, not the patch.

The rest of this article proposes a local flag-matrix grader. It rebuilds each candidate patch under several compiler envelopes, reruns the same goldens, and fails the patch if any envelope disagrees. The method is small enough to run on a laptop. It does not replace code review.

What the matrix actually measures

The grader treats a patch as a tuple of outcomes, not a boolean. Each cell is one compiler envelope:

1. -O0 -g — the historical eval default.
2. -O2 -DNDEBUG — a typical release envelope.
3. -O0 -fsanitize=address,undefined — ASan plus UBSan on the goldens.
4. -O2 -fsanitize=undefined — UBSan after the optimizer has rewritten the IR.

A pass means every golden still exits 0, stdout matches the pinned fixture, and the sanitizer log is empty. A fail in any cell fails the patch. Disagreement across cells is itself a finding. A patch that is green at -O0 and red at -O2 is not flaky. It is under-specified.

This is adjacent to coverage gates and resource envelopes, but it is not the same measurement. Coverage asks whether the hunk ran. rusage asks how fat the process got. The flag matrix asks whether the program still has a defined meaning when the compiler is allowed to believe the language.

A subject small enough to grade

The example below is a proposed local harness, not a measured benchmark. It encodes a realistic AI mistake: mixing a signed counter with a size computation that can overflow.

// subject/ring.hpp
#pragma once
#include <cstddef>
#include <cstdint>
#include <vector>

struct Ring {
  std::vector<std::uint32_t> buf;
  int head = 0;   // signed on purpose
  int count = 0;

  explicit Ring(int cap) : buf(static_cast<std::size_t>(cap), 0) {}

  void push(std::uint32_t v) {
    int cap = static_cast<int>(buf.size());
    int idx = head + count;
    if (idx >= cap) idx -= cap;   // breaks if head+count overflows
    buf[static_cast<std::size_t>(idx)] = v;
    if (count < cap) ++count;
    else head = (head + 1) % cap;
  }

  std::uint32_t front() const {
    return buf[static_cast<std::size_t>(head)];
  }
};

A model asked to make push faster will sometimes replace the wrap with a bitmask, or widen only half the arithmetic. Goldens that stay near small cap values never tickle signed overflow. The -O2 cell and the UBSan cell do.

// eval/golden_ring.cpp
#include "ring.hpp"
#include <cstdio>

int main() {
  Ring r(8);
  for (int i = 0; i < 20; ++i) r.push(static_cast<std::uint32_t>(i));
  if (r.front() != 12u) {
    std::fprintf(stderr, "front=%u\n", r.front());
    return 1;
  }
  return 0;
}

That golden is honest and still weak. Weak goldens are the point. The matrix exists to make their weakness visible instead of hiding it behind one debug binary.

Pin the envelopes in a manifest

Compiler flags drift the same way prompts drift. A one-line CXXFLAGS in a README is not a pin. Put the matrix in a file the grader hashes and commit that file next to the goldens.

{
  "cxx": "c++",
  "std": "c++17",
  "goldens": ["eval/golden_ring.cpp"],
  "include": ["subject"],
  "envelopes": [
    {"name": "dbg",    "cxxflags": ["-O0", "-g", "-Wall", "-Wextra", "-Werror"]},
    {"name": "rel",    "cxxflags": ["-O2", "-DNDEBUG", "-Wall", "-Wextra", "-Werror"]},
    {"name": "asan",   "cxxflags": ["-O0", "-g", "-fsanitize=address,undefined", "-fno-omit-frame-pointer"]},
    {"name": "ubsan2", "cxxflags": ["-O2", "-fsanitize=undefined", "-fno-sanitize-recover=undefined"]}
  ]
}

-Werror belongs in the debug and release cells. AI patches often introduce unused helpers and silent fallthrough that debug builds swallow if warnings are only advisory. The sanitizer cells omit -Werror only when a given toolchain warns about sanitizer-incompatible flags. Record that exception in the manifest. Do not hide it in a wrapper script.

Grader: rebuild, rerun, compare

The script below is a proposed driver. It compiles each golden under each envelope, runs the binary, and writes a JSONL row per cell. It does not scrape vendor dashboards. The grade happens on the machine that produced the binary.

#!/usr/bin/env python3
"""Proposed flag-matrix grader for AI C++ patches. Unexecuted example."""
from __future__ import annotations

import hashlib, json, os, subprocess, sys, tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
MANIFEST = json.loads((ROOT / "eval" / "matrix.json").read_text())

def run(cmd, extra_env=None, timeout=20):
    env = os.environ.copy()
    if extra_env:
        env.update(extra_env)
    return subprocess.run(
        cmd, capture_output=True, text=True, timeout=timeout, env=env
    )

def compile_one(src: Path, out: Path, flags: list[str]) -> subprocess.CompletedProcess:
    cmd = [
        MANIFEST["cxx"], f"-std={MANIFEST['std']}",
        "-I", str(ROOT / MANIFEST["include"][0]),
        *flags, str(src), "-o", str(out),
    ]
    return run(cmd)

def main() -> int:
    patch_id = sys.argv[1] if len(sys.argv) > 1 else "local"
    rows = []
    fail = 0
    with tempfile.TemporaryDirectory() as td:
        td = Path(td)
        for golden in MANIFEST["goldens"]:
            src = ROOT / golden
            for envp in MANIFEST["envelopes"]:
                out = td / f"{Path(golden).stem}_{envp['name']}"
                c = compile_one(src, out, envp["cxxflags"])
                cell = {
                    "patch": patch_id,
                    "golden": golden,
                    "envelope": envp["name"],
                    "compile_rc": c.returncode,
                    "compile_err": c.stderr[-2000:],
                }
                if c.returncode != 0:
                    cell["status"] = "compile_fail"
                    fail += 1
                    rows.append(cell)
                    continue
                extra = {}
                joined = " ".join(envp["cxxflags"])
                if "sanitize" in joined:
                    extra["UBSAN_OPTIONS"] = "print_stacktrace=1:halt_on_error=1"
                    extra["ASAN_OPTIONS"] = "detect_leaks=1:halt_on_error=1"
                r = run([str(out)], extra_env=extra)
                cell["run_rc"] = r.returncode
                cell["stderr"] = r.stderr[-2000:]
                cell["stdout_sha"] = hashlib.sha256(r.stdout.encode()).hexdigest()[:16]
                cell["status"] = "pass" if r.returncode == 0 and not r.stderr else "run_fail"
                if cell["status"] != "pass":
                    fail += 1
                rows.append(cell)
    outp = ROOT / "eval" / "last_matrix.jsonl"
    outp.write_text("".join(json.dumps(x) + "\n" for x in rows))
    print(f"wrote {outp} fail_cells={fail}")
    return 0 if fail == 0 else 2

if __name__ == "__main__":
    raise SystemExit(main())

Wire it with a short Make target so a human can reproduce one cell without Python.

CXX ?= c++
STD ?= c++17
GOLDEN ?= eval/golden_ring.cpp

dbg:
    $(CXX) -std=$(STD) -O0 -g -Wall -Wextra -Werror -I subject $(GOLDEN) -o /tmp/ring_dbg
    /tmp/ring_dbg

rel:
    $(CXX) -std=$(STD) -O2 -DNDEBUG -Wall -Wextra -Werror -I subject $(GOLDEN) -o /tmp/ring_rel
    /tmp/ring_rel

asan:
    $(CXX) -std=$(STD) -O0 -g -fsanitize=address,undefined -I subject $(GOLDEN) -o /tmp/ring_asan
    UBSAN_OPTIONS=halt_on_error=1 ASAN_OPTIONS=halt_on_error=1 /tmp/ring_asan

Run make dbg rel asan on the unpatched tree first. That is the noise floor for this measurement. If release already disagrees with debug on main, the matrix is grading the subject, not the model. Fix the subject before scoring patches.

How silent regressions show up

Read the JSONL as a table, not as a score. Four patterns matter.

1. All four cells pass. The goldens still may be weak. The matrix did not prove correctness. It proved agreement under these observers.
2. Debug passes, release fails. Treat this as UB or as a NDEBUG-dependent assert until proven otherwise. Do not rerun until green. Capture the stderr and the flag set.
3. Debug and release pass, ASan or UBSan fail. The goldens executed a bug the optimizer did not yet exploit. Keep the patch rejected.
4. Compile fails only under -Werror. That is a spec violation if the manifest pinned warnings as errors. Models pad patches with unused helpers. The compiler is the grader.

A useful extra column is stdout_sha. AI patches sometimes "fix" a golden by printing a different but still plausible line. Exit code 0 is not an I/O contract. Hash the bytes.

Disagreement is the signal worth keeping. A dashboard that averages the four cells into one pass-rate will hide the only interesting row. Store the tuple. Fail closed if any required cell is missing, not only if a cell is red.

Where generation stops and grading starts

Candidate patches have to come from somewhere. A local eval loop can call an open-source coding assistant, save the diff, and then refuse to trust that diff until the matrix is green.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode fits only as a generation step. The project is open source, with operator-supplied free-tier model access and a free server option that can emit candidate patches. The grade still belongs on a machine that has a C++ compiler, libasan, and the golden sources. Convenience is how -O0 defaults sneak back into remote notebooks. Keep matrix.json in the repository that owns the C++.

The workflow does not depend on that product. Any model that can emit a unified diff can sit in front of the same grader. A free server that writes diffs is useful. A compiler envelope that can reject them is the actual eval.

Limitations

Sanitizers do not see every UB. Data races need TSan and a concurrent golden. Strict-aliasing bugs sometimes need -fstrict-aliasing plus a release cell, not UBSan alone. -O2 is not -O3, and it is not LTO. A matrix with four cells is a sample of the compiler, not a proof.

The example golden is tiny. Real suites need the same envelopes applied to every binary the patch can touch, including tests the model did not edit. Link-time mismatches (-fsanitize on one translation unit, not on another) produce false stories. The grader above compiles one file. A CMake project needs the envelopes as preset cache files, not as ad-hoc c++ lines.

Wall time grows linearly with cells. Four envelopes times thirty goldens is a different job than one debug binary. Cache object files by (patch_sha, envelope_name) or the loop will train people to skip it. Skipping it returns the harness to the original lie: debug green equals shipable C++.

Who should not use this

Teams without a local C++ toolchain should not pretend a web playground is a release envelope. Firmware trees that cannot link sanitizer runtimes should keep the -O0 / -O2 disagreement cells and drop ASan, rather than skip the matrix. Safety-certified codebases need their own qualified compilers; this grader is not a qualification artifact.

Maintainers hunting model leaderboard points will hate the extra red cells. That is the point. Do not use the matrix as a substitute for a negative-compilation spec, a coverage gate on the hunk, or a resource envelope. Those catch other lies. This one catches the lie that debug green equals defined C++.

A working order of operations

1. Record four envelopes in eval/matrix.json. Hash the file in the eval lock.
2. Run the matrix on main until every cell is quiet. That is the baseline, not a score.
3. Generate a candidate patch from whatever model is in use.
4. Apply the patch on a worktree. Refuse to merge if any cell changes from pass to fail.
5. If debug and release disagree, stop and read sanitizer traces before asking the model to try again.
6. Promote a new golden only when it fails at least one envelope on a known-bad patch and passes all envelopes on main.

The order is boring on purpose. Eval harnesses fail when the interesting part is the model and the compiler is a default. Green asserts at -O0 are a beginning. They are not a ship signal. The flag matrix is a cheap way to ask the compiler the question the goldens forgot.

MonkeyCode provides free models that can run this workflow.