Debug Totals Lie at Release: An Opt-Split Gate for C++ Patch Evals

The failure mode is ordinary, and it hides in a green log. A grader accepts a C++ patch because the debug binary prints 18432. A later release build of the same sources prints 18416. Nobody edited the prompt between those two runs. The optimizer met a signed addition that had already left the representable range.

That split is a silent regression. A single golden print, captured under one flag set, cannot see it.

This note describes an opt-split gate for AI C++ patches. One frozen input tape, one hand-written oracle, and the candidate built at -O0 and -O2. The gate fails when the oracle stays stable and the candidate does not. A prompt hash pin stops a later draft from changing the test without a recorded edit.

What a single print misses

Generated C++ often looks finished inside a driver that prints one number. The driver and the patch share a translation unit. Warnings are quiet, or they were never enabled. The build uses whatever optimization the local default happens to be, which is frequently closer to -O0 than to the flags a package will ship.

Signed overflow, a read of an uninitialized scalar, and a dependency on evaluation order can all survive that setup. A second flag set sometimes keeps the arithmetic inside the range the oracle demands. Sometimes it stops hiding the bug. The opt-split gate does not prove the absence of undefined behavior. It catches the cheaper mismatch: debug output and release output disagree, while a checked oracle prints the same bytes twice.

Files the gate owns

Four inputs define a run, and none of them is a dashboard screenshot. prompt.txt is the only prose the draft step may read. prompt.pin stores the SHA-256 of that file from the last accepted wording change. tape.txt lists signed integers, one token per line, and ends with END.

oracle.cpp is written by a person. candidate.cpp is the patch under test, and it must export the same signature the oracle exports. The contract stays narrow so the grade is about values, not about prose.

// contract.hpp
#pragma once
#include <cstdint>
#include <optional>

std::optional<std::int64_t> fold_checked(const std::int64_t* data, std::size_t n);

A returned integer is the exact sum. std::nullopt means the fold is rejected. The candidate is scored against the oracle decision, not against a golden string sitting in the prompt.

Step 1. Pin the prompt before a draft exists

A changed prompt is a changed test, even when every tape byte is identical. The pin check refuses to grade on drift. It does not try to be clever about whitespace.

python3 - << 'PY'
import hashlib, pathlib, sys
prompt = pathlib.Path("prompt.txt").read_bytes()
digest = hashlib.sha256(prompt).hexdigest()
pin = pathlib.Path("prompt.pin")
if not pin.exists():
    pin.write_text(digest + "\n", encoding="ascii")
    print("pinned", digest)
    sys.exit(0)
expected = pin.read_text(encoding="ascii").strip()
if digest != expected:
    print("prompt pin mismatch")
    print("expected", expected)
    print("actual  ", digest)
    sys.exit(2)
print("prompt pin ok", digest)
PY

Treat a mismatch as a harness failure, not as a patch failure. Update prompt.pin only in a commit that quotes the new wording. Nightly jobs must not rewrite the pin to make themselves green.

Step 2. Freeze the tape outside the driver

The driver reads tape.txt, calls fold_checked, and prints REJECT or a decimal integer. It contains no expected answers. A model that can see the golden line inside the driver will often echo it, and the echo will match at every optimization level.

// tape_driver.cpp
#include "contract.hpp"
#include <fstream>
#include <iostream>
#include <string>
#include <vector>

int main() {
    std::ifstream in("tape.txt");
    std::vector<std::int64_t> data;
    std::string tok;
    while (in >> tok) {
        if (tok == "END") {
            break;
        }
        data.push_back(std::stoll(tok));
    }
    std::optional<std::int64_t> got = fold_checked(data.data(), data.size());
    if (!got) {
        std::cout << "REJECT\n";
        return 0;
    }
    std::cout << *got << "\n";
    return 0;
}

Start with three tapes, and keep them in separate files. One sum fits in 64 bits. One sum crosses the top of the signed 64-bit range. One file contains only END, the empty range.

The samples below are unexecuted fixtures until a local compiler has produced the four binaries. The method does not depend on a hosted runner.

10
20
30
END

9223372036854775807
1
END

Step 3. Build both sides at two levels

Hold the standard, the warning set, and the include path fixed. Change only the optimization flag. Record the compiler identity before the first object file exists, so a later green row can be tied to a toolchain.

mkdir -p out
g++ --version | head -n 1 | tee out/compiler.txt
CXXFLAGS="-std=c++20 -Wall -Wextra -Werror -I."
g++ $CXXFLAGS -O0 -o oracle_o0 oracle.cpp tape_driver.cpp
g++ $CXXFLAGS -O2 -o oracle_o2 oracle.cpp tape_driver.cpp
g++ $CXXFLAGS -O0 -o cand_o0 candidate.cpp tape_driver.cpp
g++ $CXXFLAGS -O2 -o cand_o2 candidate.cpp tape_driver.cpp

A candidate that fails to compile receives a compile grade, and the matrix stops. Partial binaries are not compared. An oracle that fails to compile means the gate is misconfigured. That result is not a patch defect.

The oracle itself should be boring. This implementation is a proposal for the fixture, not a claim that a model emitted it, and not a measured result.

#include "contract.hpp"
#include <cstdint>
#include <limits>
#include <optional>

std::optional<std::int64_t> fold_checked(const std::int64_t* data, std::size_t n) {
    __int128 acc = 0;
    for (std::size_t i = 0; i < n; ++i) {
        acc += data[i];
        if (acc > std::numeric_limits<std::int64_t>::max()) {
            return std::nullopt;
        }
        if (acc < std::numeric_limits<std::int64_t>::min()) {
            return std::nullopt;
        }
    }
    return static_cast<std::int64_t>(acc);
}

Run each binary from a directory that contains only the chosen tape.txt. Capture stdout bytes. Do not trim, and do not pass the text through a shell that rewrites newlines.

./oracle_o0 > out/oracle_o0.txt
./oracle_o2 > out/oracle_o2.txt
./cand_o0 > out/cand_o0.txt
./cand_o2 > out/cand_o2.txt
cmp -s out/oracle_o0.txt out/oracle_o2.txt

Step 4. Score a four-cell matrix

Compare the files with cmp. A missing newline is a mismatch. A numeric parser that forgives spaces will hide the drift this gate exists to catch.

Oracle -O0 vs -O2	Candidate vs oracle at -O0	Candidate vs oracle at -O2	Grade
differ	any	any	oracle unstable; do not score the patch
match	differ	any	functional miss
match	match	differ	opt-split fail
match	match	match	pass for this tape

The third row is the silent regression. The patch met the contract in the debug build and broke it, or changed it, at -O2. Store the four stdout files beside the prompt hash. A later job that flips only cand_o2 is a regression even when the debug line still matches the earlier example.

Repeat the matrix for every tape. The patch passes the gate only when every tape passes. One opt-split fail fails the run. Do not average rows into a soft score. A mean will bury a release-only break under three comfortable debug matches.

Step 5. Attach a draft host without baking in its limits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The matrix does not need a hosted model. A local editor can drop candidate.cpp into the tree, and the four g++ lines still apply. MonkeyCode enters only when a team wants a draft step or a remote build host. It is an open-source project. Operator materials describe free model access and a free server option. Both are availability claims, so confirm them in the current project docs on the day of the run.

Keep the draft blind. It may read prompt.txt and contract.hpp. It may not read oracle.cpp, the tapes, or earlier stdout. Showing the oracle to the model turns the grade into a transcription exercise, and transcription will pass at both optimization levels for the wrong reason.

Do not hard-code a token allowance, a machine size, or a rental duration into prompt.pin or the grader. Those numbers move, and a stale ceiling will outlive the plan that published it. If the account cannot start a build that day, fail the job closed and leave the previous pin untouched.

The free server, when it is actually available, is a fit for the four compile-and-run lines. The pass/fail rule should stay in a script the reviewer can diff on a laptop. A clean split is local pin check, remote compile, local grade. The host returns stdout files and the compiler version line, and the reviewer's copy of the matrix script assigns the grade.

Teams that already keep prompt.pin can send the frozen prompt to the current free model access and run this opt-split matrix on the free server, after reading that day's limits in the project docs. The grade script should still run unchanged if that host is offline.

Limitations, and who should skip the gate

Agreement across -O0 and -O2 is a screen, not a proof. Undefined behavior can survive both flag sets, and it can disappear under both. Sanitizer builds belong in a separate gate. This matrix also ignores throughput. A patch may pass every tape and still be too expensive to ship, so grade cost with a different fixture.

Skip the approach when the specification allows divergence under fast-math flags, or when the contract is approximate and a tolerance is the real oracle. Skip it when the oracle and the candidate are supposed to reject different edges. That mismatch is a spec argument, not a regression.

Skip it for patches that cannot be built as one extra translation unit, such as a plugin ABI or a generated lexer. The sample driver uses std::stoll and will throw on a corrupt tape. That is a harness bug, and the run should fail before any patch grade is written.

The __int128 accumulator is a GCC and Clang extension, not portable C++. On a toolchain without it, replace the oracle with a checked add written in standard C++. Pin that choice next to the compiler version line. Otherwise two machines will disagree about the oracle and the matrix will blame the patch.

What the log must hold

Each run record needs the prompt hash, the compiler version string, the four flag lines, and the stdout hashes. A green debug line without those fields is how a single total survives review. The release build was never graded, so nobody had a file to open.

The opt-split gate makes that file mandatory. Same tape, two optimization levels, and a human oracle that does not travel inside the prompt. When the release total moves and the oracle does not, the patch does not ship.

Comments

[Arhan Canli]

The bug class in your example is the one where the compiler is allowed to delete the check. A candidate that guards with if (sum + x < sum) on signed int64 looks fine at -O0 and can lose the branch at -O2. That only shows up on a tape that actually reaches the guard, so with three tapes the gate is only as strong as the tape that crosses the top of the range; add one tape that overflows only on the last element and one that overflows by crossing the negative end (INT64_MIN plus -1), since a candidate can handle one direction and not the other.

Because the oracle already says "REJECT" for those tapes, a candidate that wraps identically at both levels still fails, which is good. What it would not tell you is why. Building the candidate once more with -fsanitize=undefined (or -ftrapv) on the same tapes costs one line and turns "outputs differ" into "signed overflow at line N", which is the thing the model needs fed back to fix it.

One thing I could not tell from the post: does the gate rebuild with the same compiler version each night, and do you store the compiler.txt line next to the verdict? An -O2 divergence can appear or vanish across a gcc minor version, so a verdict without it can't be replayed.