Must-Not-Compile Is a Spec: A Negative-Compilation Envelope for AI C++ Patches

A payments adapter review landed on a Tuesday afternoon. The model had produced a three-file diff. The unit suite returned green in twelve seconds, and the reviewer merged it.

A week later a plugin author constructed TokenHandle from a raw integer. The original header had blocked that path with an explicit private constructor and a deleted converting constructor. The patch had moved the constructor to public so a test the model added in the same diff would compile.

Runtime assertions never saw the leak. The type had become a wider type while every EXPECT_* still passed. Positive tests reward compilation. They do not record the compilation that must keep failing.

The hole in treating build success as progress

Most AI C++ patch evals stop at three bits. The compiler exit code is zero. The test binary exits zero. Sometimes stdout is hashed. That triad is a spec for “does something run.” It is not a spec for “the public surface did not grow.”

Language-level contracts live in the failures. Deleted special members, explicit constructors, private factories, requires clauses, and static_assert checks exist so that certain translation units never become programs. An edit that turns those failures into successes is a behavior change. Graders that treat every successful compile as progress will score that edit as a win.

The envelope below treats selected compile failures as first-class oracles. It is a proposed, reproducible harness. It is not a published leaderboard and it does not report model scores.

What the envelope records

A negative-compilation envelope is a directory of tiny translation units plus a grader. Each unit includes the public header under test and then performs one operation the API is supposed to reject.

The grader keeps three buckets:

1. Must-build units. These compile, link, and run. They are the ordinary happy path.
2. Must-fail units. The compiler must return a non-zero status. A diagnostic substring may be pinned.
3. Must-still-fail after the patch. If a must-fail unit starts compiling, the patch is a surface regression even when runtime tests stay green.

The third bucket is the point of the harness. AI patches frequently unstick a test by weakening a type.

Verdict table

must-build after patch	must-fail after patch	Verdict
compiles and tests pass	still fails to compile	accept
compiles and tests pass	now compiles	surface regression
no longer compiles	still fails	patch broke the build
runtime assertion fails	still fails	functional regression
compiles and tests pass	fails, but diagnostic text drifted	toolchain mismatch, not a model win

A minimal library under test

The example is a move-only handle. Copy is deleted. Integer construction is not part of the public surface. The snippets are labeled examples, not extracted production code.

// include/token_handle.hpp
#pragma once
#include <cstdint>
#include <utility>

class TokenHandle {
public:
  static TokenHandle from_trusted(std::uint64_t id) noexcept {
    return TokenHandle(id);
  }

  TokenHandle(const TokenHandle&) = delete;
  TokenHandle& operator=(const TokenHandle&) = delete;

  TokenHandle(TokenHandle&& other) noexcept : id_(other.id_) {
    other.id_ = 0;
  }

  TokenHandle& operator=(TokenHandle&& other) noexcept {
    if (this != &other) {
      id_ = other.id_;
      other.id_ = 0;
    }
    return *this;
  }

  std::uint64_t id() const noexcept { return id_; }

private:
  explicit TokenHandle(std::uint64_t id) noexcept : id_(id) {}
  std::uint64_t id_;
};

A positive unit that must keep compiling and running:

// cases/pos_move_ok.cpp
#include "token_handle.hpp"
#include <cassert>

int main() {
  auto a = TokenHandle::from_trusted(42);
  auto b = std::move(a);
  assert(b.id() == 42);
  return 0;
}

Two negative units that must keep failing. These files are not broken tests. They are the contract.

// cases/neg_copy.cpp
#include "token_handle.hpp"

int main() {
  auto a = TokenHandle::from_trusted(1);
  TokenHandle b = a; // copy must remain deleted
  (void)b;
}

// cases/neg_from_int.cpp
#include "token_handle.hpp"

int main() {
  TokenHandle h = 7; // converting construction must remain impossible
  (void)h;
}

Manifest and grader

Pin the compiler, the language mode, and the include path. A JSON manifest keeps the envelope reviewable without extra YAML dependencies.

{
  "compiler": ["g++", "-std=c++20", "-Wall", "-Wextra", "-Werror"],
  "include": "include",
  "must_build": ["cases/pos_move_ok.cpp"],
  "must_fail": [
    {"path": "cases/neg_copy.cpp", "diagnostic": "deleted"},
    {"path": "cases/neg_from_int.cpp", "diagnostic": ""}
  ]
}

Proposed grader in Python 3. It shells out to the real compiler. It does not parse C++.

#!/usr/bin/env python3
"""Negative-compilation envelope. Proposed example, not a published score."""
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
BIN = Path("/tmp/negcomp_a.out")


def compile_unit(compiler, include, src):
    cmd = compiler + [f"-I{include}", str(src), "-o", str(BIN)]
    return subprocess.run(cmd, capture_output=True, text=True)


def main():
    spec = json.loads((ROOT / "envelope.json").read_text())
    compiler = spec["compiler"]
    include = str(ROOT / spec["include"])
    failures = []

    for rel in spec["must_build"]:
        proc = compile_unit(compiler, include, ROOT / rel)
        if proc.returncode != 0:
            failures.append("must_build failed: " + rel + "\n" + proc.stderr)
            continue
        run = subprocess.run([str(BIN)], capture_output=True, text=True)
        if run.returncode != 0:
            failures.append("runtime failed: " + rel)

    for item in spec["must_fail"]:
        proc = compile_unit(compiler, include, ROOT / item["path"])
        if proc.returncode == 0:
            failures.append("SURFACE REGRESSION: " + item["path"] + " now compiles")
            continue
        needle = item.get("diagnostic") or ""
        blob = (proc.stderr or "") + (proc.stdout or "")
        if needle and needle not in blob:
            failures.append(
                "must_fail, diagnostic missed " + repr(needle) + ": " + item["path"]
            )

    if failures:
        print("\n".join(failures))
        return 1
    print("envelope: must-build green, must-fail still failing")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run the grader before and after a candidate patch:

python3 grade_envelope.py
git apply /tmp/model.patch
python3 grade_envelope.py; echo "grade_exit=$?"
git checkout -- include/token_handle.hpp

A model that publishes the constructor to satisfy neg_from_int.cpp will flip that unit from fail to success. The second run prints SURFACE REGRESSION. Exit code zero on the unit tests is then irrelevant.

Numbered workflow for an AI C++ patch eval

1. Freeze envelope.json in the same commit as the headers it describes. Do not let the model edit the must-fail list in the same turn as the production sources.
2. Generate the patch in an isolated worktree. Keep envelope files read-only to the agent, or reject diffs that touch cases/neg_*.cpp.
3. Apply the patch. Rebuild must-build units. Run them.
4. Recompile every must-fail unit. Require a non-zero compiler status. If a diagnostic substring is pinned, require it.
5. Accept the patch only when all three hold: must-build still builds, tests still pass, must-fail still fails.
6. Record g++ --version or clang++ --version next to the envelope. Diagnostic text is not portable across major versions.

Step 2 is the lock. Oracle suites fail when the model rewrites the oracle. Negative units fail the same way if they are writable.

Where generation and grading split

Generating the candidate patch and compiling the envelope are separate jobs. The first is a language-model call. The second is a compiler farm.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free-tier model access can sit on the generate side of that split, and its free server option can sit on the grade side so the must-fail matrix does not occupy a laptop that is already compiling the product. Those are availability options, not quotas, model names, or latency claims. Teams that already pin a CI image should keep grading inside that image. The envelope does not depend on a particular vendor.

Maintainers who already score AI C++ patches can drop envelope.json onto an idle runner and compare pre-patch and post-patch compiler statuses. That is the only product-shaped suggestion in this write-up.

Compiler notes that break naive graders

GCC and Clang do not emit the same words for a deleted constructor. Pinning diagnostic to "deleted" can fail on a compiler that says use of deleted function versus one that says attempting to reference a deleted function. Status-only checks are more portable and weaker.

CMake try_compile is an alternative to the Python loop. The inversion is easy to misread in review.

try_compile(NEG_COPY_BUILT
  ${CMAKE_BINARY_DIR}/neg_copy
  SOURCES ${CMAKE_SOURCE_DIR}/cases/neg_copy.cpp
  CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${CMAKE_SOURCE_DIR}/include"
)
if(NEG_COPY_BUILT)
  message(FATAL_ERROR "neg_copy compiled; public surface widened")
endif()

try_compile returns true on success. The envelope treats true as failure. The Python grader makes that inversion obvious in one if proc.returncode == 0.

Concepts and SFINAE change the picture again. A constraint failure may be a substitution failure inside a template, not a hard error at top-level main. Negative units should call the constrained function in a context that forces a hard error. Otherwise the unit compiles and the envelope lies.

Limitations

The envelope does not prove API stability. It proves that a finite list of snippets still fail. A model can widen the surface along an unlisted axis: a new converting operator, a friend declaration, or a defaulted copy after a refactor into a trivially copyable layout.

Diagnostic substring matching is brittle across compiler upgrades. Status-only matching is coarse. Both need a pinned toolchain.

Compile-fail tests do not catch runtime undefined behavior that still compiles. They do not replace sanitizers, ABI dumps, or human review.

Header changes that only affect inlining can still be correct while flipping a must-fail unit if the unit was over-specified. Treat false positives as envelope bugs until a human reads the diagnostic.

Who should not use this

Do not adopt the envelope for patches that never touch headers or type definitions. A leaf .cpp bugfix has no compile-fail contract to protect.

Do not treat it as a security control. A private constructor is not a sandbox.

Do not run a single-compiler manifest as the only gate on a codebase that ships three toolchains. Split envelopes per compiler, or check status only.

Skip it when the team cannot keep cases/neg_*.cpp out of the model’s writable set. A model that deletes the negative units will score perfectly.

Closing

Runtime green is a weak statement about C++. The language spends a large fraction of its design budget on programs that must not exist. An eval that never asks the compiler to refuse those programs will systematically prefer models that widen types until the tests compile.

Keep the must-fail list small, versioned, and unwritable to the agent. Grade the patch against that list with the same compiler the project ships. The merge that only looks at ctest will keep promoting public constructors that were never in the spec.

MonkeyCode provides free models that can run this workflow.