DEV Community

Finley Li
Finley Li

Posted on

A Tiered Verification Gate for Cheap Model-Generated C++ Patches

#testing #ai #cpp #devops

A freshly released coding model can generate several C++ patch candidates in the time it takes to write a code review. When generation is cheap, the review cost becomes the bottleneck. A patch that compiles and passes the visible unit test can still carry undefined behavior, a data race, or a resource leak that appears only under a sanitizer.

This article describes a reproducible four-tier gate for model-generated C++ patches. The gate treats model output as untrusted input and runs it through compile checks, AddressSanitizer/UndefinedBehaviorSanitizer, ThreadSanitizer, and a repeated stress run. MonkeyCode's free model access and free server option fit into two specific slots: producing patch candidates and running slower isolation jobs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pipeline is intentionally model-agnostic. Commands do not need model names because the gate only cares about the diff and the executable.

Why a tiered gate is useful

A single "does it pass the test?" question is too weak for C++ because:

  • Undefined behavior can survive normal compilation and unit tests.
  • Data races often appear only under ThreadSanitizer with a suitable interleaving.
  • Leaks and use-after-free often appear only under AddressSanitizer.
  • A cheap model can produce many plausible patches, so the reviewer needs a mechanical first pass.

The tiers below are ordered from cheapest to most expensive so that bad patches fail before consuming a remote runner slot.

The four tiers

Tier Question Primary signal Default environment
0 Does the patch compile cleanly? -Werror exit code local
1 Does it trip ASan/UBSan? sanitizer exit code local
2 Does it trip TSan? sanitizer exit code local or free server
3 Does it survive repeated runs? repeated exit codes free server

Tier 3 is where a disposable free server is most relevant. It can run loops that would be inconvenient on a laptop, such as repeated sanitizer runs, larger inputs, or a stress build that takes several minutes.

Workspace layout 

gate/
  config.env
  gate.sh
  run_remote.sh
  fixtures/
    use_after_free.cpp
    data_race.cpp
Enter fullscreen mode Exit fullscreen mode

config.env keeps toolchain and runner settings separate from logic.

# config.env
CXX=g++
STD=c++20
RUNNER=
REMOTE=0
Enter fullscreen mode Exit fullscreen mode

Tier 0: compiler as first filter

Tier 0 is a deliberately strict compile. Warnings are errors, and the output goes to a temporary object file so the executable is not trusted yet.

$CXX -std="$STD" -Wall -Wextra -Werror -c "$CANDIDATE" -o /tmp/gate_t0.o
Enter fullscreen mode Exit fullscreen mode

A patch that fails here should return to the model or the reviewer immediately. The generated code has not earned the right to run.

Tier 1: ASan and UBSan

The following command builds with AddressSanitizer and UndefinedBehaviorSanitizer and runs the result once.

$CXX -std="$STD" -fsanitize=address,undefined -g -O1 "$CANDIDATE" -o /tmp/gate_t1
/tmp/gate_t1 < /dev/null
Enter fullscreen mode Exit fullscreen mode

A minimal fixture that passes Tier 0 but fails Tier 1 is shown below.

#include <iostream>

int main() {
    int* p = new int(7);
    delete p;
    std::cout << *p << '\n'; // use-after-free
}
Enter fullscreen mode Exit fullscreen mode

ASan reports the use-after-free and exits nonzero, which is exactly the signal the gate needs.

Tier 2: TSan

Tier 2 builds with ThreadSanitizer and runs a small repeat loop. This is where a remote runner starts to make sense.

$CXX -std="$STD" -fsanitize=thread -g -O1 "$CANDIDATE" -o /tmp/gate_t2
for i in 1 2 3; do /tmp/gate_t2 < /dev/null; done
Enter fullscreen mode Exit fullscreen mode

A compact data race fixture is below.

#include <thread>
#include <vector>

int main() {
    int counter = 0;
    std::vector<std::thread> threads;
    for (int i = 0; i < 8; ++i) {
        threads.emplace_back([&] {
            for (int j = 0; j < 1000; ++j) ++counter;
        });
    }
    for (auto& t : threads) t.join();
    return counter;
}
Enter fullscreen mode Exit fullscreen mode

ThreadSanitizer reports the unsynchronized write to counter. The exact report may vary by toolchain, but the nonzero exit code is the gate's contract.

Tier 3: repeated stress on a disposable server

Tier 3 is a loop, not a proof. It exists to catch flakes and resource pressure that single runs miss. The example uses SSH because it is tool-agnostic; an operator can replace the transport with whatever the free server option exposes.

gate.sh:

#!/usr/bin/env bash
set -euo pipefail

CXX="${CXX:-g++}"
STD="${STD:-c++20}"
CANDIDATE="${1:?usage: gate.sh candidate.cpp}"

# Tier 0: compile with warnings as errors
$CXX -std="$STD" -Wall -Wextra -Werror -c "$CANDIDATE" -o /tmp/gate_t0.o

# Tier 1: ASan + UBSan
$CXX -std="$STD" -fsanitize=address,undefined -g -O1 "$CANDIDATE" -o /tmp/gate_t1
/tmp/gate_t1 < /dev/null

# Tier 2: TSan with a small repeat count
$CXX -std="$STD" -fsanitize=thread -g -O1 "$CANDIDATE" -o /tmp/gate_t2
for i in 1 2 3; do /tmp/gate_t2 < /dev/null; done

echo "gate passed"
Enter fullscreen mode Exit fullscreen mode

run_remote.sh:

#!/usr/bin/env bash
set -euo pipefail
source ./config.env

if [[ "${REMOTE}" == "1" ]]; then
  test -n "${RUNNER}" || { echo "RUNNER is required when REMOTE=1" >&2; exit 2; }
  scp "$1" "${RUNNER}:/tmp/gate/"
  ssh "${RUNNER}" "cd /tmp/gate && CXX=${CXX} STD=${STD} bash gate.sh $(basename "$1")"
else
  bash gate.sh "$1"
fi
Enter fullscreen mode Exit fullscreen mode

Usage:

chmod +x gate.sh run_remote.sh
./run_remote.sh fixtures/use_after_free.cpp    # fails at Tier 1
REMOTE=1 RUNNER=runner-host ./run_remote.sh fixtures/data_race.cpp
Enter fullscreen mode Exit fullscreen mode

The snippets are reference implementations, not an executed benchmark. Test them in a container before using them in CI.

Limitations

  • Sanitizers catch only what the input actually exercises. A green run is not a proof of correctness.
  • A free server may be queued, reset between jobs, or time-limited. Do not assume persistent artifacts.
  • The SSH example is only a transport sketch; the actual free server option may use a different interface.
  • The gate does not replace human code review. It reduces the number of plausible patches that reach the reviewer.

Who should not use this approach

Teams with the following constraints should keep the entire pipeline on-premises:

  • Patches touch private keys, credentials, or regulated data.
  • Compliance rules forbid third-party model or runner access.
  • The codebase depends on hardware-specific behavior that sanitizers cannot model.
  • The project requires a deterministic, auditable execution environment for every patch.

For those cases, the same tiered script is still useful, but both generation and execution should run on infrastructure the team controls.

The point of the gate is not to trust the model output. It is to make the first review step cheap and reproducible before a human spends time on a plausible but unsafe patch.

Where cheap model access and a disposable remote runner are already available, the same script can be pointed at them with a configuration change. The gate remains useful even if those options are replaced.

Top comments (0)