Sanitizers Don't Flatter Anyone: Scoring Free Coding Models on C++ Undefined Behavior Fixes

In a previous post I built a small harness that lets the compiler argue with demo-quality C++ code produced by coding models. That harness had a blind spot: plenty of model output compiles cleanly and is still wrong. Code that passes -Wall -Werror -pedantic can still read out of bounds, use-after-free, or rely on signed overflow the moment the optimizer gets aggressive.

This follow-up closes that gap. Instead of asking "does it compile?", the question becomes "does it survive AddressSanitizer and UndefinedBehaviorSanitizer?" — a much less forgiving judge. And because model access and CI compute both cost money, I'll also walk through how I structure the loop so the whole evaluation runs on free tiers, including MonkeyCode's free model access and its free server option, without making the results depend on any specific paid plan.

Why sanitizers are a better grader than a compiler

A compiler checks that code is well-formed. Sanitizers check that a specific execution is well-behaved. For evaluating model-generated fixes, that difference matters:

• Compiler says yes, runtime says no. The classic model failure: it "fixes" a bounds bug by adjusting a loop condition that looks right but is off by one on the empty-input path. GCC/Clang accept it; ASan kills it on the first test case.
• UB is where models bluff hardest. Signed overflow, strict aliasing violations, shifting by the bit width — models often reproduce the shape of a fix (add a cast, reorder statements) without removing the UB. UBSan with -fno-sanitize-recover=all turns each bluff into a hard failure with a line number.
• It's deterministic enough to score. Unlike "does the output look correct," a sanitizer exit code is a binary signal you can aggregate across a corpus.

The limitation cuts both ways: sanitizers only judge the executions you feed them. A fix that passes your tests can still be wrong on untested inputs. I'll come back to that.

The artifact: a UB-repair benchmark harness

The corpus is a set of small, self-contained C++ files, each with exactly one known UB bug, plus a test driver. The task given to the model is always the same: fix the undefined behavior without changing intended behavior on valid inputs.

Here's the core scoring script:

#!/usr/bin/env bash
# score_fix.sh — compile a candidate fix under sanitizers and run the corpus tests.
# Usage: ./score_fix.sh candidate.cpp
cand="$1"
CXX="${CXX:-clang++}"
FLAGS="-std=c++20 -g -O1 -fsanitize=address,undefined -fno-sanitize-recover=all -Werror"

# 1. Must compile clean under sanitizers
if ! "$CXX" $FLAGS "$cand" -o /tmp/cand_bin 2>/tmp/build.log; then
  echo "FAIL:build"; exit 0
fi

# 2. Must survive every test input
for t in tests/*.txt; do
  if ! /tmp/cand_bin < "$t" > /tmp/out.txt 2>/tmp/asan.log; then
    if grep -qE 'AddressSanitizer|runtime error' /tmp/asan.log; then
      echo "FAIL:sanitizer:$(basename $t)"; exit 0
    fi
    echo "FAIL:crash:$(basename $t)"; exit 0
  fi
  if ! diff -q /tmp/out.txt "${t%.txt}.expected" >/dev/null; then
    echo "FAIL:wrong-answer:$(basename $t)"; exit 0
  fi
done
echo "PASS"

Three scoring decisions worth stealing:

1. -O1, not -O0. Some UB (especially signed-overflow-driven optimizations) only manifests when the optimizer assumes it can't happen. -O1 catches more real fixes and more fake ones.
2. -fno-sanitize-recover=all. First UB aborts. You want the exit code, not a log full of recovered warnings that a pass/fail regex has to interpret.
3. Sanitizer failure is scored separately from wrong answer. A model that removes the UB but breaks valid-input behavior fails differently than one that papers over the bug. When you aggregate results, keep these categories apart — they measure different weaknesses.

A minimal corpus entry looks like this:

// bugs/overflow_midpoint.cpp — classic signed-overflow midpoint
#include <vector>
#include <iostream>
int midpoint(int lo, int hi) { return (lo + hi) / 2; }  // UB when lo+hi overflows
int main() {
  int a, b; std::cin >> a >> b;
  std::cout << midpoint(a, b) << "\n";
}
// tests include INT_MAX-1, INT_MAX — the "obvious" fix (lo + (hi-lo)/2) is what we're probing for

With 20–30 such entries you get a meaningful per-model scoreboard in an afternoon.

Running the loop on free resources

The pipeline has two cost centers: model calls (generate the fix) and compute (compile + run under sanitizers). My current setup:

• Model side: MonkeyCode offers free access to coding models, which is what I route the "fix this UB" prompts through. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I deliberately don't hard-code a model name into the harness — the script takes the model's response over stdin, so whichever model is available on the free tier at run time slots in. That also keeps the benchmark honest: scores belong to a dated run, not to a brand.
• Compute side: the compile-and-sanitize step runs on MonkeyCode's free server option for the interactive iteration loop, and locally for anything I want to archive. Sanitizer builds of tiny single-file programs are cheap; the bottleneck is model latency, not CPU.

The loop itself is boring on purpose:

for bug in bugs/*.cpp; do
  prompt=$(cat prompts/fix_ub.txt "$bug")
  call_model "$prompt" > /tmp/candidate.cpp      # your client here
  echo "$(basename $bug) $(./score_fix.sh /tmp/candidate.cpp)"
done | tee results.txt

No retries, no "please try again" loops. One shot per bug per model. Retry-until-pass measures the model's luck and your patience, not its UB understanding.

When this approach is worth it — and when it isn't

Situation	Verdict
Comparing free-tier models before committing to a paid plan	Good fit — sanitizer scores are cheap and hard to game
Deciding whether a model's C++ output is safe to merge	Necessary but not sufficient — pair with code review
Evaluating models on large, multi-file codebases	Poor fit — single-file sanitizer harnesses don't scale to that
Proving absence of UB	Impossible — sanitizers check executions, not all executions
Rust/Python/JS model eval	Needs a different oracle (miri, sanitizers don't apply directly)

And who should skip this entirely: if you don't have a curated corpus with known-correct expected outputs, building one will take longer than the evaluation itself. Also skip it if your real question is "which model writes the most idiomatic C++" — that's a style judgment sanitizers can't make, and forcing it into a pass/fail metric will mislead you.

Limitations I'm not hiding

• Test coverage is the ceiling. A model fix that passes 12 test inputs has survived 12 executions, period. Mutation testing (flip operators in the fixed code, re-run) partially addresses this but doubles runtime.
• Free tiers change. Model availability, rate limits, and server capacity on any free offering can shift without notice. The harness treats model identity as a run parameter precisely so results stay comparable when the backend changes.
• Small corpus, small claims. With ~25 bugs I can say "model A sanitizer-failed on 30% of UB fixes, model B on 10%" for this corpus. I cannot say B understands UB better in general.

The useful habit underneath all of this: pick a judge that can't be sweet-talked. Compilers, sanitizers, fuzzers — anything that returns an exit code instead of a vibe. If you're comparing coding models on a free tier and want a starting point, MonkeyCode's free model access plus the scoring script above is enough to generate your own scoreboard this weekend; I'd genuinely like to hear what failure categories other people see, because the wrong-answer-vs-sanitizer split has been the most informative signal in my runs so far.