A new coding model lands, a clipped demo circulates, and within hours the comment sections have settled the question: it writes C++ now. I've stopped participating in that ritual, not because the models are bad — many are genuinely strong — but because the evidence offered is worthless. A happy-path demo tells you almost nothing about whether the code survives contact with undefined behavior tooling. So instead of arguing, I built a small, portable rig that turns any model release into a set of sanitizer verdicts I can reproduce and share. This post is that rig, the reasoning behind its design, and an honest note about where free infrastructure — including MonkeyCode's free model access and free server option — fits into running it.
Why compiler acceptance proves so little
C++ gives a model an enormous amount of rope. Code can type-check cleanly, warn about nothing, pass the three unit tests the demo author wrote, and still contain a dangling reference or an aliasing violation that only detonates under optimization. The compilers won't tell you — the language explicitly licenses them to assume undefined behavior never happens. That's what sanitizers are for: AddressSanitizer and UndefinedBehaviorSanitizer convert latent landmines into observable runtime failures. If we're going to have opinions about model-generated C++, they should be anchored there, not in whether a demo compiled on the first try.
My working rule: a model release gets no verdict from me until its output has been through a fixed battery of UB-flavored tasks, judged by tooling rather than vibes.
Designing the battery as a contract, not a vibe
The key decision was making the evaluation data-driven. Every task lives in a small manifest so the same battery can be pointed at any model, any time, without editing code:
# battery.yaml — one entry per task; the runner is model-agnostic
tasks:
- id: dangling_after_growth
family: lifetime
brief: >
A helper keeps a reference into a std::vector across a push_back
that may reallocate. Repair it; observable output must not change.
inputs: [vectors/small.txt, vectors/growth_edge.txt, vectors/large.txt]
- id: narrow_index_math
family: arithmetic
brief: >
Averaging two 32-bit indices overflows before the result widens.
Repair it; output format is unchanged.
inputs: [index/mid.txt, index/max_pair.txt, index/zero.txt]
- id: moved_but_still_read
family: value_semantics
brief: >
An accessor hands out a reference to a member that was moved from
two calls earlier. Repair it without removing the move.
inputs: [move/single.txt, move/repeated.txt, move/interleaved.txt]
- id: pun_through_char
family: aliasing
brief: >
A buffer is reinterpreted through an incompatible struct pointer
as a "fast path." Repair it; the fix must still work at -O2.
inputs: [alias/aligned.txt, alias/offset.txt]
- id: erase_while_walking
family: iterators
brief: >
A loop erases matching elements while iterating. Repair it;
ordering of remaining elements is part of the contract.
inputs: [erase/none.txt, erase/all.txt, erase/alternating.txt]
Two properties matter here. First, the prompt to the model is only the brief plus the buggy source — nothing hints that sanitizers will do the judging, because real code review doesn't come with that hint either. Second, every task demands a one-sentence root-cause statement alongside the fix. I'll come back to why that sentence is load-bearing.
The runner
The driver is a short Python script rather than a pile of shell, mostly so results come out as structured JSON I can diff across model releases:
#!/usr/bin/env python3
"""verdict.py — compile and execute one model submission per task.
Usage: python3 verdict.py battery.yaml submissions_dir/ results.json"""
import json, subprocess, sys, tempfile
from pathlib import Path
import yaml # pip install pyyaml
CONFIGS = [
("g++", "-O0", "-fsanitize=address,undefined -fno-omit-frame-pointer"),
("g++", "-O2", "-fsanitize=undefined"),
("clang++", "-O2", "-fsanitize=address,undefined"),
]
BASE = ["-std=c++20", "-Wall", "-Wextra", "-Werror"]
SAN_MARKERS = ("runtime error", "AddressSanitizer", "ERROR: leak")
def attempt(cc, opt, san, src, workdir):
binary = Path(workdir) / "a.out"
cmd = [cc, *BASE, opt, *san.split(), str(src), "-o", str(binary)]
compiled = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
if compiled.returncode != 0:
return "compile_fail"
return None # caller handles execution
def execute(binary, input_path):
try:
run = subprocess.run(
[binary], stdin=open(input_path), capture_output=True,
text=True, timeout=10,
)
except subprocess.TimeoutExpired:
return "timeout"
haystack = run.stderr or ""
if any(m in haystack for m in SAN_MARKERS):
return "sanitizer_hit"
return "clean" if run.returncode == 0 else "crash"
def main(manifest_path, sub_dir, out_path):
battery = yaml.safe_load(Path(manifest_path).read_text())
report = {}
for task in battery["tasks"]:
src = Path(sub_dir) / f"{task['id']}.cpp"
if not src.exists():
report[task["id"]] = {"status": "missing"}
continue
cells = []
for cc, opt, san in CONFIGS:
with tempfile.TemporaryDirectory() as wd:
fail = attempt(cc, opt, san, src, wd)
if fail:
cells.append({"cc": cc, "opt": opt, "result": fail})
continue
for inp in task["inputs"]:
cells.append({
"cc": cc, "opt": opt, "input": inp,
"result": execute(str(Path(wd) / "a.out"), inp),
})
clean = sum(c["result"] == "clean" for c in cells)
report[task["id"]] = {"clean": clean, "total": len(cells), "cells": cells}
Path(out_path).write_text(json.dumps(report, indent=2))
print(json.dumps({k: f"{v.get('clean', 0)}/{v.get('total', 0)}"
for k, v in report.items()}, indent=2))
if __name__ == "__main__":
main(*sys.argv[1:4])
Nothing here is clever, and that's deliberate. Anyone can rerun it against any model's output and get byte-identical verdicts. Evaluation you can't fork isn't evaluation; it's marketing.
The part people skip: grading the explanation
Each submission includes that one-sentence root-cause statement, and I score it by hand. This consistently turns out to be the most informative signal in the whole rig. Across the runs I've done, a meaningful slice of submissions pass every sanitizer cell while the accompanying explanation misidentifies the bug — the model produced a correct-looking repair by pattern-matching the shape of the code, not by understanding the defect. That distinction is not academic. A model that fixes without understanding will happily reintroduce the same defect in next week's structurally different code, and your review process won't catch it because this time the fix isn't there to prompt you.
My rule of thumb: a task only counts as a true pass if the sanitizer cells are clean and the explanation correctly names the violated rule. Scoreboards that ignore the second half flatter every model they measure.
Where free infrastructure actually fits
Running this rig has two costs: compute for the model, and compute for the grading. The grading side is trivial — any laptop compiles five small files. The model side is where fresh releases get awkward. Open-weight launches often outpace what a typical C++ developer can self-host, and hosted access during the first hype weeks tends to be metered, waitlisted, or both.
This is the specific slot where I've been using MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. It currently provides free access to a selection of coding models plus a free server option, which maps neatly onto this workflow: when a release drops, I paste the buggy source and brief, collect the fix, and pipe it into verdict.py without standing up hardware or opening a billing page. That's the entire role it plays — a low-friction way to get model output into a hostile compiler pipeline while the release is still new enough to be interesting. If you already have capable local hardware, self-hosting beats any hosted option on reproducibility grounds, and I say that as someone describing the hosted route.
There's a broader point underneath the logistics. Open releases promise that nobody has to take the vendor's demo on faith. That promise is only kept if people actually run independent checks with methods others can repeat. A manifest, a driver script, and a sanitizer pipeline are my version of that; the whole thing fits in a gist.
Routing table for your own runs
| Situation | Route worth taking | Trade-off to accept |
|---|---|---|
| Capable GPU on your desk | Self-host weights, grade locally | Setup time; you're the ops team |
| No GPU, want a same-day read on a new release | A free hosted path such as MonkeyCode's free models and server | Availability and limits can shift; treat as screening, not gospel |
| Whole-repository, long-context evaluation | Serious self-hosting or a paid tier sized for it | Free access generally isn't provisioned for repo-scale context |
| Proprietary or regulated code in the prompt | Self-host, no exceptions | No hosted endpoint — free or paid — gets that code |
What this rig does not tell you
- A clean sweep means "no UB observed on these inputs under these flags." Sanitizers only witness executed paths, and my input sets are finite by design. It's a smoke screen, not a correctness proof.
- Five task families is a deliberately small battery. A model that clears it has earned a deeper evaluation, not your codebase.
- Free-tier anything is a moving target. I have not verified quotas, context ceilings, or how long current terms hold, so nothing in my tooling hard-depends on any single provider, and neither should yours.
- Scores age terribly. Any per-release numbers I published today would be stale within a month and unverifiable to you right now — which is precisely why the artifact in this post is the harness rather than a leaderboard.
- If your daily C++ is greenfield numerics with no legacy surface, defect-repair grading measures a muscle you rarely use; you'd be better served by a design-level evaluation.
Closing thought
The next release that floods your feed deserves neither blind excitement nor reflexive dismissal. Give it the same five tasks, let -Werror and two sanitizers have the first word, and read the explanation before you celebrate the fix. The tooling in this post is the whole method — take it, rerun it, and if your verdicts on a fresh release diverge from what the demo gifs implied, that delta is the most interesting thing you can post.
Top comments (0)