Let the Compiler Argue With the Demo: A Repeatable C++ Grader for Coding Models

Finley Li — Wed, 05 Aug 2026 07:03:12 +0000

Benchmark screenshots are abundant. The narrower question a working C++ team needs answered is harsher: can a coding assistant repair the sort of defect that actually appears in our tree, under our warning policy, without smearing edits across files that were off limits? I stopped asking for impressions and built a tiny grader that turns each model answer into the same three artifacts: a strict build, a hidden behavior check, and a patch-boundary check.

The example below is C++, but the method travels. The model is interchangeable; the gates are not.

Grade the thing a maintainer would grade

A polished demo can hide the exact places where C++ gets unpleasant: warnings that become errors only in CI, bounds that look fine until UBSan runs, header hygiene that differs between GCC and Clang, and fixes that are technically correct yet rewrite half the file. So the unit of evaluation is not plausibility. It is a pass or fail answer to three mechanical questions:

Build gate: does the candidate compile with the same flags your project already uses, including warnings-as-errors and sanitizers?
Behavior gate: does it match output the model never got to inspect?
Scope gate: if the instruction was repair this defect only, did the patch stay inside that fence?

Because all three can be scripted, each candidate costs roughly one command to judge. That changes the economics: you can repeat runs instead of treating one lucky completion as evidence.

Use tasks taken from scars, not puzzle sites

Keep the corpus small and mean. Five to ten tasks are enough if each one came from a real embarrassment: an off-by-one in a sliding window, a reference that outlived its source, a missing virtual destructor, an integer promotion surprise, a const fix that should not alter ABI, a feature request constrained to one translation unit.

Here is a different starter task. The model sees this file and is told to fix the sliding-window sums without changing the interface or adding dependencies:

// window_sums.cc - candidate sees this buggy version
#include <iostream>
#include <vector>

int main() {
    const std::vector<int> values{2, -1, 4, 3, -2};
    const std::size_t k = 3;

    for (std::size_t i = 0; i <= values.size() - k; ++i) {
        int sum = 0;
        for (std::size_t j = i; j <= i + k; ++j) {
            sum += values[j]; // defect: one item too many, then out of range
        }
        std::cout << sum << std::endl;
    }
}

The hidden side of the task is only an expected file and an unforgiving build:

# expected.stdout contains exactly: 5 6 5 on separate lines
c++ -std=c++17 -Wall -Wextra -Wpedantic -Werror \
  -fsanitize=address,undefined -fno-omit-frame-pointer \
  candidate.cc -o candidate
./candidate > got.stdout
cmp expected.stdout got.stdout

A solid answer changes the inner bound to j < i + k and handles the values.size() < k edge before the outer loop subtracts. A weak answer may still print 5 6 5 once while walking past the vector on the final iteration; that is why output comparison and sanitizer silence are separate gates.

The reusable piece: one JSONL in, one table out

The grader below is intentionally boring. It reads a JSONL file where each line has a model label, a task directory, and a source file, then emits one JSON verdict per candidate. Swap the model client elsewhere; keep this contract stable.

#!/usr/bin/env python3
# verdict.py - strict-build, hidden-output, sanitizer-noise grader
import json
import os
import pathlib
import subprocess
import sys
import tempfile

FLAGS = [
    '-std=c++17', '-Wall', '-Wextra', '-Wpedantic', '-Werror',
    '-fsanitize=address,undefined', '-fno-omit-frame-pointer', '-g',
]

def sh(cmd, cwd=None):
    return subprocess.run(cmd, cwd=cwd, text=True, capture_output=True, timeout=90)

def judge(row):
    out = {'model': row['model'], 'task': row['task'], 'built': False,
           'matched': False, 'clean_runtime': False, 'scoped': None}
    task = pathlib.Path(row['task'])
    with tempfile.TemporaryDirectory() as td:
        exe = pathlib.Path(td) / 'candidate'
        build = sh(['c++', *FLAGS, row['source'], '-o', str(exe)])
        out['built'] = build.returncode == 0
        if not out['built']:
            out['why'] = build.stderr[-700:]
            return out
        run = sh([str(exe)], cwd=task)
        want = (task / 'expected.stdout').read_text()
        out['matched'] = run.stdout == want
        out['clean_runtime'] = run.returncode == 0 and run.stderr == ''
        allowed = task / 'allowed.diff'
        if allowed.exists():
            patch = sh(['diff', '-u', str(task / 'original.cc'), row['source']])
            out['scoped'] = allowed.read_text() in patch.stdout
    out['ok'] = out['built'] and out['matched'] and out['clean_runtime'] \
        and out['scoped'] is not False
    return out

for line in open(sys.argv[1]):
    print(json.dumps(judge(json.loads(line))))

Feed it lines such as:

{"model":"candidate-a","task":"tasks/window_sums","source":"answers/a/window_sums.cc"}

The scope check is deliberately crude in this sketch: if a task requires patch discipline, store an approved unified-diff shape in allowed.diff and tighten the comparison later. The important part is that scope is data, not a vibe.

Making the model calls cheap enough to repeat

The compiler time is mostly already yours; the completions are where evaluation usually gets expensive. For the no-invoice pass behind this article I used MonkeyCode, because the operator supplying this preset says free model access and a free server option are currently available. Verify the current terms before you depend on either one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow stayed provider-neutral: prompt with the task file and the defect statement, extract the fenced code, save it under answers/<label>/<task>.cc, then run verdict.py. Put the endpoint behind an environment variable so a changed free tier is a config edit rather than a rewrite. The reason free access matters is not thrift for its own sake; it buys repetition. A single sample per task is weather. Five runs at fixed settings starts to look like climate, especially when one candidate oscillates between clean fixes and warning-introducing rewrites.

The separate server option is also practical for C++: sanitizer-instrumented binaries are heavier than a chat transcript suggests. Keeping the build-and-run loop off the laptop that is also running your editor, browser, and meeting software prevents the experiment from becoming a local resource fight.

Patterns worth expecting

I am not publishing a model leaderboard; the sample is too small and your repository has its own failure signature. The repeatable patterns were these:

The first elimination is usually warnings-as-errors, not algorithmic insight. Candidates introduced unused locals, sign-compare noise, and shadowed names while chasing the right idea.
Matching stdout and running clean under ASan or UBSan are different events. A candidate can land the expected numbers after stepping outside a container earlier in the same run.
Rankings moved after repetition. Peak cleverness was less useful than boring consistency when the task resembled maintenance rather than invention.
Scope checks exposed polite overreach: extra refactors, renamed helpers, and header churn that would make review slower even when tests passed.

Limits and poor fits

Canonical defects are contaminated by training data. Iterator, bounds, and lifetime tasks measure reliability on well-known traps. Add at least two tasks distilled from your own incident history before drawing conclusions.
A ten-task corpus can flag a model that cannot survive -Werror; it cannot prove a two-point edge between competitors. Treat close scores as ties.
Free capacity is weather too. Rate limits, available models, and server headroom can shift. The harness should outlive any one provider arrangement.
Skip this style if the work is novel algorithm design, safety-certified control flow, or anything governed by formal review gates. Use property-based tests, proofs, or mandated human sign-off instead.

A practical first afternoon

Do not clone somebody else's list. Mine git log --grep=fix for five defects you still remember, reduce each to a runnable directory with hidden expected output, and wire the candidates you can obtain cheaply into the grader. If you need a zero-cost place to begin that loop, MonkeyCode's current free model access and free server option are one possible source of completions and compute; the verdict script does not care where answers originate.

The larger habit is the point: model choice is an empirical question with a refresh button. Build the gates once, rerun them when a new assistant or a new free tier appears, and let your compiler, sanitizers, and patch boundaries argue with the launch posts.

DEV Community: Finley Li