Golden Cases Catch Silent Regressions: A Minimal Eval Harness for AI C++ Patches

AI coding models have changed the default workflow. The model writes the first draft; the human reviews the diff. The bottleneck is no longer generation — it is verification.

Here is a scenario that repeats in C++ codebases. A config parser accepts key= lines. That is the contract. A coding model proposes a cleanup: skip lines that look empty, use a faster map, drop the redundant branch.

The patch compiles. CI stays green. The review takes five minutes.

Two weeks later, a user reports that a configuration option silently disappeared. The parser stored an empty value for key=; the new version skips the line entirely. No test covers the case.

The model did not know the behavior contract. The reviewer did not have time to re-read every diff line. The missing piece is a harness that pins observable behavior to golden cases.

What a golden case is

Golden cases are input/output pairs that describe how a function must behave. They are not unit tests in the usual sense. A unit test asserts implementation details; a golden case asserts observable behavior.

For a parser, a golden case is a line of input and the exact output it must produce. For an algorithm, it is a set of arguments and the expected result. The point is to make the behavior contract executable.

The harness

An eval harness for AI-generated C++ patches needs five pieces:

1. A set of golden cases in a machine-readable format.
2. A build step that compiles the candidate patch with sanitizers.
3. A runner that executes each case against the compiled binary.
4. A grader that compares actual output to expected output.
5. A report that names the failing cases.

The harness below is about 70 lines of Python. It is deliberately small. A larger framework would obscure the mechanics.

#!/usr/bin/env python3
"""golden_harness.py — grade a C++ patch against golden cases."""

import json
import subprocess
import sys
import tempfile
from pathlib import Path


def load_cases(path):
    with open(path) as fh:
        return json.load(fh)["cases"]


def build(src, out):
    cmd = ["g++", "-std=c++20", "-O2", "-Wall", "-Wextra",
           "-fsanitize=address,undefined", src, "-o", out]
    return subprocess.run(cmd, capture_output=True, text=True, timeout=120)


def run_case(binary, case):
    try:
        return subprocess.run(
            [binary] + case.get("args", []),
            input=case.get("stdin", ""),
            capture_output=True, text=True,
            timeout=case.get("timeout_s", 5),
        )
    except subprocess.TimeoutExpired:
        return None


def main():
    if len(sys.argv) != 3:
        print("usage: golden_harness.py <patch.cpp> <golden.json>")
        return 2

    src, golden_path = sys.argv[1], sys.argv[2]
    cases = load_cases(golden_path)

    with tempfile.TemporaryDirectory() as td:
        binary = str(Path(td) / "patch_bin")
        build_proc = build(src, binary)
        if build_proc.returncode != 0:
            print("BUILD FAILED")
            print(build_proc.stderr)
            return 1

        passed = 0
        for case in cases:
            proc = run_case(binary, case)
            if proc is None:
                print(f"TIMEOUT {case['name']}")
                continue
            expected = case.get("expect_stdout", "")
            expected_code = case.get("expect_exit", 0)
            ok = proc.returncode == expected_code and proc.stdout == expected
            passed += int(ok)
            if not ok:
                print(f"FAIL {case['name']}")
                print(f"  exit: got {proc.returncode}, want {expected_code}")
                print(f"  stdout: got {proc.stdout!r}, want {expected!r}")
                if proc.stderr:
                    print(f"  stderr: {proc.stderr!r}")

        print(f"{passed}/{len(cases)} golden cases passed")
        return 0 if passed == len(cases) else 1


if __name__ == "__main__":
    sys.exit(main())

The golden file is plain JSON. Each case describes one behavior contract.

{"cases": [
  {"name": "empty_value", "stdin": "a=\n", "expect_stdout": "a=\n", "expect_exit": 0},
  {"name": "value_contains_equals", "stdin": "a=b=c\n", "expect_stdout": "a=b=c\n", "expect_exit": 0},
  {"name": "duplicate_key_last_wins", "stdin": "a=1\na=2\n", "expect_stdout": "a=2\n", "expect_exit": 0},
  {"name": "spaces_in_value", "stdin": "a=hello world\n", "expect_stdout": "a=hello world\n", "expect_exit": 0},
  {"name": "spaces_in_key", "stdin": " a=1\n", "expect_stdout": " a=1\n", "expect_exit": 0},
  {"name": "sorted_output", "stdin": "b=2\na=1\n", "expect_stdout": "a=1\nb=2\n", "expect_exit": 0},
  {"name": "no_newline_at_eof", "stdin": "a=1", "expect_stdout": "a=1\n", "expect_exit": 0}
]}

The reference implementation is a small parser. It stores the last value for each key and prints sorted output.

#include <iostream>
#include <map>
#include <string>

int main() {
    std::map<std::string, std::string> kv;
    std::string line;
    while (std::getline(std::cin, line)) {
        auto eq = line.find('=');
        if (eq == std::string::npos) {
            continue;
        }
        kv[line.substr(0, eq)] = line.substr(eq + 1);
    }
    for (const auto& [key, value] : kv) {
        std::cout << key << '=' << value << '\n';
    }
    return 0;
}

Run the harness once to confirm the baseline.

python3 golden_harness.py parser.cpp golden.json
# 7/7 golden cases passed

The silent regression

Now the model proposes a cleanup. The diff is one line: line.substr(eq + 1) becomes line.substr(eq). It compiles cleanly. It looks like a harmless simplification.

The harness disagrees.

FAIL empty_value
  stdout: got 'a==\n', want 'a=\n'
FAIL value_contains_equals
  stdout: got 'a==b=c\n', want 'a=b=c\n'
FAIL duplicate_key_last_wins
  stdout: got 'a==2\n', want 'a=2\n'
FAIL spaces_in_value
  stdout: got 'a==hello world\n', want 'a=hello world\n'
FAIL spaces_in_key
  stdout: got ' a==1\n', want ' a=1\n'
FAIL sorted_output
  stdout: got 'a==1\nb==2\n', want 'a=1\nb=2\n'
FAIL no_newline_at_eof
  stdout: got 'a==1\n', want 'a=1\n'
0/7 golden cases passed

Every case fails because the bug is systematic. The = stays in the value, and the contract breaks everywhere. A human reviewer might catch the off-by-one in a diff. The harness catches it in under a second, before the diff reaches the reviewer.

Turning the harness into a loop

Golden cases only help if they run before human review. The loop has four steps:

1. Write golden cases from real bug reports, edge cases, and the function's specification. Start with a small set; add one case for every regression found later.
2. Send the task and the golden cases to a coding model. Ask for a patch, not a summary.
3. Run the harness on the candidate patch. If a case fails, return the failure output to the model and ask for a fix.
4. Merge only when the patch passes every golden case and a human reads the diff.

The failure output is the feedback channel. Send it back to the model verbatim: "The harness reports 0/7. Fix the patch so all seven cases pass." The model sees concrete diffs, not a vague "it's broken."

The loop converts model output from "looks plausible" to "passes the behavior contract." It also produces a growing regression suite that outlives any single patch.

Where the free tier fits

Running this loop needs two things: a model that can generate candidate patches, and an environment that can compile and run them. MonkeyCode, an open-source project, currently offers both on a free tier. The free model access ships with a 10-million-token allowance for generating patches. The free server option provides a hosted model endpoint, so the harness does not require a self-hosted inference server.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance is not the point. The point is that the loop — generate, compile, run golden cases, return failures, retry — can run end to end without self-hosting a model server. A 10-million-token allowance is enough for many generate-and-fix iterations on small C++ patches, which is exactly what this harness consumes. Teams that want to try the loop without provisioning hardware can use the free server option for model access; the golden cases above are a ready-made first run.

Limitations

Golden-case harnesses have sharp edges.

They only know the cases you write. A patch can pass every golden case and still break untested behavior. The harness is a contract test, not a proof.

Exact string comparison fails on nondeterministic output. Timestamps, hash map ordering, and parallel logs produce false failures. Fix this with a normalizer or a JSON grader instead of raw string equality.

The build step is the bottleneck. Compiling a large C++ project for every candidate patch is slow. The harness works best on small, self-contained units or extracted reproducers.

Teams with GUI, latency, or interactive behavior contracts should not use this approach. Golden stdout comparisons cannot capture what a widget looks like or how a request feels.

Who should use it

Use this harness when the patch is small and the behavior is contract-like: parsers, formatters, serializers, sorting, string manipulation, protocol encoding.

Skip it when the behavior is visual, interactive, or performance-critical in ways that a single run cannot measure. For those, the harness is still useful as a first gate, but it cannot be the only gate.

The reviewer bottleneck is not going away

The practical response is to make the model's output checkable before a human spends ten minutes reading a diff.

A 70-line harness, seven golden cases, and one compile command are enough to start. The golden cases are the part that keeps working after the model, the token budget, and the server option change.

Add the empty-value case first. It is the one that always breaks.