The Placebo Patch: Measure Your C++ Eval Harness's Noise Floor Before You Trust a Regression

A four-person team running an AI patch loop on a C++ service added a merge gate last quarter. One rule: if any golden case changes verdict, the candidate patch is rejected. On a Tuesday morning the gate rejected a patch that renamed buf_len to buffer_length in a single function.

Nothing changed at runtime. The rename moved one assertion message, one stack trace line, and the interleaving of two log lines. The harness reported three regressions, and all three were artifacts.

The gate was measuring itself. What it lacked was a control arm, and a control arm is cheap to build.

The number nobody records

Every eval harness has a noise floor: the rate at which verdicts change when nothing semantic changes. That number is a property of a triple — harness, toolchain, machine — and it moves whenever any leg moves.

Teams usually discover it the expensive way. A regression is reported, a developer spends a day bisecting, and the conclusion is "flaky case, rerun it." Then the gate loses its authority, and real regressions get rerun to death alongside the imaginary ones.

A placebo corpus fixes this in about eighty lines of glue.

Step 1 — Build a placebo corpus

A placebo patch is a semantically neutral edit whose only job is to expose harness brittleness. Five classes cover most of the ground.

ID	Placebo patch	What it perturbs
P0	null patch (identical tree)	pure run-to-run variance
P1	comment-only insertion	nothing intentional
P2	local variable rename in one TU	debug info, assertion messages, symbol layout
P3	reorder #include lines in one file	declaration order, token positions
P4	clang-format applied to one function	line and column numbers
P5	add an unused static helper	symbol table, link order

P0 is the honest baseline. P1 through P5 should be invisible; when they are not, the case is reacting to shape rather than behavior.

Store them as unified diffs under placebo/patches/, named so they sort predictably.

placebo/
  patches/
    00-null.patch        # empty diff, applied as a no-op
    01-comment.patch
    02-rename.patch
    03-include-order.patch
    04-format.patch
    05-dead-static.patch
  out/                   # one JSON per (patch, run)

Writing P2 and P4 by hand across a 2,000-line translation unit is dull work with a sharp oracle attached: if a supposedly neutral edit accidentally changes behavior, the first matrix run exposes it, not a reviewer. That self-verifying property is what makes the variant generation a reasonable job for free model access.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness in this setup generates its neutral variants through MonkeyCode's free model access. Quota, throughput, and availability terms belong to the project's own documentation and should be read there rather than from any number quoted in an article.

Step 2 — Run the matrix

The harness needs one property: it must emit machine-readable verdicts, one entry per case. A --json flag on stdout is enough.

#!/usr/bin/env bash
# placebo/run_matrix.sh
set -euo pipefail

K="${K:-5}"                       # runs per patch
OUT=placebo/out
mkdir -p "$OUT"

for patch in placebo/patches/*.patch; do
  name=$(basename "$patch" .patch)
  for run in $(seq 1 "$K"); do
    if [ "$name" != "00-null" ]; then
      git apply "$patch"
    fi
    cmake --build build -j"$(nproc)" >/dev/null
    ./build/harness --json > "$OUT/$name.$run.json"
    git checkout -- .             # drop the placebo patch
  done
done

Two details decide whether the data means anything. Build with the same compiler, flags, and machine for the whole matrix, because a mixed matrix measures the toolchain. Keep K odd if the report is going to talk about a modal verdict.

Step 3 — Turn verdicts into a per-case admission report

The matrix is only raw material. The useful output is one row per golden case with two booleans: did it disagree with itself across runs, and did it flip under a neutral patch?

#!/usr/bin/env python3
"""noise_report.py - admission verdict per golden case."""
import collections, glob, json, pathlib

REF_PATCH = "00-null"

by_patch = collections.defaultdict(list)
for path in sorted(glob.glob("placebo/out/*.json")):
    record = json.loads(pathlib.Path(path).read_text())
    by_patch[record["patch"]].append(record)

cases = sorted(next(iter(by_patch.values()))[0]["cases"])
report = {}

for case in cases:
    reference = by_patch[REF_PATCH][0]["cases"][case]["status"]

    flake = any(
        len({r["cases"][case]["status"] for r in runs}) > 1
        for runs in by_patch.values()
    )
    placebo_sensitive = any(
        reference not in {r["cases"][case]["status"] for r in runs}
        for patch, runs in by_patch.items()
        if patch != REF_PATCH
    )

    report[case] = {
        "flake": flake,
        "placebo_sensitive": placebo_sensitive,
        "admission": ("gate" if not flake and not placebo_sensitive
                      else "advisory" if not flake
                      else "quarantine"),
    }

pathlib.Path("placebo/noise_report.json").write_text(
    json.dumps(report, indent=2, sort_keys=True))
excluded = [c for c, r in report.items() if r["admission"] != "gate"]
print(f"{len(excluded)}/{len(cases)} cases excluded from the merge gate")

Three outcomes fall out of the two booleans, and each one has a different owner.

run-to-run flake	placebo sensitive	admission	action
no	no	gate	eligible to block merges
no	yes	advisory	normalize the observable, or stop blocking on it
yes	either	quarantine	fix or delete on a deadline, never gate

The middle row is what pays for the exercise. A case that never flakes but flips when a function is reformatted is not measuring behavior; it is measuring line numbers.

Step 4 — Gate on the promotable set only

The gate script stays deliberately small. It reports a regression only when the case is in the gate set, and it prints everything it suppressed.

#!/usr/bin/env python3
"""gate.py CANDIDATE.json REFERENCE.json"""
import json, sys

noise = json.load(open("placebo/noise_report.json"))
candidate = json.load(open(sys.argv[1]))
reference = json.load(open(sys.argv[2]))

gateable = {c for c, r in noise.items() if r["admission"] == "gate"}
changed = {
    c for c in candidate["cases"]
    if candidate["cases"][c]["status"] != reference["cases"][c]["status"]
}

print(json.dumps({
    "regressions": sorted(changed & gateable),
    "suppressed_noisy": sorted(changed - gateable),
}, indent=2))

sys.exit(1 if changed & gateable else 0)

suppressed_noisy still needs to be visible. A case that changed verdict but is not gateable is exactly the information a human needs to decide whether to repair the case instead of the patch.

Step 5 — Re-measure instead of inheriting

The noise floor is not a constant, so the report has an expiry date. Re-run the matrix when the compiler version, the harness, or the CI machine class changes.

A five-patch by five-run matrix multiplies suite time by twenty-five. That makes it a nightly job rather than a per-pull-request job. The operator runs it on MonkeyCode's free server option, which is a defensible use of idle cycles: the control arm is slow, deterministic, and produces nothing user-visible.

If the suite is too slow even for that, cut K to three and say so in the report. A wide interval that is labeled honestly beats a narrow one that is invented.

Limitations

The control arm measures variance, not correctness. If every golden case encodes the same wrong expectation, all placebo runs agree with each other, and the gate stays confidently wrong.

Networked or timing-dependent cases belong outside the placebo arm. Five runs over a live HTTP dependency measure the network's mood, not the harness's stability. Keep the corpus on deterministic unit-level cases.

The gateable set can shrink. Track its size over time. A rough rule of thumb: if the promotable set falls below about sixty percent of the corpus, the corpus needs rewriting rather than more quarantine.

Who should skip this

A solo project with one fast, deterministic suite and no automated patch loop gains little here, because there is no gate to defend.

Anyone without a reproducible reference build should fix that first. The reference run is the baseline that every comparison in the report is drawn against.

Teams whose individual cases take minutes cannot run P x K matrices at all. They should start with P0 alone and one run per patch, just to find out whether the floor is measurable on their hardware.

Closing

A gate that fires on a variable rename costs more trust than it buys. Measuring the floor takes one script, one nightly job, and a decision table, and it converts "flaky case, rerun it" into a number somebody can act on. Nothing in this workflow requires a paid runner — the whole experiment above was run on MonkeyCode's free tier, which is also a low-stakes way to find out whether the control arm earns a permanent slot in CI.