Six golden cases turn red in the same CI run. The team spends two days reverting a patch that was correct, and the single case that had genuinely regressed gets fixed last, almost by accident.
That scenario is a composite — the timeline is illustrative, the pattern is not. Any harness that grades model-written C++ against a fixed set of golden cases will eventually produce a run where the same case disagrees with itself. Two things follow. A two-state pass/fail result cannot describe that world, and a harness without a vocabulary for "I do not know" will convert infrastructure noise into engineering decisions.
The artifacts below are templates. They target Python 3.11 and a CMake/CTest tree, and they were not executed for this article, so treat paths and the runner call as starting points.
Step 1 — Make the result three-state
A golden case can fail for reasons that have nothing to do with the patch: a rate-limited model call, a build that timed out on a shared box, a runner that lost its cache. Grading those as fail is how noise becomes a revert. The runner below records infra_unknown as a distinct outcome and refuses to reason about it later.
# triage.py — repeat each golden case K times, then classify. Template.
import hashlib, os, re, subprocess, tempfile
from dataclasses import dataclass
K = int(os.environ.get("REPEAT_K", "5"))
INFRA_RE = re.compile(r"(429|503|connection reset|build timeout|no space left)", re.I)
@dataclass
class Attempt:
case: str
repeat: int
status: str # pass | fail | infra_unknown
exit_code: int
digest: str
note: str = ""
def digest(blob: str) -> str:
# Timing and pid lines track the scheduler, not the behaviour.
cleaned = "\n".join(
line for line in blob.splitlines()
if not re.search(r"(total time|pid=|elapsed)", line, re.I)
)
return hashlib.sha256(cleaned.encode()).hexdigest()[:12]
Step 2 — Reuse the case, not the process
Every repeat must start from a clean state. The environment pinning in this runner is deliberate: LC_ALL=C and TZ=UTC remove two classic sources of "it passed on the other machine", and a per-repeat TMPDIR keeps stale artifacts from leaking across attempts.
def run_case(case: str, repeat: int) -> Attempt:
with tempfile.TemporaryDirectory() as td:
proc = subprocess.run(
["ctest", "--test-dir", "build", "-R", f"^{case}$",
"--output-on-failure"],
capture_output=True, text=True, timeout=900,
env={**os.environ, "LC_ALL": "C", "TZ": "UTC",
"REPEAT": str(repeat), "TMPDIR": td},
)
blob = proc.stdout + proc.stderr
if INFRA_RE.search(blob):
return Attempt(case, repeat, "infra_unknown",
proc.returncode, "", blob[-400:])
return Attempt(case, repeat,
"pass" if proc.returncode == 0 else "fail",
proc.returncode, digest(blob))
Step 3 — Classify into four verdicts
The interesting bucket is not flaky. It is stable_pass_varying: a case that passes every single repeat while printing two different digests. That case is a silent regression waiting for a slightly different machine, and it is invisible to any harness that only reads exit codes.
def classify(attempts: list[Attempt]) -> dict:
graded = [a for a in attempts if a.status != "infra_unknown"]
infra = len(attempts) - len(graded)
if infra * 2 > len(attempts):
return {"verdict": "inconclusive", "infra": infra}
passes = sum(a.status == "pass" for a in graded)
digests = len({a.digest for a in graded if a.status == "pass"})
if passes == 0:
verdict = "stable_fail"
elif passes < len(graded):
verdict = "flaky"
elif digests > 1:
verdict = "stable_pass_varying"
else:
verdict = "stable_pass"
return {"verdict": verdict, "infra": infra,
"passes": passes, "digests": digests}
Step 4 — Bind each verdict to a CI action
A verdict that does not change what CI does is a log line, not a gate.
| verdict | CI action |
|---|---|
stable_pass |
green |
stable_pass_varying |
yellow: normalise the output before it becomes a red |
flaky |
red on main, quarantineable on a branch |
stable_fail |
red: the regression the harness exists to find |
inconclusive |
no grade at all; re-run the sweep later |
That last row carries the most weight. Against a shared model endpoint, inconclusive will be common, and a harness that turns it into a failure teaches reviewers to ignore failures.
Step 5 — Quarantine with an expiry date
Unstable cases need somewhere to live that is not /dev/null and not a permanently ignored test.
# quarantine.yaml
- case: containers/unordered_dedup
reason: iteration order leaks into stdout digest
owner: finley
opened: 2026-10-19
expires: 2026-11-02
# gate.py — an expired quarantine is a build failure, not a reminder
import datetime as dt, sys, yaml
today = dt.date.today()
bad = [q for q in yaml.safe_load(open("quarantine.yaml"))
if dt.date.fromisoformat(q["expires"]) < today]
if bad:
sys.exit("expired quarantine: " + ", ".join(q["case"] for q in bad))
Step 6 — Treat the source of the nondeterminism
Repeat-k is triage, not treatment. Three C++ changes remove most of the churn seen in practice.
// 1. Order leaks into output: per-process hashing, different bytes each run.
std::unordered_set<std::string> seen; // digest churns
std::set<std::string> seen_ok; // stable bytes, assertable
// 2. Wall clock: inject it, then assert on it.
struct Clock {
virtual ~Clock() = default;
virtual std::time_t now() const = 0;
};
struct FixedClock final : Clock {
std::time_t t;
std::time_t now() const override { return t; }
};
// 3. Worker count: 3 threads locally, 32 in CI, different interleavings.
const unsigned workers =
std::min(4u, std::max(1u, std::thread::hardware_concurrency()));
Add -fsanitize=address,undefined to the triage build rather than the shipping build. Uninitialised reads and signed overflow produce exactly the intermittent failures that repeat-k flags, and a sanitizer report names the bug instead of leaving a pass-rate puzzle. setarch -R is also worth a try when a crash reproduces under one address layout only.
What k buys, and what it does not
| k | model calls per case | catches a ~50% flake | catches a ~10% flake |
|---|---|---|---|
| 2 | 2 | often | rarely |
| 5 | 5 | ~94% | ~41% |
| 10 | 10 | ~99.9% | ~65% |
The arithmetic is the binomial one: a 10% flake survives ten repeats with probability 0.9^10 ≈ 0.35. So repeat-k is a coarse filter, and it is honest to say so before anyone wires k=20 into every pull request. Rare nondeterminism is caught better by digest divergence on green runs and by a nightly sweep, not by raising k in the merge path.
Where free model access and a free server fit
A k=10 sweep over 40 golden cases is 400 model calls, and on a metered endpoint that arithmetic appears on an invoice. MonkeyCode's free model access and free server option make such a sweep affordable for a side project or a first pass on a feature branch, which is the cheapest moment to discover that six of your cases were never stable.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Two constraints shape how the tool is used here. Free capacity is shared and rate-limited, so the INFRA_RE branch in Step 1 is not decoration — it is the point of the three-state design. The free tier also belongs in the triage loop, not the release gate, because a merge decision should never depend on capacity someone else controls. The offer itself (the operator states 10M free tokens plus a free server option) is worth confirming on the project's own page before planning around it.
To see the classifier fire before wiring it into CI, point it at the two cases you already suspect and spend the first sweep on a scratch branch.
Who should skip this
- Projects whose evals are already deterministic: fixed seeds, ordered containers, injected clocks, no concurrency. Repeat-k would buy nothing there.
- Teams that cannot store per-attempt records. Without digests and repeat indices, the classifier has no input and no audit trail.
- Anyone planning k=20 on every pull request. The wall time is real, and nightly is the honest cadence.
- Harnesses that grade prose rather than behaviour. Output digests are meaningful when the program prints bytes, not when a model writes a paragraph.
The habit worth keeping is small: a red run should produce a verdict with a name, and flaky and inconclusive are names. Without them every red run looks identical, and the team slowly learns to treat all of them as noise — including the one that was real.
Top comments (0)