Shuffle the Golden Cases: Catching Eval State Leaks That Order Hides

A patch-eval suite can be green for months and still be lying. The lie usually has nothing to do with the patch. It has to do with the sequence.

Here is the failure shape. Case 12 compiles src/vec.cpp with -DEVAL_VARIANT=sse and drops build/vec.o into a scratch directory shared by the whole run. Case 17 wants the scalar variant, but its build script skips compilation when build/vec.o already exists. Case 17 links the SSE object, passes its test, and reports PASS. The patch under review never touched vector code.

Nothing in case 17's log mentions case 12. Nothing in the diff mentions either. The suite is order-dependent, and because the order was fixed in the manifest, the dependency was invisible.

1. Treat order as an input you must vary

Most eval harnesses treat the case list as configuration. It is really an input variable — as much as the patch, the compiler flags, or the model version.

If a verdict changes when that input changes, the suite has a leak. If it does not, order independence is a property you can assert, not a hope you maintain.

The gate is simple: run the suite in a canonical order twice, then in a handful of seeded shuffles. Compare verdict vectors, not logs. Flag any case whose verdict moves.

2. Make the manifest the only source of order

Cases live in a JSON manifest. Each entry owns its command, its timeout, and its expected verdict. No shell script reorders anything downstream.

[
  {"id": "c01_vec_basic",   "cmd": ["bash", "cases/c01.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c12_vec_sse",     "cmd": ["bash", "cases/c12.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c17_vec_scalar",  "cmd": ["bash", "cases/c17.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c22_asan_owner",  "cmd": ["bash", "cases/c22.sh"], "expect": "FAIL", "timeout": 300}
]

Two details matter. First, expect is recorded even for cases that must fail — a negative case that starts passing is a regression too. Second, every case runs in the same scratch root per suite run. Isolating each case in a fresh container would hide exactly the bug this gate exists to find.

3. Run canonical twice, then seeded shuffles

The harness below is the whole gate, roughly sixty lines. It is plain Python on purpose: it needs to be readable enough that someone debugging a flip can trust it.

# shuffle_gate.py — order-dependence gate for a patch-eval suite
import json, random, shutil, subprocess, sys, time
from pathlib import Path

CASES = {c["id"]: c for c in json.loads(Path("cases.json").read_text())}
ROOT  = Path("scratch")

def run_suite(order, tag):
    work = ROOT / tag
    if work.exists():
        shutil.rmtree(work)          # reset once per run, never per case
    work.mkdir(parents=True)
    rows = []
    for cid in order:
        case = CASES[cid]
        t0 = time.monotonic()
        try:
            p = subprocess.run(case["cmd"], cwd=work, capture_output=True,
                               text=True, timeout=case.get("timeout", 300))
            got = "PASS" if p.returncode == 0 else "FAIL"
        except subprocess.TimeoutExpired:
            got = "TIMEOUT"
        rows.append({"id": cid, "verdict": got,
                     "secs": round(time.monotonic() - t0, 2)})
    return rows

def vector(rows):
    return {r["id"]: r["verdict"] for r in rows}

Canonical order runs twice with different tags so a TIMEOUT or a flaky model call shows up before shuffling begins. Shuffles use a fixed seed list committed to the repo, because a flip you cannot reproduce is a flip you cannot fix.

def main():
    base_order = list(CASES)
    a = run_suite(base_order, "canon_a")
    b = run_suite(base_order, "canon_b")
    report = {"canonical": {}, "shuffles": []}

    va, vb = vector(a), vector(b)
    for cid in base_order:
        if va[cid] != vb[cid]:
            report["canonical"][cid] = {"a": va[cid], "b": vb[cid]}

    for seed in (17, 41, 1234, 90210):
        order = base_order[:]
        random.Random(seed).shuffle(order)
        rows = run_suite(order, f"shuf_{seed}")
        v = vector(rows)
        flips, prev = [], None
        for r in rows:
            if v[r["id"]] != va[r["id"]]:
                flips.append({"case": r["id"], "base": va[r["id"]],
                              "now": v[r["id"]], "after": prev})
            prev = r["id"]          # heuristic attribution, see step 5
        report["shuffles"].append({"seed": seed, "order": order, "flips": flips})

    Path("leaks.json").write_text(json.dumps(report, indent=2))
    dirty = report["canonical"] or any(s["flips"] for s in report["shuffles"])
    sys.exit(1 if dirty else 0)

if __name__ == "__main__":
    main()

The exit code is the point. A gate that prints a warning gets ignored by week three.

4. Read the verdict vector, not the log tail

Logs are noisy. Four hundred lines of compiler output and one ASan report look the same whether a case leaked state or genuinely regressed.

The vector is small and diffable. On a clean suite, leaks.json carries empty objects and four empty flip lists. On a dirty one it names the case, the canonical verdict, the shuffled verdict, and the case executed immediately before it.

The after field is a lead, not a verdict. A leak can travel through the filesystem or the environment and surface many cases later — the SSE object in step 1 survives three unrelated cases before case 17 consumes it.

5. Confirm by isolation, then quarantine

Attribution only becomes evidence when the pair reproduces in isolation. Run the two cases alone, in the suspected order, with the same scratch root. If the flip returns, the pair is real.

Three artefacts come out of confirmation:

1. A minimal reproducer — two case IDs and a seed, added as a permanent assertion in tests/test_order.sh.
2. A quarantine entry in leaks.json with a note pointing at the shared resource: build/vec.o, port 8080, ASAN_OPTIONS, /tmp/eval_cache.
3. A fix on the offending side, not the victim. Case 12 should build out-of-tree; case 17 should not be silently patched to clean the world first.

Quarantine means the entry stays visible and the gate keeps failing until the fix lands. Deleting the case removes the signal and restores the false green.

6. Shuffling is not free, and that changes the budget

A four-shuffle matrix on a twenty-case suite is five full suite runs. On a shared build machine that cost lands on wall clock and disk, not just CPU.

This is where free model access changes the triage loop rather than the verdict. Confirmed flips are rare; most shuffled runs are clean and never need the expensive model. Re-grading triage runs with a free-tier model through MonkeyCode keeps the matrix affordable, and the free server option gives the harness a persistent box for the seed list, the scratch root, and leaks.json between runs. The strong model is then reserved for the few flips that survive isolation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat about shared boxes: a free server is also a shared-resource magnifier. Ports, /tmp, and disk quotas are the most common leak carriers in exactly this environment, which makes the gate more valuable there, not less.

7. Decision table for the flip report

Observation	Likely cause	Action
Flip only under seeded shuffles, stable in both canonical runs	Filesystem, port, or env leak between cases	Confirm pair in isolation; fix the producing case
Flip between canon_a and canon_b, same order	Case nondeterminism or unseeded model output	Pin seed and flags; re-check the grader with a canary
Case passes alone, fails after any neighbour	Environment mutation by whoever ran first	Diff env and cwd before and after the neighbour
Whole suite flips in one run	Harness-level state: disk full, port held, stale scratch root	Reset the scratch root once per run, never per case
Flip appears only in -O2 builds	Optimization-sensitive undefined behaviour	Route it to an opt-split gate, not this one

8. Limits, and who should skip this

The gate finds order coupling. It says nothing about whether a grader is correct, whether golden cases cover the patched line, or whether the model chose a sound fix.

Skip it when every case already runs in a fresh container or VM — isolation removes the leak class by construction, and the shuffle matrix is pure cost. Skip it for suites so slow that five runs exceed the CI budget; sample ordered pairs instead of full shuffles. Skip it below roughly ten cases, where a human reading the manifest beats a matrix.

Adopt it when cases share a workspace, a port, a cache, or an environment — which is to say, on most real C++ patch suites running on one machine.

A suite that only passes in one order is not a suite. It is a script with a fixed cast, and the patch you are grading is not the thing deciding the outcome. Commit the seed list alongside the cases, so the next person sees the same flips you did.