Order Is a Hidden Input: Permutation-Sweeping an Agent-Patched Test Suite

An agent patch can pass your entire suite in the default order and still be order-coupled. The green run proves the suite works in one sequence, not that the tests are independent. Before I trust a machine-written patch, I want to know which tests only pass because something ran before them.

That is a different question from "did the patch's tests touch the changed lines" or "are the assertions strong enough." Those ask about coverage and oracle quality. This asks about the suite as a stateful program, and agent patches are unusually good at perturbing that state: a new module-level cache, a fixture that mutates a shared row, an import with a side effect, a monkeypatch that never gets undone.

Below is the sweep I run, the attribution step that turns a failure matrix into a minimal predecessor set, and the policy for what to block versus what to freeze.

1. Establish the order contract first

Run the suite twice: once in the default order, once with a fixed seed but no shuffling plugin active. If those two disagree, you have a flake, not order coupling, and the sweep will just produce noise.

Record the baseline as a set of node IDs plus outcomes. Everything downstream diffs against this.

# baseline, default order
pytest -q --tb=no -p no:cacheprovider --junitxml=.order/base.xml

# confirm the runner is stable before sweeping for coupling
pytest -q --tb=no -p no:cacheprovider --junitxml=.order/base-retry.xml

If a test flips between those two files, quarantine it first. A permutation sweep over a suite with known flakes cannot attribute anything — you cannot distinguish "failed because of predecessor" from "failed because it is Tuesday."

2. Sweep seeded permutations, not all permutations

A 400-test suite has more orderings than you can run. Sample.

# order_probe.py — seeded permutation sweep
import os, random, subprocess, xml.etree.ElementTree as ET

def collect():
    out = subprocess.run(["pytest", "--collect-only", "-q"],
                         capture_output=True, text=True, check=True)
    return [l.strip() for l in out.stdout.splitlines() if "::" in l]

def run_order(nodeids, seed, outdir=".order"):
    os.makedirs(outdir, exist_ok=True)
    xml = os.path.join(outdir, f"run-{seed}.xml")
    cmd = ["pytest", "-q", "--tb=no", "-p", "no:cacheprovider",
           f"--junitxml={xml}", *nodeids]
    env = dict(os.environ, PYTHONHASHSEED=str(seed), ORDER_SEED=str(seed))
    subprocess.run(cmd, capture_output=True, text=True, env=env)
    results = {}
    for case in ET.parse(xml).iter("testcase"):
        nid = f"{case.get('classname')}::{case.get('name')}"
        bad = any(case.find(t) is not None for t in ("failure", "error"))
        results[nid] = "fail" if bad else "pass"
    return results

def sweep(seeds):
    ids = collect()
    matrix = {}
    for seed in seeds:
        order = ids[:]
        random.Random(seed).shuffle(order)
        for nid, outcome in run_order(order, seed).items():
            matrix.setdefault(nid, {})[seed] = outcome
    return matrix

Two caveats are baked into that code on purpose. PYTHONHASHSEED changes with the seed, so you catch set/dict iteration dependencies for free — but it also means a failure could be hash-order rather than predecessor-order, and step 3 has to tell them apart. And junitxml mangles node IDs; verify the mapping against --collect-only output before you trust the matrix keys.

Useful metrics to report rather than vague "some flakiness":

• Coupling rate = tests whose outcome varies across seeds ÷ tests collected.
• Baseline-clean rate = coupled tests that pass in the default order (these are the ones your CI currently hides).
• Minimized pair count = distinct predecessor→target edges after step 3.

Report what you measure. A coupling rate you did not compute is a claim you cannot defend.

3. Minimize each failure to a predecessor set

A failing permutation tells you almost nothing — 37 tests ran before the target. Delta-debug the prefix until removing any remaining predecessor stops reproducing the failure.

# illustrative: expects run_order() from above
def reproduces(prefix, target, seed):
    return run_order(prefix + [target], seed).get(target) == "fail"

def minimize(prefix, target, seed):
    current = list(prefix)
    changed = True
    while changed:
        changed = False
        for p in list(current):
            trial = [x for x in current if x != p]
            if reproduces(trial, target, seed):
                current = trial
                changed = True
    return current

This is quadratic in prefix length, which is fine because you only run it for tests that already failed. The output is a ridiculously small repro — usually one or two tests — and that small repro is the artifact you attach to the patch review.

4. Classify before you fix

The minimized pair is evidence, not a diagnosis. Sort it:

Signal	Likely cause	Action
Fails with exactly one minimized predecessor, both in the same module	Module-level state or a leaked monkeypatch	Block the patch; state must be reset in a fixture
Fails only when target runs first	Setup lives in a suite-scoped fixture the test implicitly depended on	Block; the test is not runnable in isolation
Fails across seeds, minimized prefix empty	Hash-order or randomness inside the test itself	Quarantine with expiry, not a block
Fails in shuffled order but also on the base commit	Pre-existing coupling the patch merely exposed	Do not block the patch; file it, fix separately
Fails only in the reversed order	Teardown ordering assumption	Block; teardown must not depend on sibling teardown

That fourth row is the one reviewers skip. Run the same sweep on the base commit and diff the minimized-pair sets. A coupling delta of zero means the patch did not introduce the leak; it means the patch made an old one visible, which is still worth knowing but is not grounds for rejection.

5. Bound the freeze

Order-coupled tests are expensive to fix and tempting to skip. Bound it explicitly:

1. A freeze entry names the minimized predecessor, not just the test.
2. The freeze has a hard expiry date in the test file.
3. Frozen tests still run in the sweep, so you learn if the coupling spreads.
4. Expired freezes fail the build, not the test.

A freeze without a named predecessor is just a delete with extra steps. The pair is the unit of work.

Where a free tier actually helps

Two parts of this workflow are tedious rather than hard: writing the ddmin harness variants, and reading a pile of minimized pairs to guess at shared modules. That classification is a decent use of a code model — you paste the pair and its imports, and ask for the likely shared state. Running a few dozen permutations against a suite is also a decent use of remote compute if you would rather not pin your laptop for twenty minutes. MonkeyCode's free model access and free server option cover exactly those two slots for me; the sweep logic and the block/quarantine judgment stay with me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

• It only finds order coupling. Zero minimized pairs does not mean the patch is correct — it means the tests are independent. Pair it with diff coverage and assertion-strength checks.
• Cost scales with suite time. 30 permutations × a 12-minute suite is six hours. Sample fewer seeds, or sweep only the test files the patch touched.
• Suites with no per-test DB isolation will report nearly everything as coupled. Fix isolation first; the sweep will otherwise just print the same shared-row complaint a thousand times.
• Don't run this if you cannot produce a base-commit comparison. Without it, every pre-existing leak looks like the agent's fault and reviewers will start ignoring the gate within a week.
• Don't run this on a suite you have not already stabilised. Flakes and coupling are indistinguishable at the matrix level.

Start with five seeds on the patch's touched test files. If the minimized-pair set is non-empty, you have something concrete to put in the review, and that beats a comment saying "looks fine, tests pass."

If you want to try the sweep without standing up infrastructure first, the free server option is the least-friction place to start.