Test Order Is an Input: Shuffle the Suite Before You Trust an Agent Patch

A green suite is a single sample, and the parameter that most often decides that sample is execution order. Generated patches exploit this quietly: they are produced against a suite that runs identically every time, so a patch can pass by consuming state an earlier test left in a module-level registry, a cache, or a session-scoped fixture — and never by satisfying an invariant.

The remedy is not a larger suite. Run the suite under N seeded orders, compute the coupling delta between the base commit and the patched commit, convert whatever coupling you can name into an isolation property, and freeze the remainder out of your evidence set.

Everything below is a proposal you can run as written. No timing, coverage, or pass-rate numbers are reported here, because none were measured. The seed counts are defaults to tune, not results.

1. Why one green run is a weak signal

A test suite is a program with global state. Python module objects, lru_caches, singleton clients, and class attributes all survive from one test to the next. When order never changes, that survival is invisible: the suite is deterministic, but deterministic the way a single path through a graph is deterministic.

Agent-generated patches are unusually good at finding those paths. A patch that registers a handler and relies on an earlier test having initialized the registry will pass forever on a fixed order and fail the moment it moves. Nothing in the diff announces this. The suite reports green, and the green is real — for that one order.

So stop treating the pass/fail bit as the artifact. Treat the outcome vector across seeds as the artifact.

2. Step 1: make order a parameter you can set

Do not depend on a plugin's global shuffle to be reproducible. Add the shuffle yourself, so the seed is an explicit input you can replay and pin.

# conftest.py
import os
import random

def pytest_collection_modifyitems(config, items):
    # ORDER_SEED unset -> keep the committed order (default dev loop).
    seed = os.environ.get("ORDER_SEED")
    if seed is None:
        return
    random.Random(int(seed)).shuffle(items)

This shuffles the flat item list, which crosses module boundaries. Two cautions. First, disable any other reordering plugin (-p no:randomly, -p no:xdist) for these runs, or you are measuring two shuffles at once. Second, do not vary order and PYTHONHASHSEED in the same run — if you do, an order-coupled test and a set-iteration-coupled test produce identical symptoms and you cannot attribute the flip.

3. Step 2: run the matrix and classify

Run the suite once per seed as a separate process and read the JUnit XML. One missing test case is worse than one failing test case, so count everything that is not passed as not-passed.

#!/usr/bin/env python3
# order_matrix.py
import os, subprocess, sys, json
from pathlib import Path
from xml.etree import ElementTree as ET

SEEDS = range(1, 21)
OUT = Path(".order-matrix")
OUT.mkdir(exist_ok=True)

def outcome(tc):
    if tc.find("failure") is not None or tc.find("error") is not None:
        return "failed"
    if tc.find("skipped") is not None:
        return "skipped"
    return "passed"

def run_one(seed, fixed_hash):
    xml = OUT / f"seed-{seed}.xml"
    env = dict(os.environ)
    env["ORDER_SEED"] = str(seed)
    env["PYTHONHASHSEED"] = str(fixed_hash)
    subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider",
         "-p", "no:randomly", f"--junitxml={xml}"],
        env=env, check=False,
    )
    return {f"{tc.get('classname')}::{tc.get('name')}": outcome(tc)
            for tc in ET.parse(xml).iter("testcase")}

def matrix(label, fixed_hash="0"):
    per_seed = {s: run_one(s, fixed_hash) for s in SEEDS}
    ids = sorted({tid for m in per_seed.values() for tid in m})
    return {tid: [per_seed[s].get(tid, "absent") for s in SEEDS] for tid in ids}

if __name__ == "__main__":
    label = sys.argv[1] if len(sys.argv) > 1 else "run"
    (OUT / f"matrix-{label}.json").write_text(
        json.dumps(matrix(label), indent=1)
    )

Run it twice: once at the base commit (python order_matrix.py base), once with the patch applied (python order_matrix.py patch). Classification is then mechanical.

Outcome vector	Class
passed in every seed	stable_pass
not passed in every seed	stable_fail
mixed	order_sensitive

The number that matters is not the count of order_sensitive tests. It is the set of tests that are stable_pass on base and order_sensitive on patched. That set is the coupling the patch introduced, and it is the only thing here that should block a merge.

4. How many seeds are enough

Twenty seeds will not certify order independence. With n seeds and zero failures, the rule of three puts the 95% upper bound on the per-run failure probability at roughly 3/n: about 14% at n=20, about 3% at n=100, about 1% at n=300.

That gives you a defensible way to pick n instead of guessing:

• n=20, CI budget: catches coupling that fires often. Cheap. Good for a pre-merge gate on a large suite.
• n=100, per release: a reasonable upper-bound claim of a few percent. Still misses rare coupling.
• n=300, certification of a critical path: only worth it if the suite is fast and hermetic.

Pick the bound you are willing to state, then size the matrix to it. If you cannot state a bound, you do not have a claim.

5. Step 3: turn coupling into a property

Do not fix an order-sensitive failure by pinning the order. That converts a bug into a scheduling assumption. Find what the earlier test left behind, then assert it does not leak.

# conftest.py (append)
import pytest
from app.registry import registry
from app import cache

@pytest.fixture(autouse=True)
def isolation_properties():
    registrations = len(registry)
    cached = cache.size()
    yield
    assert len(registry) == registrations, "test leaked registrations"
    assert cache.size() == cached, "test leaked cache entries"

This is the useful part of the whole workflow. A leak becomes a deterministic failure that names the leaking test, independent of order. Run the matrix again after adding the property: the test that used to be order_sensitive should now be stable_fail in every seed, and its failure message points at the state, not at the shuffle.

6. Step 4: freeze what you could not name

Some coupling is real but not yet explainable — a cross-process side effect, a file left in a non-temporary path, a fixture whose teardown is incomplete. Record it, exclude it from evidence, and give it an expiry.

{
  "test": "tests/test_billing.py::test_invoice_totals",
  "class": "order_sensitive",
  "evidence": "excluded",
  "seeds_observed": 20,
  "outcome_vector": "P P F P P P F P P P P P P F P P P P P P",
  "frozen_at": "2026-10-16",
  "freeze_until": "2026-11-15",
  "triage_issue": "ENG-4412"
}

Three rules make the freeze honest. An order-sensitive test may never be cited as proof that a patch works. The outcome vector must be stored, not the word "flaky", so a later run can be compared. The freeze expires and re-measures; a freeze that renews silently is a permanent exemption with extra steps.

7. Where MonkeyCode's free model access and free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The patch generator runs on free model access, and the matrix is a good fit for a free server option. Both claims are as supplied by the operator; no model names, quotas, or hardware are asserted here.

The fit is structural rather than promotional. Patch generation is bursty — you produce several candidate diffs, most of which will not survive the matrix. The matrix itself is an embarrassingly repeatable loop: the same suite, a different seed, one subprocess per seed, no long-lived service. That is exactly the shape a free server can absorb without you rationing seeds on a laptop.

Two constraints keep the result meaningful. Pin the lockfile and the PYTHONHASHSEED on both machines, and run the same seed range in both places. If the environment differs, you are comparing two environments, not two orders, and the flip you found on the server may never reproduce locally. Fan out by seed, but keep one failing seed's XML — it is the reproduction recipe.

8. Limitations and who should not use this

• Suites under ~20 tests. The shuffle space is too small to distinguish coupling from noise.
• Intentionally sequential flows. Migration chains, multi-step checkout, and scripted E2E journeys must run in order. Shuffling them produces failures that mean nothing and trains your team to ignore the gate.
• Cross-process state. A single-process shuffle cannot see coupling that lives in a shared database or object store. You can run each seed with a fresh namespace, but then you are also measuring environment drift and you need a baseline run to separate the two.
• Very slow suites. At n=100 with a 20-minute suite, the gate costs a day of compute. Use n=20 there and state the weaker bound.
• Anyone looking for a certification. This is a sampling method with a stated upper bound. It reduces the odds of shipping an order-dependent patch; it does not prove order independence.

9. Decision table

Observation	Meaning	Action
stable_pass in every seed	no evidence of coupling	eligible as evidence for the patch
stable_pass on base, order_sensitive on patch	the patch introduced coupling	block; add an isolation property
order_sensitive on base and on patch	pre-existing coupling	freeze with an expiry; keep out of evidence
stable_fail in every seed	stable regression	block; never freeze
test absent in some seeds	collection or import error under that order	count as not-passed; block until explained

The workflow costs one script, one fixture, and a JSON file. What it buys is a claim you can actually defend: the patch passed because it satisfied an invariant, not because the suite happened to hand it the state it needed.

If you already generate candidate patches through MonkeyCode's free model access, this is a natural second step — produce the diffs there, then let a seeded order matrix decide which ones were real.