Reject Agent Patches That Pass Only in Default Collection Order

An agent patch that is green under default pytest collection order has not been tested for isolation. Reject the merge if a shuffle seed fails while the default order passes. That failure is deterministic order coupling, not a flake, and it does not belong on a freeze list.

Default order measures one path: the discovery sequence the agent happened to write. It does not measure whether production code still holds after a cold import, a reversed file order, or a parallel worker. Treat collection order as a runner detail. Do not treat it as an oracle.

What default-order green actually encodes

Pytest collects test functions in a stable way when you do not shuffle: module path, then definition order. Agent sessions often emit a production hunk and a test module in one turn. The tests then assume module-level state, temp files, and environment variables left by earlier functions in that same file.

The suite goes green. The invariant is still false on a fresh process.

This is not intermittency. Re-running the same seed reproduces the red. Re-running default order reproduces the green. The gate should classify that pair as an isolation defect and fail the patch.

How an agent turn creates the coupling

Three patterns show up in review. They look like tests. They are sequenced scripts.

1. Module-level mutable stores. A registry, cache, or ledger lives at import scope. Test A writes. Test B reads. Swap them and B asserts against empty state.
2. Function-scoped fixtures that are never wired through. The agent builds a temp directory inside test A and never passes it to test B. Test B glob()s a default path and finds A's file.
3. Class-level setup_class that seeds data once. Methods are not independent. pytest-xdist then splits the class or reorders methods and the seed is missing.

None of these need a race. They need only a different collection order.

Artifact: a ledger that is green until you shuffle

The following example is a constructed reproduction, not a production incident. It is small enough to paste into a throwaway package and rerun locally.

# app/ledger.py
from __future__ import annotations

_ledger: list[tuple[str, int]] = []


def credit(account: str, amount: int) -> int:
    if amount <= 0:
        raise ValueError("amount must be positive")
    _ledger.append((account, amount))
    return balance(account)


def balance(account: str) -> int:
    return sum(value for name, value in _ledger if name == account)


def total() -> int:
    return sum(value for _, value in _ledger)


def reset() -> None:
    _ledger.clear()

An agent-style test file often looks like the next block. It passes under default collection. It is a script, not a suite.

# tests/test_ledger_agent.py
from app.ledger import credit, total


def test_credit_alice():
    assert credit("alice", 10) == 10


def test_total_after_alice():
    assert total() == 10

Commands:

pip install pytest pytest-randomly
pytest tests/test_ledger_agent.py
pytest tests/test_ledger_agent.py -q --randomly-seed=1
pytest tests/test_ledger_agent.py -q --randomly-seed=17

Default order is green because test_credit_alice is defined first. A shuffled seed may collect test_total_after_alice first. Then total() is 0 and the assert dies. Same binary. Same assertions. Different discovery order.

That pair of outcomes is the merge signal. Keep it visible. Do not quarantine the failing function.

Property checks, fixtures, and a freeze policy that does not launder order bugs

The useful strategy is not more agent-authored asserts. Split the work into three layers the agent does not own at merge time.

Property checks. State the invariant without a story. Credits are account-local. Totals equal the sum of accepted credits. Non-positive amounts raise and leave the store unchanged. Properties must hold after reset(), not after a sibling test.

Fixtures. Isolation is a fixture duty, not a comment. An autouse function-scoped reset is the minimum for a process-wide store. Session-scoped fixtures are for expensive immutable resources. They are not for seed data that later tests will mutate.

Freeze. Freeze lists exist for unreproducible failures. Order coupling reproduces with a seed. If default order is green and seed 17 is red, the action is reject and fix isolation. The action is not @pytest.mark.skip and not a freeze file.

Human-owned pack:

# tests/test_ledger_properties.py
import pytest
from app.ledger import credit, balance, total, reset


@pytest.fixture(autouse=True)
def isolate_ledger():
    reset()
    yield
    reset()


def test_credit_is_account_local():
    credit("alice", 5)
    credit("bob", 7)
    assert balance("alice") == 5
    assert balance("bob") == 7


def test_total_equals_sum_of_accepted_credits():
    credit("alice", 3)
    credit("alice", 4)
    credit("bob", 1)
    assert total() == 8


def test_non_positive_is_rejected_and_leaves_store_empty():
    with pytest.raises(ValueError):
        credit("alice", 0)
    assert total() == 0

Mark tests/test_ledger_properties.py read-only in CI. Agent patches may add files under tests/agent/. They may not edit the property pack. If the pack needs a new case, a human commits it.

Six-step gate

Run these steps on the patch, in order. Stop at the first failure.

1. Confirm the property pack is unchanged. git diff --exit-code -- tests/test_ledger_properties.py must succeed. A patch that "fixes" tests by editing the oracle is a different defect class.
2. Run the property pack in default order. pytest tests/test_ledger_properties.py -q. This is a smoke check only.
3. Run the same pack under several shuffle seeds. Invoke once per seed: 1, 7, 17, 42. Record the first red seed.
4. Run the pack under xdist if the project already depends on it. pytest tests/test_ledger_properties.py -q -n auto. Shared module state that survived shuffle still fails when workers split functions.
5. Classify. Default green plus shuffle red means isolation. Default red means a functional regression. Both green plus xdist red means a process-global leak that order did not expose. All green means the isolation check passed, not that review is done.
6. Do not write a freeze entry for steps 3 or 4. Repair reset() coverage or fixture scope. Re-run the failing seed until it is green without skips.

A small wrapper keeps the seeds explicit:

# tools/shuffle_gate.py
from __future__ import annotations

import subprocess
import sys

SEEDS = (1, 7, 17, 42)
TARGET = "tests/test_ledger_properties.py"


def run(extra: list[str]) -> int:
    cmd = [sys.executable, "-m", "pytest", TARGET, "-q", *extra]
    print(" ".join(cmd), flush=True)
    return subprocess.call(cmd)


def main() -> int:
    baseline = run([])
    if baseline != 0:
        print("property pack failed in default order")
        return baseline
    for seed in SEEDS:
        code = run([f"--randomly-seed={seed}"])
        if code != 0:
            print(f"isolation failure at seed {seed}; do not freeze")
            return code
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

python tools/shuffle_gate.py is the merge job. The agent can see the script. The agent cannot change SEEDS or TARGET if those paths are protected the same way as the property pack.

Decision table

Default order	Shuffle seed	xdist	Classification	Merge action
red	n/a	n/a	functional miss or broken property	reject; fix code or admit a human property
green	red	n/a	order coupling	reject; add reset/fixture; do not freeze
green	green	red	cross-process leak	reject; remove module globals or add worker reset
green	green	green	isolation check passed	continue to review; this gate is not sufficiency

The last row matters. Shuffle is a necessary condition for this class of bug. It is not a proof that the patch is correct.

Where a model and a clean runner belong

Shuffle tracebacks are verbose. They are also structured. A model can draft an autouse fixture or a reset() call from a failing seed. A human still has to admit that fixture into the property pack. Generated isolation code that the agent both wrote and merged is the original problem in a new file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is one way to draft those fixtures from a traceback; the free server option is one way to execute tools/shuffle_gate.py in a workspace that does not already contain the agent's __pycache__, leftover tmp directories, or exported session variables. Drafting and a clean process are inputs to the gate. They are not a substitute for a read-only property file.

If you use a model, constrain the prompt to propose a function-scoped autouse fixture that restores the module and not to edit asserts. Paste the failing seed and the module source. Discard any proposal that weakens an assertion or deletes a test.

Limitations

Shuffle does not detect tautological asserts, stubbed clocks that always match, or tests that reimplement the production function. Those need a different gate.

Some suites must be serial. Database migrations, browser sessions, and device tests can have a documented order. Mark that folder explicit and keep it out of the shuffle job. Do not silently shuffle a serial suite and then freeze the wreckage.

pytest-randomly also shuffles dictionary order and fixture order depending on version and configuration. Read the plugin's current docs before you treat a fixture-order failure as a product bug. Pin the plugin version in the gate image so seed 17 means the same permutation next month.

Parallel workers increase flake surface for true races. This method is about deterministic reordering. If a seed fails only under -n auto and never under a single worker with shuffle, investigate shared resources. Do not collapse that case into the freeze list either.

Who should not use this

Do not adopt a shuffle gate if you have no human-owned tests. Shuffling agent-authored scripts will produce noise and teach the agent to emit more reset() calls inside the same coupled file.

Do not adopt it if your policy is to freeze any red CI line to restore green. This method only works when isolation failures stay red until isolation is fixed.

Skip it for one-off notebooks and throwaway spikes. The cost is extra jobs and a protected path. That cost is for patches you intend to merge.

The core conclusion does not change with tooling. Default collection order is an accident of how files were written. Merge on properties that survive a shuffle. Leave freeze lists for failures you cannot replay with a seed.