A Gate Order for Agent Patches

A patch lands Friday at 16:40. Two files change, tests go green, review takes nine minutes. Monday the billing cron charges twice for 3% of accounts. Nobody wrote a bad test. The order in which the tests ran was wrong.

Agent-generated patches make this failure mode common. The agent iterates until the suite passes, not until the code is right. Green means "the checks I could see agreed", nothing more. So the gate has to be ordered, and each stage has to answer a different question.

Three questions, three stages

Property checks ask whether the logic survives inputs you did not write. Fixtures ask whether the world the test ran against is the world you shipped to. The flake freeze asks whether a passing test is allowed to count at all.

Run them in that order. Stop at the first failure. A patch that cannot hold an invariant has no business consuming fixture review time.

Here is a single pytest switch that enforces the order. It is small enough to read in one sitting.

# conftest.py — gate order for agent-generated patches
import hashlib, json, pathlib, time
import pytest
from hypothesis import settings as hyp_settings

ROOT = pathlib.Path(__file__).parent
GATE = json.loads((ROOT / "agent_gate.json").read_text())


def pytest_addoption(parser):
    parser.addoption("--gate", default="props",
                     choices=["props", "fixtures", "freeze"])


def pytest_configure(config):
    if config.getoption("--gate") == "props":
        hyp_settings.register_profile("gate", derandomize=True, max_examples=200)
        hyp_settings.load_profile("gate")


def pytest_sessionstart(session):
    if session.config.getoption("--gate") != "fixtures":
        return
    drift = []
    for path, want in GATE["fixtures"].items():
        p = ROOT / path
        if not p.exists() or want is None:
            drift.append(f"{path} (missing or unpinned)")
        elif hashlib.sha256(p.read_bytes()).hexdigest() != want:
            drift.append(path)
    if drift:
        pytest.exit(f"fixture gate failed: {drift}", returncode=2)


def pytest_collection_modifyitems(config, items):
    if config.getoption("--gate") != "freeze":
        return
    frozen = {f["test"]: f for f in GATE.get("frozen", [])}
    today = time.strftime("%Y-%m-%d")
    for item in items:
        entry = frozen.get(item.nodeid)
        if entry is None:
            continue
        if entry["until"] < today:
            pytest.exit(f"freeze expired on {item.nodeid}: fix it or re-file", returncode=3)
        item.add_marker(pytest.mark.skip(reason=f"frozen: {entry['reason']}"))

The config file stays plain and reviewable.

{
  "fixtures": {"fixtures/invoice_oct.json": null, "fixtures/plan_table.csv": null},
  "frozen": [
    {"test": "tests/test_sync.py::test_retry_wall_clock", "reason": "clock skew in CI", "until": "2026-11-01", "owner": "payments"}
  ]
}

null means unpinned, and the gate treats that as drift. You cannot quietly inherit a fixture hash from somebody else's laptop. Regenerate hashes in a commit that a human signs off on, never inside the agent's diff.

pytest -q --gate=props -x && \
pytest -q --gate=fixtures && \
pytest -q --gate=freeze -m "not live"

One more habit earns its cost: sweep the hash seed. Order dependence hides in dict iteration, and a single run will never show it.

for seed in 1 7 42 1009; do PYTHONHASHSEED=$seed pytest -q --gate=props || exit 1; done

The freeze is a rule, not a folder

Skipping a flaky test is cheap. Forgetting why is expensive. Every freeze row needs an owner and a date, and the date must fail loudly, not silently extend.

Here is how I would triage a red test inside an agent patch. Read the row top to bottom; the first match wins.

Symptom	Action
Fails on rerun with same seed	Block the patch. Not a flake.
Passes alone, fails in suite	Freeze 7 days, file order bug
Touches network or wall clock	Freeze 14 days, inject a fake
Only fails under parallel workers	Freeze 7 days, mark serial
Test file edited in this patch	Block. Re-review the test first
Already frozen twice	Block. Delete or fix it

The last row matters most with agents. A model will happily "fix" a flaky test by relaxing the assertion. Freeze count per test is the counter that catches that move.

Two honest limits. Demanding derandomize=True trades bug-finding breadth for reproducibility, so keep a nightly unseeded run. And hashes prove the fixture did not change, not that it was ever correct. Wrong data, pinned forever, is still wrong.

What this costs

Stage one is the slowest. Property tests at 200 examples per invariant add real minutes. If your invariants are vague, you pay that cost and learn nothing, so write the invariant first and let the agent patch second.

The freeze stage is nearly free. It runs at collection, before a single test executes. Expired rows fail in under a second, which is the right moment to interrupt a merge.

Where free capacity fits

Running four seeds across a growing suite is exactly the workload people abandon on a laptop. MonkeyCode offers free model access and a free server option, so the patch-generating side and the seed sweep can live on the same machine you did not have to provision. Current limits are documented by the product itself, and I would check them before planning a nightly sweep around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The order still matters more than the host. Models change weekly, and the gate order does not.

Who should skip this

If you are one engineer with a five-minute suite, three stages is ceremony. Run property checks, keep fixtures in git, and move on. Skip the freeze entirely when your suite has no known flakes; an empty list is honest.

Skip it too if your agent patches only touch documentation or generated assets. Hash gates on files nobody asserts about are pure noise. And if your team cannot staff an expiry owner, do not freeze at all. An unowned freeze list is a graveyard with a cron job.

Start with one invariant and one pinned fixture. Add the freeze the first time a test lies to you. Then keep the order fixed and let the suite grow around it.