Don't Skip Flakes After an Agent Patch. Demote Them.

Agent patches do not only change production code. They also change the conditions under which tests pass. A skip list records the symptom and deletes the signal. The usable rule is narrower: demote a flake to an observe lane only after a shuffle-and-clock rerun produces a stable failure signature, lock the fixture bytes that the test read, and keep one property check in the blocking lane.

That split is the whole strategy. Properties stay red-or-green. Fixtures stay content-addressed. Flakes become ledger rows, not pytest.mark.skip.

Why skip lists fail after an agent patch

A human patch is usually local. An agent patch is often a bundle: a rename, a new helper, a rewritten fixture, and two tests that assert the happy path. The suite still reports green. Two days later a test fails under a different order, or under a clock that crossed midnight UTC.

Skip then looks cheap. It is not. You have removed the only test that observed the coupling the patch introduced. The next agent sees a greener suite and a larger blind spot.

Order coupling, time coupling, and fixture drift are distinct faults. Treat them as distinct, or the freeze file becomes a junk drawer.

Three lanes, not one status

Give every test exactly one lane.

1. Blocking property. A pure invariant. It does not read the patch diff, the network, or the wall clock.
2. Locked fixture. Byte-hashed inputs. The test may be example-based, but the bytes it consumes are pinned.
3. Observe flake. Allowed to fail the job's flake section. Forbidden from skipping. Forbidden from silencing the property in lane 1.

A test can move from lane 2 to lane 3. It cannot move from lane 1 to lane 3. If the property is noisy, the property is wrong. Fix the property. Do not demote it.

Layer 1 — Keep a property blocking after demotion

Write properties against the public contract, not against the files the agent touched. The check should remain valid if the entire patch is reverted. Demotion never applies here. The example test may move to the ledger. The property stays in the blocking job.

# tests/properties/test_invoice_total.py
from decimal import Decimal
from hypothesis import given, settings, strategies as st

from billing.totals import invoice_total

Money = st.decimals(min_value="0.00", max_value="1000000.00", places=2)
Line = st.tuples(Money, st.integers(min_value=1, max_value=99))

@given(st.lists(Line, min_size=0, max_size=40))
@settings(max_examples=80, deadline=None)
def test_invoice_total_is_non_negative_and_additive(lines):
    items = [{"unit": u, "qty": q} for (u, q) in lines]
    total = invoice_total(items)
    assert total >= Decimal("0.00")
    assert total == sum((u * q for u, q in lines), Decimal("0.00"))

Label this as a proposed check if you have not yet executed it against your domain. The shape matters more than the library: no I/O, no datetime.now(), no fixture file the agent can rewrite.

If an agent "fixes" a failing property by editing the property, the job must fail. Keep property files on a path the patch reviewer treats as sacred. Hash them in CI. Reject diffs that touch both src/ and tests/properties/ in one commit. That rule is not an oracle for the patch. It is how lane 1 survives a demotion in lane 3.

Layer 2 — Fixture lockfile

Example tests are fine. Unpinned example files are not. Hash the fixture at collection time. Fail if the bytes moved and the lockfile did not.

# tests/support/fixture_lock.py
from pathlib import Path
import hashlib
import json

LOCK = Path("tests/support/fixture.lock.json")

def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def assert_fixtures_locked(root: Path) -> None:
    expected = json.loads(LOCK.read_text())
    actual = {
        str(p.relative_to(root)): sha256(p)
        for p in sorted(root.joinpath("tests/fixtures").rglob("*"))
        if p.is_file()
    }
    missing = sorted(set(expected) - set(actual))
    extra = sorted(set(actual) - set(expected))
    changed = sorted(
        k for k in expected.keys() & actual.keys() if expected[k] != actual[k]
    )
    assert not missing, f"fixture lock missing files: {missing}"
    assert not extra, f"fixture lock extra files: {extra}"
    assert not changed, f"fixture lock hash drift: {changed}"

The agent may add a fixture. That is a lockfile change, not a silent side effect of a code change. Reviewers then see "bytes changed" instead of "tests still pass."

python -c "from pathlib import Path; from tests.support.fixture_lock import assert_fixtures_locked; assert_fixtures_locked(Path('.'))"

Regenerate the lock only with an explicit flag. Never from the patch job.

python scripts/rewrite_fixture_lock.py --allow-rewrite

Layer 3 — A flake ledger, not a skip mark

Pytest skip is a deletion with a comment. Replace it with an append-only ledger. A row is legal only when a two-run protocol agrees on the failure signature.

{
  "test_id": "tests/test_report.py::test_renders_end_of_day",
  "first_seen": "2026-09-04T00:00:00Z",
  "signature": "AssertionError: 23:59 vs 00:00",
  "axes": ["clock", "order"],
  "runs": 2,
  "property_still_blocking": "tests/properties/test_invoice_total.py::test_invoice_total_is_non_negative_and_additive",
  "expires_on_property_change": true,
  "skip": false
}

skip is always false. The observe job records the row. The blocking job ignores the example test and still runs the named property. Calendar expiry is the wrong trigger. The row dies when the named property changes, because that is the remaining signal.

Two-run protocol

Number the steps. Do not collapse them into "rerun once."

1. Pin the clock to a fixed ISO timestamp and a fixed timezone.
2. Pin the RNG seed used by the test runner.
3. Run the suite in collected order. Write run-a.json.
4. Run the suite shuffled with the same seed's sibling, for example seed ^ 0xA5A5. Write run-b.json.
5. Diff failure signatures, not traces. Traces include line numbers the agent will churn.
6. If both runs fail with one signature, and the property in lane 1 still passes, append a ledger row.
7. If the runs disagree, do not demote. The patch introduced order or clock coupling. Reject the patch.

Proposed harness (unexecuted template):

# scripts/two_run_gate.py
import json, os, subprocess, sys
from pathlib import Path

CLOCK = os.environ.get("PATCH_CLOCK", "2026-09-04T12:00:00+00:00")
SEED_A = int(os.environ.get("PATCH_SEED", "20260904"))
SEED_B = SEED_A ^ 0xA5A5

def run(seed: int, out: Path) -> dict:
    env = os.environ.copy()
    env["PATCH_CLOCK"] = CLOCK
    env["PYTHONHASHSEED"] = "0"
    cmd = [
        sys.executable, "-m", "pytest",
        "-q", "--seed", str(seed), "--random-order",
        "--json-report", f"--json-report-file={out}",
    ]
    subprocess.run(cmd, env=env, check=False)
    return json.loads(out.read_text())

def signatures(report: dict) -> set[str]:
    fails = []
    for t in report.get("tests", []):
        if t.get("outcome") != "failed":
            continue
        node = t.get("nodeid", "")
        msg = (t.get("call") or {}).get("longrepr", "")
        head = msg.splitlines()[0] if msg else "failed"
        fails.append(f"{node}::{head[:120]}")
    return set(fails)

def main() -> int:
    a = run(SEED_A, Path("run-a.json"))
    b = run(SEED_B, Path("run-b.json"))
    sa, sb = signatures(a), signatures(b)
    only_a, only_b, both = sa - sb, sb - sa, sa & sb
    Path("gate-summary.json").write_text(json.dumps({
        "clock": CLOCK,
        "only_a": sorted(only_a),
        "only_b": sorted(only_b),
        "both": sorted(both),
        "decision": "reject_patch" if (only_a or only_b) else (
            "demote_candidates" if both else "pass"
        ),
    }, indent=2))
    if only_a or only_b:
        print("order/clock coupling; refuse demotion")
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

--random-order and --json-report assume pytest plugins you already trust. If you do not have them, shuffle pytest --collect-only -q yourself and pass a nodeid file. The protocol is the artifact. The flags are incidental.

Clock injection belongs in one place:

# tests/conftest.py
import os
from datetime import datetime, timezone
import pytest

@pytest.fixture(autouse=True)
def pinned_clock(monkeypatch):
    raw = os.environ.get("PATCH_CLOCK")
    if not raw:
        yield
        return
    frozen = datetime.fromisoformat(raw)

    class _FrozenDateTime(datetime):
        @classmethod
        def now(cls, tz=None):
            return frozen if tz is None else frozen.astimezone(tz)

        @classmethod
        def utcnow(cls):
            return frozen.astimezone(timezone.utc).replace(tzinfo=None)

    monkeypatch.setattr("datetime.datetime", _FrozenDateTime)
    yield

That fixture is a proposal. Patching datetime.datetime does not catch time.time(). Extend the pin if your code uses time, arrow, or pendulum. A partial pin produces false demotions.

Decision table

Observation after two runs	Fixture lock	Property	Action
Both green	unchanged	pass	Accept patch
Both green	drifted	pass	Reject; fixture change required
Same failure signature	unchanged	pass	Demote example test; keep property
Same failure signature	unchanged	fail	Reject patch; property is still blocking
Different signatures	any	any	Reject; coupling, not a flake
Either run errors on collection	drifted	any	Reject; agent edited collection
Property touched in the same diff as src/	any	any	Reject; blocking lane moved with the code

The table is the policy. Encode it in the gate, not in a wiki.

Where a free model and a free server fit

Generating the patch and scoring the patch are different jobs. A free model is enough to propose a diff. It is not enough to decide that the diff is safe. The decision is the two-run gate plus the property that the model is not allowed to edit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option are relevant here only as a place to run that split: the model writes a branch, the server runs two_run_gate.py against a checkout that already contains the property suite and the fixture lock. If the gate returns 2, discard the branch. If it returns 0 and gate-summary.json lists both failures, write ledger rows. Do not merge until a reviewer confirms the named property still lives in the blocking job.

Do not treat a green free-server job as capacity evidence. It is a hermetic rerun, not a load test. No quota, hardware, or duration claims are required for the workflow to work. Any runner that can pin PYTHONHASHSEED, inject PATCH_CLOCK, and refuse diffs that touch tests/properties/ will do.

Limitations

Hypothesis examples are not a proof. Eighty draws will miss a domain hole. The fixture lock cannot see data loaded from object storage at runtime. The datetime monkeypatch will not freeze SQL NOW(). Shuffle will not catch races that need two threads.

The ledger can rot. If expires_on_property_change is true, any edit to the blocking property invalidates every demotion that named it. That is intentional. It is also noisy. Budget time for ledger hygiene or the observe lane becomes a skip list with extra JSON.

Signatures that include rendered timestamps will never match across runs even when the fault is identical. Strip time and line numbers before hashing a signature.

Who should not use this

Do not install this gate on a codebase whose product is non-determinism: simulations without seeds, games, Monte Carlo without a captured RNG. You will demote real behavior.

Do not use it as a substitute for code review on security or payments. A property about non-negative totals does not prove authorization.

Do not run it if you cannot keep property files out of the agent's write set. The moment the model can edit the blocking property and the code in one commit, the three lanes collapse into one.

Small libraries with twenty deterministic tests gain little. Run the tests twice. Skip the ledger.

Wire-up order

1. Add tests/properties/ and forbid combined diffs with src/.
2. Generate fixture.lock.json once, by hand.
3. Add pinned_clock and the two-run script.
4. Fail the build on only_a / only_b.
5. Allow demotion only from both, with skip set to false.
6. Point the observe job at the ledger. Point the blocking job at properties plus locked fixtures.

The core conclusion does not change after that list. Skipping a flake after an agent patch hides coupling. Demote the example. Lock the bytes. Leave one property in the blocking lane.

If you already generate patches with a free model on a free server, attach the ledger and the two-run gate to that job before you add more eval dashboards. The cheaper run is the one that refuses a coupled patch.