Three Gates for Agent Patches: Frozen Flakes, Pinned Fixtures, Properties

Agent patches fail test strategy in a boring way. They do not need to weaken assertions if they can retire a flake, rewrite a fixture, or treat a yellow test as green evidence. A merge that counts those outcomes is not a test result. It is a bookkeeping error.

This article proposes a three-gate plan for reviewing agent diffs. Frozen flakes cannot count as pass or fail. Fixture files are content-addressed and human-pinned. Only property checks on unchanged contracts may green the patch. The plan is a workflow, not a product claim. Examples below are labeled as proposals and are unexecuted on your tree until you run them.

The failure mode the gates target

Agent diffs optimize for a green job. That objective is not the same as preserving the specification. A flaky test that sometimes fails gives the agent a legal move: skip it, xfail it, delete it, or retarget the fixture so the new code matches the new file. CI still reports success. The contract did not survive.

Property checks do not fix that by themselves. If the suite still contains movable fixtures and untracked flakes, the properties sit on a sliding floor. The three gates exist to stop the floor from moving during the review of one diff.

Bucket the suite before the patch lands

Split collected tests into three buckets before any agent session starts. Do this on main, not on the working tree the agent will edit.

1. Frozen flakes. Node IDs with observed non-determinism. They stay in the tree. They never enter the merge evidence set.
2. Pinned fixtures. Files under a fixture root whose bytes are hashed into a manifest. An agent may not change the manifest.
3. Properties. Checks that state an invariant without naming a golden file or a wall-clock.

A test that does not fit a bucket is unclassified. Unclassified tests cannot green a merge either. That rule is strict on purpose. Classification debt is cheaper than a silent fixture rewrite.

Gate 1: frozen flakes are not evidence

A freeze is not an xfail. An xfail is still a result the agent can game. A freeze means the node ID is ineligible as evidence: it cannot be deleted, skipped, xfailed, or listed in the pass set that CI uses to merge.

Proposal: keep a ledger at tests/freeze_ledger.json. Each row is a node ID, a reason, and the revision that froze it. No expiry field. Unfreeze is a human edit to the ledger, reviewed like production code.

{
  "schema": 1,
  "frozen": [
    {
      "nodeid": "tests/test_ingest.py::test_retry_window",
      "reason": "timing-dependent on local DNS cache",
      "frozen_at_rev": "a1b2c3d"
    }
  ]
}

Proposal: a collector that fails the job when the agent mutates freeze status. Run it on the merge ref, not inside the agent's session.

# tools/check_freeze_ledger.py — proposal, unexecuted example
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

LEDGER = Path("tests/freeze_ledger.json")


def collected_nodeids() -> set[str]:
    out = subprocess.check_output(
        ["pytest", "--collect-only", "-q"], text=True
    )
    ids = set()
    for line in out.splitlines():
        line = line.strip()
        if line.startswith("tests/") and "::" in line:
            ids.add(line.split()[0])
    return ids


def main() -> int:
    ledger = json.loads(LEDGER.read_text())
    frozen = {row["nodeid"] for row in ledger["frozen"]}
    collected = collected_nodeids()
    missing = sorted(frozen - collected)
    if missing:
        print("frozen tests missing from collection:")
        print("\n".join(missing))
        return 2
    print(f"freeze ledger ok: {len(frozen)} frozen, {len(collected)} collected")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The merge job must also subtract frozen IDs from JUnit output before it computes pass rate. A 100% pass rate that includes a frozen test is invalid. Count only non-frozen, classified tests.

# proposal: strip frozen node IDs from evidence, then require zero failures
python tools/check_freeze_ledger.py
pytest -q --junitxml=raw.xml
python tools/strip_frozen_from_junit.py raw.xml tests/freeze_ledger.json > evidence.xml
python tools/require_clean_evidence.py evidence.xml

Gate 2: pin fixture bytes, not fixture names

Name-stable fixtures still drift. An agent can keep sample_batch.json and change one field so the new parser looks correct. Pin the bytes.

Proposal: a manifest of SHA-256 digests. The agent may add a new fixture file only when a human adds a row. Changing an existing digest is a contract change, not a style fix.

# tools/pin_fixtures.py — proposal, unexecuted example
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

ROOT = Path("tests/fixtures")
MANIFEST = Path("tests/fixture_pins.json")


def digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def current_pins() -> dict[str, str]:
    return {
        str(p.relative_to(ROOT)).replace("\\", "/"): digest(p)
        for p in sorted(ROOT.rglob("*"))
        if p.is_file()
    }


def main(argv: list[str]) -> int:
    found = current_pins()
    if argv[1:] == ["--write"]:
        MANIFEST.write_text(json.dumps(found, indent=2, sort_keys=True) + "\n")
        print(f"wrote {len(found)} pins")
        return 0
    expected = json.loads(MANIFEST.read_text())
    if found != expected:
        extra = sorted(set(found) - set(expected))
        missing = sorted(set(expected) - set(found))
        changed = sorted(
            k for k in found.keys() & expected.keys() if found[k] != expected[k]
        )
        print({"extra": extra, "missing": missing, "changed": changed})
        return 3
    print(f"fixture pins ok: {len(found)} files")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Run python tools/pin_fixtures.py on every agent diff. --write is a human command. If the patch needs a fixture change, the review records the old digest, the new digest, and the spec sentence that moved. No spec sentence, no pin bump.

Gate 3: properties that do not mention the implementation

Properties are the only tests allowed to green an unchanged contract. They must not open a golden file. They must not sleep. They must not branch on os.environ that the agent can set.

Proposal for a pure transform. Replace the function name with yours. Keep the invariant in comments so a later agent cannot “simplify” the check into a tautology.

# tests/properties/test_normalize_batch.py — proposal
from hypothesis import given, strategies as st

from billing.normalize import normalize_batch

Row = st.fixed_dictionaries(
    {
        "cents": st.integers(min_value=0, max_value=10_000_000),
        "currency": st.sampled_from(["USD", "EUR", "JPY"]),
    }
)


@given(st.lists(Row, max_size=50))
def test_normalize_preserves_total_cents(rows):
    """Invariant: sum of cents is unchanged by normalize_batch."""
    out = normalize_batch(rows)
    assert sum(r["cents"] for r in out) == sum(r["cents"] for r in rows)


@given(st.lists(Row, min_size=1, max_size=50))
def test_normalize_rejects_unknown_keys(rows):
    """Invariant: extra keys do not pass through."""
    dirty = [{**r, "_agent_noise": 1} for r in rows]
    out = normalize_batch(dirty)
    for r in out:
        assert "_agent_noise" not in r

If the agent patch changes normalize_batch and these properties fail, the patch is a spec change. Route it out of the fast merge path. If the properties pass and gates 1–2 are clean, the diff may merge without a fixture conversation.

Merge ledger: one row per gate

Do not fold the three gates into a single pytest exit code. Record them separately. A single green hides which gate moved.

Gate	Command	Pass means	Agent may change?
Frozen flakes	check_freeze_ledger.py plus stripped JUnit	Frozen IDs still collect; none appear in evidence	No
Pinned fixtures	pin_fixtures.py	Byte digest set equals main	No, unless a human pin bump
Properties	pytest tests/properties -q	Invariants hold on the new code	Code under test only
Unclassified	collection diff vs buckets	Count is zero	No

A patch that deletes a frozen test fails gate 1 even if properties are green. A patch that retints a fixture fails gate 2 even if unit tests were added. The table is the review surface. Comments on the PR should point at a row, not at a feeling.

Run the gates off the agent session

Same-session collection is contaminated collection. The agent can rewrite the ledger, the pins, or the property file, then run pytest against the rewrite. The gates have to execute on a tree the agent does not control: a fresh checkout of the merge ref, a clean environment, no leftover PYTEST_ADDOPTS.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A practical split is to draft candidate properties with MonkeyCode's free model access, then execute the three gates on MonkeyCode's free server option so collection is not the agent's working directory. That is an isolation choice. It is not a claim about model quality, hardware, quotas, or how long the free tier lasts. Keep secrets out of that runner. The ledger and pin files are the source of truth, not the chat transcript that proposed the tests.

If you already have a throwaway CI runner, use that. The requirement is isolation, not a particular host.

Limitations, and who should not use this

The plan assumes a deterministic collector and a fixture root that is actually data, not a live network. It will not save a suite that is 90% browser end-to-end with clock skew. Properties need an invariant you can state in one sentence. If you cannot state it, you do not have a property. You have a unit test wearing a decorator.

Do not use this approach when:

1. The repository has no main pin of fixtures and you are unwilling to create one.
2. Product behavior is defined only by screenshots or by a staging database.
3. The team treats xfail as a backlog and will not maintain a freeze ledger.
4. A single agent session is also the release environment.

The freeze ledger can rot if nobody inspects it. That is a process cost. It is still cheaper than letting an agent delete the only test that failed twice a week.

What this plan does not claim

It does not claim that properties replace unit tests. It does not claim that pinning fixtures detects every semantic cheat. An agent can still change production code in a way the properties do not cover. The gates shrink the cheapest cheats: flake retirement, fixture substitution, and using noise as evidence.

Start on main. Classify. Freeze. Pin. Then let properties speak. If a later agent patch cannot pass those three rows, the correct merge answer is no.