Do Not Freeze the Test. Freeze the Seed and the Fixture Digest.

A flake freeze that records only a pytest node id is a skip list. It does not keep the oracle. When an agent patch rewrites a fixture, a generator, or an equality assertion, the same test name can pass for a different reason. Freeze the generator seed and the fixture digest. Keep the test enabled.

This protocol is for merge gates that accept machine-written patches. It is not a flake dashboard. It is a reproducibility contract: a red property check must be replayable from a seed, a digest, and one command.

The failure a skip list cannot see

Agent patches fail tests in ways that collapse in CI summaries. A property can miss an invariant. A fixture can drift because the patch rewrote the golden file. Noise can come from clock, iteration order, or I/O. A skip list labels all three as flaky and then hides the next real regression.

Equality tests are the usual casualty. An agent updates expected.json in the same diff that changes the producer. The node id stays green. The oracle is gone.

Property tests fail on a seed. That is useful. If you freeze the test name after one noisy run, you also freeze the seed that would have caught the next patch. The freeze target is wrong.

What to lock instead

Lock four fields. Not one.

1. Seed. The integer or hex that drives the generator.
2. Fixture digest. A SHA-256 of the files the test is allowed to read.
3. Reason class. One of ORDER, CLOCK, IO, HASH, SHRINK.
4. Replay command. A single shell line that must reproduce the red.

If any field is missing, the freeze is invalid. The test stays in the merge vote. Calendar expiry is optional. Digest match is not.

Proposed ledger schema

The JSON below is a proposed on-disk ledger, not a production dump. Store it next to the test tree. Do not store it in the agent write-set.

{
  "schema": "seed-freeze-v1",
  "entries": [
    {
      "nodeid": "tests/test_invoice_total.py::test_total_non_negative",
      "seed": 1847912033,
      "fixture_digest": "sha256:9c1e4b0a",
      "reason": "CLOCK",
      "replay": "pytest tests/test_invoice_total.py::test_total_non_negative --hypothesis-seed=1847912033",
      "opened_at": "2026-09-19T00:00:00Z",
      "expires_at": null,
      "status": "open"
    }
  ]
}

If the fixture digest changes, the freeze is void. The test votes again. A row without a copy-pastable replay field is not a freeze. It is a skip with extra keys.

Numbered gate workflow

Run this sequence on every agent patch. Stop at the first failed step.

1. Compute the write-set. Diff the patch against the merge base. List every path the agent touched.
2. Classify tests. Tests whose files intersect the write-set are local. The rest are witness. Local equality tests that also rewrite their own fixtures do not vote.
3. Hash fixtures. For each property test, hash the fixture directory it may read. Record the digest before the patch is applied and after.
4. Run witness properties with pinned seeds. Use a stored seed corpus. Do not draw new entropy in the merge lane.
5. On red, classify the reason. Replay once with the same seed. Green replay means noise (ORDER, CLOCK, or IO). Red replay plus a changed digest means HASH. Red replay, stable digest, and a smaller counterexample means SHRINK.
6. Write a freeze only for noise. HASH and SHRINK block the merge. ORDER / CLOCK / IO may open a ledger row that pins the seed. The test remains enabled.
7. Reject freezes without a replay command. If CI cannot paste the command, discard the row.

Artifact: seed lock and digest helper

The module below is a proposed helper. It is not a claim about a private corpus. Wire it into pytest as a plugin or a fixture.

# seed_freeze.py
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Literal

Reason = Literal["ORDER", "CLOCK", "IO", "HASH", "SHRINK"]

@dataclass(frozen=True)
class Freeze:
    nodeid: str
    seed: int
    fixture_digest: str
    reason: Reason
    replay: str
    status: str = "open"

def digest_tree(root: Path, ignore: Iterable[str] = (".pyc",)) -> str:
    h = hashlib.sha256()
    paths = sorted(p for p in root.rglob("*") if p.is_file())
    for path in paths:
        if any(path.name.endswith(suf) for suf in ignore):
            continue
        rel = path.relative_to(root).as_posix().encode()
        h.update(rel)
        h.update(b"\0")
        h.update(path.read_bytes())
        h.update(b"\0")
    return "sha256:" + h.hexdigest()

def load_ledger(path: Path) -> list[Freeze]:
    if not path.exists():
        return []
    raw = json.loads(path.read_text())
    return [Freeze(**row) for row in raw.get("entries", [])]

def freeze_is_valid(entry: Freeze, current_digest: str) -> bool:
    if entry.status != "open":
        return False
    if entry.fixture_digest != current_digest:
        return False
    if not entry.replay.startswith("pytest "):
        return False
    return True

def decide(
    seed: int,
    before: str,
    after: str,
    replay_red: bool,
    first_red: bool,
    shrunk: bool,
) -> str:
    """Return a merge action. Proposed classifier, not a trained model."""
    del seed  # seed is recorded by the caller; it does not change the class
    if after != before:
        return "BLOCK_HASH"
    if first_red and replay_red and shrunk:
        return "BLOCK_SHRINK"
    if first_red and replay_red:
        return "BLOCK_INVARIANT"
    if first_red and not replay_red:
        return "PIN_SEED_NOISE"
    return "PASS"

BLOCK_* fails the patch. PIN_SEED_NOISE writes a ledger row and leaves the test on. PASS is a no-op. The seed is an argument so callers cannot forget to persist it. It is not an input to the class.

A matching property test, labeled as an example, looks like this:

# tests/test_invoice_total.py
from decimal import Decimal
from hypothesis import given, settings, strategies as st

amounts = st.lists(
    st.decimals(min_value="0.01", max_value="10000.00", places=2),
    min_size=1,
    max_size=40,
)

@settings(deadline=None, derandomize=False)
@given(lines=amounts)
def test_total_non_negative(lines):
    total = sum((Decimal(x) for x in lines), Decimal("0"))
    assert total >= 0
    assert total == sum(lines, Decimal("0"))

Pin the seed at the command line. Do not pin it inside the test body. A seed baked into source becomes another file the agent can edit.

Commands for a single patch

# 1. Write-set versus merge base
git diff --name-only origin/main...HEAD > /tmp/write-set.txt

# 2. Fixture digest (compare two worktrees or two git trees)
python - <<'PY'
from pathlib import Path
from seed_freeze import digest_tree
print(digest_tree(Path("tests/fixtures")))
PY

# 3. Replay one property with a pinned seed (Hypothesis flag)
pytest tests/test_invoice_total.py::test_total_non_negative \
  --hypothesis-seed=1847912033 \
  --hypothesis-show-statistics

# 4. Refuse a freeze that has no replay line
python - <<'PY'
import json, sys
ledger = json.load(open("tests/seed_freeze.json"))
for row in ledger["entries"]:
    if not str(row.get("replay", "")).startswith("pytest "):
        sys.exit("invalid freeze: missing replay")
print("ledger ok")
PY

Hypothesis is an example generator. Any deterministic generator with a seed argument works. If the generator cannot accept a seed, stop. This protocol does not apply.

Keep the ledger path and the witness tests out of the write-set. A one-line check:

if grep -F 'tests/seed_freeze.json' /tmp/write-set.txt; then
  echo "agent edited the freeze ledger" >&2
  exit 1
fi

Decision table

first run	replay, same seed	fixture digest	shrink found smaller	action
green	n/a	unchanged	n/a	PASS
red	green	unchanged	n/a	PIN_SEED_NOISE
red	red	changed	n/a	BLOCK_HASH
red	red	unchanged	yes	BLOCK_SHRINK
red	red	unchanged	no	BLOCK_INVARIANT

HASH means the patch mutated the oracle's inputs. SHRINK means the generator found a smaller failing example on a stable fixture. Those are different merge blocks. Treating both as flake produces the skip list this protocol exists to avoid.

PIN_SEED_NOISE can hide a real race. Re-run the pinned seed on a second worker. If it is red there, promote the action to BLOCK_INVARIANT.

HASH versus SHRINK, in one paragraph

A changed digest with a red replay is a fixture problem. The producer and the golden file moved together, or the patch rewrote data the property reads. A stable digest with a smaller counterexample is a logic problem. The code under test still violates the invariant, and the generator proved it with less input. Freezing either case by node id deletes the evidence. Recording the seed keeps it.

Where model calls and test replay split

Generating patches and replaying seeds is cheap in CPU and expensive in attention. A merge lane that draws a new model call on every retry will also draw new entropy into the tests. Keep model access and test execution on separate leases.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option can host the replay commands above without folding the generator into the same process that mutates fixtures. That split is the point. The model proposes a patch. The server runs decide() against a seed corpus the model cannot edit. If you already have that split locally, you do not need another host. If you do not, a free server is one place to keep the ledger out of the agent's write-set.

Limitations

The protocol assumes the property test has a seed. Snapshot tests, UI pixel diffs, and live-network checks do not. Hashing a fixture tree does not detect semantic drift inside a binary the tests never read. An agent can still add a tautology in a new file that is not in the witness set.

The classifier in decide() is a table. It will not estimate a p-value. Teams that need probabilistic flake scoring should keep that scorer outside the freeze ledger. Mixing a skip probability with a seed lock recreates the skip list under a different name.

Voided freezes need an owner. If nobody inspects digest mismatches, the ledger becomes a graveyard of open rows that never vote.

Who should not use this

Do not use seed freezes if your tests have no generator. Equality-only suites need a different demotion rule: tests that rewrite their own expected files should lose their merge vote, not gain a freeze row.

Do not use this as a way to keep a red suite green. A ledger that grows faster than the seed corpus is a skip list with extra fields.

Do not apply it to tests the agent is allowed to edit. Witness properties, fixture trees used for digests, and tests/seed_freeze.json must sit outside the patch write-set.

What to count instead of freeze count

Count voided freezes (digest mismatch) and BLOCK_SHRINK hits per 100 patches. Those two numbers say whether the oracle is alive. Freeze count alone does not.

If voided freezes rise, the agent is touching fixtures. Move those files out of the write-set. If BLOCK_SHRINK stays at zero across a long window, the seed corpus is stale. Add seeds from new counterexamples. Do not add skips.