Freeze Agent-Patch Tests Against a Fixture Digest, Not a Name

A flaky freeze is valid for one fixture digest only. If an agent patch rewrites the bytes under a test, the old skip is evidence about a different program. Name-keyed xfail lists do not record that. The merge gate should reject any freeze whose digest no longer matches.

This article is a freeze protocol, not a skip-list generator. It treats an agent patch as a hypothesis that must keep properties true on locked inputs. Timing noise is allowed a short evidence window. Fixture drift is not.

The failure mode a skip list cannot see

Agent patches often touch helpers, golden files, and “obvious” sample JSON in the same diff as the production change. A test that flickered twice last week may now fail because the fixture moved, not because the runner is noisy. Those two events look identical in a JUnit name.

They are not identical in a content hash. A freeze keyed on tests/test_apply.py::test_roundtrip will outlive the bytes that produced the flake. The next patch inherits a mute button. Reviewers then score a green suite that no longer measures the relation they think it measures.

Freeze key: three fields, no extras

Store every freeze as a triple.

1. property_id — a stable name for an invariant, never a pytest node id.
2. fixture_digest — SHA-256 of the fixture bytes the property actually read.
3. evidence_window — N independent reruns, with pass/fail counts, recorded before the freeze is legal.

If any field is missing, the record is incomplete. Incomplete records do not skip. They block, or they run.

The property catalog is the source of truth. Test function names are an implementation detail of how you invoke the catalog. When an agent renames a test, the property_id must stay put. When it rewrites a fixture, the digest must change, and every freeze that cited the old digest expires immediately.

Ledger schema (worked example)

The ledger is JSONL. One object per freeze candidate. Human-readable, diffable, easy to fail a CI step on.

{
  "property_id": "config.roundtrip.canonical_json",
  "fixture_path": "fixtures/agent_patch/sample_config.json",
  "fixture_digest": "sha256:9c1d...",
  "runs": 7,
  "passes": 5,
  "fails": 2,
  "fail_signatures": ["TimeoutError", "TimeoutError"],
  "status": "freeze_candidate",
  "reason": "intermittent_on_stable_digest"
}

status is not a vibe. It is a function of the decision table below. Do not write skip into the ledger. Skipping is a downstream action the gate may take only after status is frozen and the live digest still matches.

Decision table

Live digest vs lock	Property on majority of runs	Fail signatures	Gate action
match	holds	empty	merge-ok for this property
match	violated, same signature every run	stable	block; not a flake
match	mixed	same signature	freeze_candidate after N runs
match	mixed	many signatures	do not freeze; isolate the runner
mismatch	any	any	treat as fixture drift; drop old freeze
missing fixture	n/a	n/a	block; catalog is broken

The important row is digest mismatch. That row is how agent patches sneak past a freeze. A name-based skip never sees it.

Workflow

Run this as a pre-merge lane, not as a replacement for the full suite.

1. Catalog properties before the patch is scored. Each property is a predicate over fixtures: round-trip, idempotence, sort stability, no extra keys, bounded blast radius of a pure function. If you cannot name the property without pointing at a test file, it is not catalogued yet.
2. Lock fixtures by digest. Hash the files the property reads. Commit the lockfile. An agent patch that “cleans up” JSON whitespace is a digest change. Call it that.
3. Spend cheap isolated runs on the evidence window. N independent processes, one property, one fixture set. Do not reuse a warm worker that still has the previous patch’s modules imported.
4. Classify with the table, not with a flake plugin default. Stable violation is a failed hypothesis. Mixed results on a stable digest are the only freeze candidates.
5. Write the ledger, then freeze by key. The freeze file maps (property_id, fixture_digest) to the evidence window. Pytest node ids may appear as comments. They are not keys.
6. Re-hash on every patch. If the digest moved, delete the freeze. If the property_id moved, treat it as a new property with zero evidence.

N is a budget, not a superstition. Seven isolated runs is a starting point for a cheap lane. It is not a statistical proof. Raise N only when the fail signatures are identical and the fixture is hermetic. If signatures diverge, more runs will not create a valid freeze. They will only bury a shared-state bug.

Runnable classifier

The following is a worked example you can run locally. It is labeled as such: it does not claim production timings. It hashes a fixture, reruns a property in subprocesses, and prints a ledger line.

# freeze_ledger.py
from __future__ import annotations

import hashlib, json, subprocess, sys, textwrap
from pathlib import Path

PROPERTY_ID = "config.roundtrip.canonical_json"
FIXTURE = Path("fixtures/sample_config.json")
N = 7

PROPERTY_SRC = textwrap.dedent(
    r"""
    import json, sys
    from pathlib import Path

    raw = Path(sys.argv[1]).read_text()
    data = json.loads(raw)
    dumped = json.dumps(data, sort_keys=True, separators=(',', ':'))
    again = json.dumps(json.loads(dumped), sort_keys=True, separators=(',', ':'))
    if dumped != again:
        raise SystemExit('roundtrip_violation')
    """
)

def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def one_run(tmp: Path, fixture: Path) -> str:
    script = tmp / "property.py"
    script.write_text(PROPERTY_SRC)
    proc = subprocess.run(
        [sys.executable, str(script), str(fixture)],
        capture_output=True, text=True,
    )
    if proc.returncode == 0:
        return "pass"
    err = (proc.stderr or proc.stdout or "unknown").strip().splitlines()
    return err[-1] if err else f"exit:{proc.returncode}"

def main() -> None:
    if not FIXTURE.exists():
        raise SystemExit("missing fixture; catalog is broken")
    tmp = Path(".freeze_tmp")
    tmp.mkdir(exist_ok=True)
    results = [one_run(tmp, FIXTURE) for _ in range(N)]
    passes = results.count("pass")
    fails = [r for r in results if r != "pass"]
    sigs = sorted(set(fails))
    lock = digest(FIXTURE)
    if not fails:
        status, reason = "merge_ok", "property_holds"
    elif len(sigs) == 1 and len(fails) == N:
        status, reason = "block", "stable_violation"
    elif len(sigs) == 1:
        status, reason = "freeze_candidate", "intermittent_on_stable_digest"
    else:
        status, reason = "block", "divergent_signatures"
    rec = {
        "property_id": PROPERTY_ID,
        "fixture_path": str(FIXTURE),
        "fixture_digest": f"sha256:{lock}",
        "runs": N,
        "passes": passes,
        "fails": len(fails),
        "fail_signatures": fails,
        "status": status,
        "reason": reason,
    }
    print(json.dumps(rec, indent=2))
    if status == "block":
        raise SystemExit(2)

if __name__ == "__main__":
    main()

Seed a fixture and run it:

mkdir -p fixtures
printf '%s\n' '{"b": 2, "a": 1}' > fixtures/sample_config.json
python freeze_ledger.py

Wire the same record into CI as an artifact. A later job re-hashes fixtures/sample_config.json and refuses any freeze whose digest is absent from the new lockfile. That check is the entire protocol. Everything else is bookkeeping.

A minimal pytest hook can enforce the key without a plugin maze:

# conftest.py — proposal, not a published plugin
import hashlib, json
from pathlib import Path
import pytest

LEDGER = Path("freeze.jsonl")

def _digest(p: Path) -> str:
    return hashlib.sha256(p.read_bytes()).hexdigest()

def pytest_collection_modifyitems(config, items):
    frozen = {}
    if LEDGER.exists():
        for line in LEDGER.read_text().splitlines():
            rec = json.loads(line)
            if rec.get("status") == "frozen":
                frozen[(rec["property_id"], rec["fixture_digest"])] = rec
    for item in items:
        prop = item.get_closest_marker("property_id")
        fix = item.get_closest_marker("fixture")
        if not prop or not fix:
            continue
        path = Path(fix.args[0])
        key = (prop.args[0], f"sha256:{_digest(path)}")
        if key in frozen:
            item.add_marker(pytest.mark.skip(reason=f"frozen:{key[0]}"))

If the agent rewrites the fixture, _digest changes, key misses, and the skip evaporates. That is the intended behavior.

Where a cheap lane belongs

Evidence windows are repetitive. They should not consume the same runner pool as integration tests that hit real services. A separate worker that only re-executes catalogued properties is enough.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have MonkeyCode’s free model access and free server option, the honest split is: the free server runs the isolated reruns that fill evidence_window; a free model may only propose extra property_id names from the diff, and those names stay out of the catalog until a human accepts them. Unreviewed model output is not a lockfile. It is a draft comment.

Do not send the freeze ledger to a model and ask it whether the patch is “probably fine.” The gate reads hashes and counts. It does not read confidence prose.

Limitations

This protocol does not measure performance, network retries, or UI flake. It assumes fixtures are files (or other byte-stable blobs) and that properties are deterministic on those bytes. Shared clocks, live clocks, and unordered network mocks will produce divergent signatures. Divergent signatures are a runner bug until proven otherwise.

N independent runs are not a confidence interval. They are a budgeted filter. A property that fails 1/7 times on a stable digest might still be a real race that will hit production. Freezing it is an explicit debt entry, not a fix. Put an owner and a digest on that debt. When the fixture moves, the debt cannot quietly travel with the test name.

The classifier above uses subprocess isolation, not container isolation. If your properties load native extensions or write into a shared temp directory, raise the isolation. The ledger will not save you from a contaminated worker.

Who should not use this

Do not use a digest freeze if you have no hermetic fixtures. Generated timestamps inside JSON make every run a new digest, which makes every freeze illegal, which is correct, and also useless until you strip the noise.

Do not use it as a substitute for an oracle on security or money paths. A frozen property that says “round-trip still works” does not say “authorization still works.”

Do not use it to hide a patch that changed I/O contracts. Contract changes must add properties or change them in the same review. A freeze is for residual timing noise on an unchanged relation.

Teams that cannot run isolated subprocesses in CI will also get little from this. The evidence window is the method. A single in-process rerun is not an evidence window.

The merge question is then small. Did the digest move? Did the property hold on a budgeted set of isolated runs? If you cannot answer both from the ledger, you do not have a freeze. You have a skip list with better branding. Attach the ledger to the patch and argue from the three fields. That is the whole review.