Name-Keyed Flake Freezes Fail Agent Patches. Freeze the Signature Instead.

A freeze keyed to a pytest nodeid is not a freeze. Agent patches rename tests, split files, and rewrite assertion messages, so the next run treats the same failure as new. Bind the freeze to a normalized failure signature, lock fixtures by content hash, and keep property checks independent of test names. That three-part ledger is the gate. The rest of this article is the procedure, not a scoreboard.

This workflow is a proposed harness. Examples below are labeled and unexecuted. Do not treat the sample hashes, TTLs, or counts as production measurements.

Why name keys collapse

Agent-authored patches often “clean up” names. They merge modules. They turn assert total == 19 into assert total == 19, "tax included". If quarantine stores tests/test_bill.py::test_total, every one of those edits lifts the freeze. The flake returns as a green merge.

Line numbers are equally unstable. An agent that inserts a comment shifts every frame. File paths move when the patch “reorganizes” tests. A durable key has to ignore those surfaces and keep the parts that actually identify the defect.

What to bind instead

Use three oracles that do not share a name space.

1. Property checks keyed by invariant id. They must fail on at least one generated input when the patch is wrong.
2. Fixtures keyed by content hash. Bytes change, the lock changes. Review is required.
3. Flake freezes keyed by failure signature. Exception type, assertion kind, and function-level frames. Not the test name.

If a layer needs the test title to work, it is the wrong layer.

Layer 1: property checks with an invariant id

Property checks survive renames because they never look up nodeid. They look up a stable invariant, then sample inputs. A check that only exercises the happy-path literal the agent just wrote is a fixture in disguise. Label it as such and move it to Layer 2.

Proposed steps:

1. Give every invariant a string id that is not derived from a test function name.
2. Draw a small, explicit input set. Include at least one value the current patch does not hard-code.
3. Fail closed if the invariant is not registered. Silent skip is a pass.
4. Record (invariant_id, input_digest, pass|fail). Do not record the pytest node id as the primary key.

# example: proposed invariant runner, unexecuted
from dataclasses import dataclass
from hashlib import sha256
import json

INVARIANTS = {}

def invariant(invariant_id):
    def wrap(fn):
        INVARIANTS[invariant_id] = fn
        return fn
    return wrap

@invariant("bill.total_non_negative")
def total_non_negative(subtotal, tax):
    # Property: totals do not go negative for non-negative inputs.
    from bill import total
    if subtotal < 0 or tax < 0:
        return True
    return total(subtotal, tax) >= 0

SAMPLES = {
    "bill.total_non_negative": [
        {"subtotal": 0, "tax": 0},
        {"subtotal": 19, "tax": 0},
        {"subtotal": 19, "tax": 2},
        {"subtotal": 10**6, "tax": 1},
    ]
}

def run_properties():
    rows = []
    for iid, fn in INVARIANTS.items():
        cases = SAMPLES.get(iid)
        if not cases:
            raise SystemExit(f"unregistered samples for {iid}")
        for case in cases:
            digest = sha256(json.dumps(case, sort_keys=True).encode()).hexdigest()[:12]
            ok = bool(fn(**case))
            rows.append({"invariant": iid, "input": digest, "ok": ok})
            if not ok:
                raise AssertionError(f"{iid} failed on {case}")
    return rows

The sample set is tiny on purpose. Four cases will not replace a fuzzer. They will catch the common agent failure: an invariant that only holds for the literal in the patch description. If you later add a generator, keep the invariant id as the key. Do not let the generator invent test names that become freeze keys.

Layer 2: content-addressed fixtures

Fixtures are allowed. They are not properties. Treat them as blobs with hashes. When an agent patch rewrites a golden file, the hash changes and the lock must trip. That trip is a review event, not an automatic pass.

Proposed steps:

1. Put golden files in a dedicated directory. Do not mix them with source.
2. Hash canonical bytes. Strip nothing except a trailing newline policy you document.
3. Store {path, sha256, bytes} in the ledger. Path is a locator, not the identity.
4. On each run, rehash. A mismatch voids any freeze that listed that path.

# example: fixture lock, unexecuted
from pathlib import Path
from hashlib import sha256
import json

FIXTURE_ROOT = Path("fixtures")

def hash_bytes(data: bytes) -> str:
    return sha256(data).hexdigest()

def snapshot_fixtures():
    rows = []
    for path in sorted(FIXTURE_ROOT.rglob("*")):
        if path.is_file():
            data = path.read_bytes()
            rows.append({
                "path": str(path.as_posix()),
                "sha256": hash_bytes(data),
                "n": len(data),
            })
    return rows

def assert_fixture_lock(expected_path="ledger/fixtures.json"):
    expected = json.loads(Path(expected_path).read_text())
    actual = {r["path"]: r["sha256"] for r in snapshot_fixtures()}
    locked = {r["path"]: r["sha256"] for r in expected}
    added = sorted(set(actual) - set(locked))
    removed = sorted(set(locked) - set(actual))
    changed = sorted(p for p in actual.keys() & locked.keys() if actual[p] != locked[p])
    if added or removed or changed:
        raise SystemExit({
            "added": added,
            "removed": removed,
            "changed": changed,
        })

A useful extra check: reject a patch that changes fixture hashes and production code in the same commit unless a reviewer allow-list is present. Agents often “fix” the oracle. The lock does not decide intent. It only makes the edit visible.

Layer 3: freeze the signature, not the node id

This is the layer that dies first in name-keyed systems. Build a signature from fields the agent is unlikely to use as a cleanup target, then store a freeze against that signature. Test names may appear as metadata. They must not be the key.

Include:

• exception class name
• assertion kind (equal, raises, approx, unknown)
• function names from the top frames inside project code
• a normalized message: collapse whitespace, drop digits that look like timestamps or memory addresses

Exclude:

• absolute paths
• line numbers
• pytest node ids
• wall-clock timestamps
• randomized temp directory names

# example: signature freeze ledger, unexecuted
import hashlib, json, re, time
from pathlib import Path

LEDGER = Path("ledger/freezes.json")
PROJECT_MARKERS = ("bill.py", "tax.py", "parse.py")

ADDR = re.compile(r"0x[0-9a-fA-F]+")
DIGITS = re.compile(r"\b\d+\b")
SPACE = re.compile(r"\s+")

def normalize_message(msg: str) -> str:
    msg = ADDR.sub("<addr>", msg or "")
    msg = DIGITS.sub("<n>", msg)
    return SPACE.sub(" ", msg).strip().lower()

def assertion_kind(msg: str) -> str:
    m = (msg or "").lower()
    if "not equal" in m or "==" in m:
        return "equal"
    if "did not raise" in m or "raises" in m:
        return "raises"
    if "approx" in m:
        return "approx"
    return "unknown"

def project_functions(tb_text: str):
    names = []
    for line in (tb_text or "").splitlines():
        if "File " not in line:
            continue
        if not any(m in line for m in PROJECT_MARKERS):
            continue
        # keep the next 'in <func>' if present; ignore line numbers
        if ", in " in line:
            names.append(line.split(", in ", 1)[1].strip())
    return tuple(names[:6])

def signature(exc_class: str, message: str, traceback_text: str) -> str:
    payload = json.dumps({
        "exc": exc_class,
        "kind": assertion_kind(message),
        "msg": normalize_message(message),
        "fn": project_functions(traceback_text),
    }, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

def load_ledger():
    if not LEDGER.exists():
        return {"freezes": []}
    return json.loads(LEDGER.read_text())

def freeze(sig: str, reason: str, ttl_s: int, now=None):
    now = int(now or time.time())
    data = load_ledger()
    data["freezes"] = [f for f in data["freezes"] if f["sig"] != sig]
    data["freezes"].append({
        "sig": sig,
        "reason": reason,
        "until": now + ttl_s,
        "issued": now,
    })
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    LEDGER.write_text(json.dumps(data, indent=2))

def freeze_allows(sig: str, now=None) -> bool:
    now = int(now or time.time())
    for row in load_ledger().get("freezes", []):
        if row["sig"] == sig and row["until"] >= now:
            return True
    return False

TTL is a review budget, not a quality score. Example value: 3 days of wall time, stored as seconds. When it expires, the same signature must fail the gate again. Do not auto-renew because the test was renamed. Renewal is a human write to the ledger.

One gate, four exits

Wire the layers in a single command so an agent cannot pass by skipping a file.

1. Run property checks. Any invariant miss is a hard fail. No freeze applies.
2. Rehash fixtures. Added, removed, or changed blobs fail unless a reviewer file lists those paths.
3. Run the rest of the suite. For each failure, compute a signature.
4. If freeze_allows(sig), record a freeze hit and continue. If not, fail the gate. If the failure is new, do not invent a freeze from the agent patch itself.

# example commands, unexecuted
python -m tools.run_properties
python -m tools.assert_fixture_lock
pytest -q --tb=short | python -m tools.apply_signature_gate

Agents should not be able to append to ledger/freezes.json in the same patch that introduces the failure. A separate allow-step keeps the freeze file append-only from humans. If your review tool cannot enforce that split, the ledger is documentary only. Do not call it a gate.

Decision table

Observation	Layer	Gate action
Wrong result on a sampled input	Property	Fail. No freeze.
Golden bytes changed with the patch	Fixture lock	Fail until review lists the path.
Same exception, kind, and functions; intermittent	Signature freeze	Allow only with an unexpired ledger row.
Intermittent, but function frames change every run	None	Fail. Unpinned entropy, not a freeze candidate.
Test renamed, signature unchanged	Signature freeze	Freeze still binds.
Test renamed, assertion kind changed	Signature freeze	Treat as a new failure.

Unpinned entropy is the usual reason frames change: clocks, unordered dict dumps, network, filesystem directory order. Freeze will not help. Pin the source or delete the test.

Where a remote runner fits

The ledger is just files and a Python process. It can run on a laptop. It can run on CI. The useful split is authorship versus observation: generate candidate invariant samples in one place, execute the gate in another, and compare the ledger diff as the review artifact.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft extra sample rows for an invariant id you already defined. The free server option can run the same three commands above so the ledger is produced off your laptop. Neither one should be allowed to write ledger/freezes.json. That file stays human-authored.

If you already have CI, you do not need a second runner. Use what you have. The method does not depend on a vendor.

Limitations

Signature collision is possible. Short hashes and aggressive digit stripping can merge distinct bugs. If two failures share a signature, the freeze will cover both. Lengthen the hash and keep function names. Do not put messages that contain business amounts into the identity unless you want amount changes to look like new bugs.

Project-frame filtering can be too tight. A failure that never enters PROJECT_MARKERS will hash to an empty function list. That collapses many infrastructure flakes into one signature. Expand the marker list, or fail open to “no freeze eligible” when fn is empty.

Properties with tiny sample sets miss narrow regressions. That is expected. The point of Layer 1 is rename-stable checking, not coverage. Pair it with whatever unit tests you already trust.

Fixture hashing does not detect semantic drift in binary assets you cannot review. Images and protobufs need a different oracle. Do not pretend a sha256 of a PNG is an invariant.

Who should not use this

Do not use a signature freeze on security or correctness paths where an intermittent failure is the incident. A race in auth code is not a flake budget. Fix it or block the merge.

Do not use this if no human owns ledger/freezes.json. An agent that can freeze its own failures will freeze them all.

Do not use this as a substitute for pinning time, locale, and RNG. A freeze on unpinned entropy trains the suite to ignore the real bug.

Teams with fewer than a handful of agent-generated tests per week will spend more time maintaining the ledger than they save. A plain pytest rerun with a name-stable suite is enough there.

The core conclusion does not change with scale. If the freeze key can be edited by the same patch that caused the failure, the freeze is theater. Signatures, fixture hashes, and invariant ids are editable too, but they are edited on purpose and they show up as ledger diffs. That is the review surface. Use it, or drop the freeze and fail closed.