Agent Patches Evade Name Freezes. Bind Flakes to Failure Signatures.

A flake freeze keyed by test name is not a freeze. An agent can rename the case, split it, or wrap it in a helper and the exemption vanishes from review. Bind the freeze to a hash of the failure signature, require property checks compiled from a human-owned spec, and store fixtures as inputs only. Evaluate that untrusted suite on a scratch runner before it touches the merge gate.

Name-based exemptions assume a stable human author. Agent patches do not provide that author. The rest of this note is a concrete binding method, not a slogan.

What a name freeze actually records

A typical freeze file stores tests/test_parser.py::test_roundtrip. That string is a path. It is not evidence. After an agent patch, the same intermittent fault can reappear as test_roundtrip_v2, test_roundtrip_with_fixture, or a parametrized id the reviewer has never seen.

The merge gate then reports a clean freeze list. The flake is still in the product. The exemption moved. Reviewers who only diff test names will miss it.

Three bindings that survive a rename

Keep three artifacts next to the patch, not inside the agent's commit message.

1. Failure signature. Hash the normalized exception type, the first application frame, and the assertion predicate. Drop timestamps, line numbers, and absolute paths.
2. Spec-sourced property. Write the invariant in a file the agent is not allowed to edit in the same change. The property must fail if the production edit is reverted.
3. Input-only fixture digest. Hash fixture bytes that the code reads. Do not store expected outputs beside those bytes.

If any of the three is missing, the patch is unproven. Green tests are not a substitute.

Decision table

Observation	Bind to	Merge?
Same test name, new exception type	New signature; old freeze does not apply	No, until re-triaged
New test name, same signature	Same freeze record; flag as rename	No auto-merge; human ack
Property still passes after reverting the production diff	Property is not about this patch	No
Fixture file changed and digest moved	Treat as a new oracle surface	No
Signature matches freeze, property fails on revert, fixture digest unchanged	Evidence holds	Yes, with the freeze recorded

The table is the gate. A checklist in a PR template is not.

Workflow

1. Normalize the failure before anyone freezes it

Collect JUnit (or TAP) from a single isolated run. Do not freeze from a combined CI log. Combined logs mix order, retries, and other tests' stdout.

pytest -q --junitxml=scratch/junit.xml tests/target_mod

Parse XML in a small script. Keep only fields that would still match after a rename.

# signature.py — example harness, not production metrics
from __future__ import annotations

import hashlib
import re
import xml.etree.ElementTree as ET
from pathlib import Path

FRAME_RE = re.compile(
    r'File "[^"]+", line \d+, in (?P<func>\S+)'
)
PRED_RE = re.compile(r'AssertionError: (?P<pred>.+?)(?:\n|$)')


def normalize_trace(message: str) -> str:
    lines = []
    for raw in message.splitlines():
        line = raw.strip()
        line = re.sub(r'/[^\s]+/', '<path>/', line)
        line = re.sub(r':\d+', ':N', line)
        line = re.sub(r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}', '<ts>', line)
        lines.append(line)
    return '\n'.join(lines)


def signature_of(testcase: ET.Element) -> dict:
    failure = testcase.find('failure')
    if failure is None:
        return {}
    msg = failure.get('message') or ''
    body = failure.text or ''
    blob = normalize_trace(msg + '\n' + body)
    frame = FRAME_RE.search(blob)
    pred = PRED_RE.search(blob)
    payload = '|'.join([
        (failure.get('type') or 'unknown').strip(),
        (frame.group('func') if frame else 'nofunc'),
        (pred.group('pred') if pred else blob[:180]),
    ])
    digest = hashlib.sha256(payload.encode()).hexdigest()[:16]
    return {
        'sig': digest,
        'type': failure.get('type'),
        'func': frame.group('func') if frame else None,
        'name': testcase.get('name'),
    }


def load_failures(path: Path) -> list[dict]:
    root = ET.parse(path).getroot()
    out = []
    for case in root.iter('testcase'):
        sig = signature_of(case)
        if sig:
            out.append(sig)
    return out

Short names are for humans. The 16-hex digest is for the gate.

2. Refuse name-keyed freeze files

Store freezes by signature. Keep the last seen test name only as a comment field.

# freeze/signatures.yaml  — human-owned; agent patches cannot add rows
version: 1
records:
  - sig: 7c1a9e2b44d0af13
    last_name: test_roundtrip
    reason: clock-skew on leap-smear hosts
    ticket: QA-4412
    owner: parser-team

A CI step must fail the build if it finds a freeze keyed only by node id.

# check_freeze_shape.py
import sys
import yaml
from pathlib import Path

data = yaml.safe_load(Path('freeze/signatures.yaml').read_text())
bad = []
for row in data.get('records', []):
    if 'sig' not in row or not re.fullmatch(r'[0-9a-f]{16}', str(row['sig'])):
        bad.append(row)
    if 'name' in row and 'sig' not in row:
        bad.append(row)
if bad:
    sys.stderr.write('name-keyed or malformed freeze rows\n')
    sys.exit(1)

If an agent rewrites the YAML, the shape check still passes. Ownership does not. Put freeze/ in CODEOWNERS so the agent cannot land a new row without a reviewer who is not the patch author.

3. Compile properties from a spec the patch cannot edit

A property that the agent wrote against its own production diff is circular. Keep invariants in spec/properties/ and deny that path in the same change as src/.

# spec/properties/test_parse_invariants.py
import ast
from parser import parse_config

ALLOWED = {'host', 'port', 'retries'}

def test_unknown_keys_rejected():
    doc = parse_config('host: a\nport: 2\nretries: 1\ncolor: red\n')
    assert set(doc) <= ALLOWED

def test_port_is_int_in_range():
    doc = parse_config('host: a\nport: 8080\nretries: 0\n')
    assert isinstance(doc['port'], int)
    assert 1 <= doc['port'] <= 65535

def test_property_fails_if_patch_reverted(tmp_path):
    # Label: unexecuted against your tree until you wire git show HEAD1:src/parser.py
    source = Path('src/parser.py').read_text()
    tree = ast.parse(source)
    names = {n.name for n in ast.walk(tree) if isinstance(n, ast.FunctionDef)}
    assert 'parse_config' in names

The interesting check is not that the property is green. Revert the production files and run the same file. If it stays green, the property does not describe this patch.

git stash push -m agent-prod -- src
pytest -q spec/properties
status=$?
git stash pop
# expect non-zero while production is stashed

Drafting candidate predicates is slow when done by hand. A free-model pass over the spec is enough to propose assertions. It is not enough to accept them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can turn a short spec paragraph into candidate predicates; a human still has to delete any predicate that quotes the patch instead of the spec. The free server option is a scratch runner for the signature suite so exploratory agent-patch evaluation does not occupy the merge queue. Neither claim includes a model name, a quota, or a hardware profile, because those are not specified here.

4. Digest fixture inputs, not expected trees

Put only the bytes the parser reads under fixtures/input/. Compute a digest at gate time. If the agent adds expected.json next to an input, fail.

# fixture_lock.py
from pathlib import Path
import hashlib
import sys

ROOT = Path('fixtures')
blocked = list(ROOT.rglob('expected.*')) + list(ROOT.rglob('*out*.json'))
if blocked:
    sys.stderr.write('output-shaped fixture files are not allowed\n')
    sys.exit(1)

digests = {}
for path in sorted(ROOT.joinpath('input').rglob('*')):
    if path.is_file():
        digests[str(path)] = hashlib.sha256(path.read_bytes()).hexdigest()

lock = Path('fixtures/inputs.sha256')
wanted = {line.split()[1]: line.split()[0] for line in lock.read_text().splitlines() if line}
if digests != wanted:
    sys.stderr.write('fixture input digest mismatch\n')
    sys.exit(1)

Expected values belong in properties, which state a rule. They do not belong in a golden file the agent can edit until the rule is true by construction.

5. Run the untrusted suite on a scratch runner, then promote

Merge CI should see only three commands. Everything else is noise the agent can game with retries.

python check_freeze_shape.py
python fixture_lock.py
python - <<'PY'
from pathlib import Path
from signature import load_failures
import yaml, sys
allowed = {r['sig'] for r in yaml.safe_load(Path('freeze/signatures.yaml').read_text())['records']}
seen = {f['sig'] for f in load_failures(Path('scratch/junit.xml'))}
unknown = seen - allowed
if unknown:
    print('unfrozen signatures', sorted(unknown))
    sys.exit(1)
PY
pytest -q spec/properties src

Use a scratch machine for the first loop: generate signatures, propose properties, watch renames. Promote the three files (signatures.yaml, inputs.sha256, spec/properties/*) only after a human ack. The merge queue then runs the same commands. It does not run the agent's extra tests as evidence.

What this does not prove

Signature length 16 hex is a convenience, not a cryptographic bound. Collisions are unlikely in one repo and still possible across generated traces that share one assertion string. If two faults produce the same predicate text, split the predicate, do not lengthen the hash and call it solved.

Environment drift still breaks signatures. A glibc locale change can rewrite an exception message and look like a new fault. Pin the locale and the timezone in the scratch runner. If you cannot pin them, do not freeze.

Properties that only call the public API with one input are unit tests in costume. A property needs a class of inputs or a relation (round-trip, idempotence, monotonicity). One literal is a fixture wearing a test_property_ prefix.

This method does not score model quality. It scores whether the patch is bound to evidence that survives a rename.

Who should not use it

Skip this if the suite is UI-only and the failure text is a screenshot hash. There is no stable exception type to bind.

Skip it if there is no human-owned spec. Free-model drafts from the production diff will overfit. That is the failure mode this process exists to block.

Skip it if flake rate is dominated by shared hardware you do not control. Signature freezes will accumulate until they are a second bug tracker. Fix the hardware queue first.

Skip it if the team wants the agent to own freeze/ and spec/properties/. Once those paths are writable in the same commit as src/, the bindings are theatre.

Limits of the scratch runner

A scratch runner is for evaluation. It is not a second source of truth. If it disagrees with merge CI, believe merge CI and treat the scratch run as contaminated. Clock skew, missing locale pins, and partial checkouts are enough to create that disagreement.

Do not copy agent-authored tests from the scratch workspace into merge CI because they were green there. Green on an untrusted suite is the default. The revert check is the filter.

The durable output of the scratch pass is three small files. Not a transcript. Not a chat log. Not a coverage percentage.

If you already have a merge queue and you only need a place to compute signatures before those three files are real, MonkeyCode's free server option is sufficient as that scratch runner. Keep the gate itself where your CODEOWNERS already apply.