Bind Flake Freezes to Assertion Hashes, Not Test Names

Agent patches do not earn a merge by renaming a flaky test, rewriting its message, or commenting out the check. Bind every freeze to a content hash of the assertion and the fixture it ran against. If the assertion text moves, the freeze does not follow. The scorer owns that map. The workspace copy of the tests is untrusted input.

Test names are a weak identity. Agents rename freely. File paths move during “cleanup.” Assertion messages get softened until a matcher no longer fires. A freeze keyed on test_api_create_user dies the moment the function is called test_api_create_user_v2. A freeze keyed on the assertion source and the fixture digest survives that rename, and it flags the rewrite.

This article is a testing strategy for agent-generated patches. It uses three artifacts only: pure property checks, content-addressed fixtures, and an assertion-hash freeze map. The merge rule is simple. A patch may add properties. It may not delete a hashed assertion, and it may not satisfy a freeze by editing the assertion that produced it.

Why names and skip marks fail

Most suites identify a test by node id: file, class, function, parameterization. That is enough for humans. It is not enough when the author of the diff is an agent that is scored on “tests green.”

Three cheap edits all produce green:

1. Delete the test function, or wrap it in pytest.mark.skip.
2. Rewrite assert body["status"] == "ok" into assert "status" in body.
3. Point the test at a new fixture whose payload never exercises the failing branch.

None of those edits prove the behavior. They only change the identity the freeze was hanging on. Hash the assertion, and those edits become mismatches instead of passes.

Three artifacts, one scorer

Keep the oracle pack off the agent’s writable tree. The pack is what the scorer fetches. The agent checkout is what the scorer runs code from. Mixing them is how a patch rewrites the exam.

Properties are pure functions of (fixture_input, candidate_output). They do not read clocks, clocks’ env vars, or the agent’s working tree. They return hold, break, or error. No I/O inside the predicate.

Fixtures are recorded inputs plus a digest of the bytes the property is allowed to see. The digest is sha256 of the canonical encoding, not of a pretty-printed JSON file the agent can re-wrap. If the bytes change, the fixture id changes. Old freezes do not silently attach to a new payload.

Freeze map is a signed list of freeze keys. Each key is:

sha256(test_id || "\n" || assertion_src || "\n" || fixture_digest)

test_id is a stable logical name you assign, not the pytest node id. assertion_src is the normalized source of the single check (whitespace collapsed, no comment tokens). fixture_digest is the fixture’s content hash. A freeze is a quarantine record. It is not a pass.

Decision table

Score each assertion independently. Do not fold the run into one job emoji.

Event	What the workspace did	Scorer verdict	Merge?
Property holds, no freeze	Unrelated or additive	HOLD	Allowed if all rows allow
Property breaks, no freeze	Real fail or new flake	BREAK	No
Property breaks, freeze key matches	Known flake, assertion intact	QUARANTINE	No
Property holds, freeze key still present	Flake may have died	THAW-CANDIDATE	No, human only
Assertion source changed, old key still in map	Rewrite or soften	REWRITE	No
Logical test id gone, key still in map	Delete or skip	DELETE	No
Fixture digest in pack ≠ digest in run	Pack swap or local edit	PACK-MISMATCH	No
Pack signature fails	Tamper or wrong key	PACK-REJECT	No

QUARANTINE is the important row. The run is not green. The patch is not mergeable. The assertion is also not “fixed” by silence. That is the whole point of hashing the check instead of skipping the file.

Layout the agent cannot own

Proposed layout. Keep oracle/ out of the agent write set. Fetch it at score time.

repo/
  src/                  # agent may write
  tests/runner/         # thin loader, no expected values
oracle/                 # fetched by scorer, not cloned writable
  pack.hmac
  fixtures/
    create_user.json
    create_user.sha256
  properties.py
  freeze.json

tests/runner/ may import properties.py only after the scorer has verified the pack. If the runner can see expected values in the agent tree, the isolation is theater.

Reference harness

The following is a compact, runnable sketch. Label it as a local reference, not a production SLA. It uses HMAC on the pack bytes so the scorer can refuse a swapped oracle. Put the key in CI, not in the repo.

# scorer.py — reference harness, run on the scoring host
from __future__ import annotations

import hashlib, hmac, json, os, re, sys
from pathlib import Path

FREEZE = "QUARANTINE"

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def normalize_assert(src: str) -> str:
    src = re.sub(r"#.*", "", src)
    return re.sub(r"\s+", " ", src).strip()

def freeze_key(test_id: str, assertion_src: str, fixture_digest: str) -> str:
    payload = f"{test_id}\n{normalize_assert(assertion_src)}\n{fixture_digest}"
    return sha256_bytes(payload.encode())

def verify_pack(oracle_dir: Path, key: bytes) -> bytes:
    blob = b""
    for p in sorted(oracle_dir.rglob("*")):
        if p.is_file() and p.name != "pack.hmac":
            rel = p.relative_to(oracle_dir).as_posix().encode()
            blob += rel + b"\0" + p.read_bytes() + b"\0"
    expected = (oracle_dir / "pack.hmac").read_text().strip()
    got = hmac.new(key, blob, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(got, expected):
        raise SystemExit("PACK-REJECT")
    return blob

def load_freeze_map(path: Path) -> set[str]:
    data = json.loads(path.read_text())
    return set(data["keys"])

def score_row(test_id, assertion_src, fixture_digest, result, freeze_keys):
    key = freeze_key(test_id, assertion_src, fixture_digest)
    if result == "error":
        return "PACK-MISMATCH" if False else "BREAK"
    if result == "break" and key in freeze_keys:
        return FREEZE
    if result == "break":
        return "BREAK"
    if result == "hold" and key in freeze_keys:
        return "THAW-CANDIDATE"
    return "HOLD"

A property module in the pack stays boring on purpose. Boring predicates are harder to game than custom matchers that live next to the agent.

# oracle/properties.py — lives in the signed pack
from typing import Any, Literal

Verdict = Literal["hold", "break", "error"]

def prop_create_user_status(inp: dict[str, Any], out: dict[str, Any]) -> Verdict:
    if "status" not in out:
        return "break"
    if out["status"] not in {200, 201, 400, 409}:
        return "break"
    if out["status"] in {400, 409} and "error_code" not in out.get("body", {}):
        return "break"
    return "hold"

PROPERTIES = {
    "create_user.status": {
        "fn": prop_create_user_status,
        "assertion_src": (
            'assert out["status"] in {200, 201, 400, 409}; '
            'assert error_code if status in {400, 409}'
        ),
        "fixture": "create_user",
    }
}

Commands on the scoring host, not in the agent sandbox:

export ORACLE_HMAC_KEY  # injected by CI, never committed
git clone --depth 1 "$PATCH_SHA" work
rsync -a "$ORACLE_MIRROR/oracle/" ./oracle/
python scorer.py --checkout work --oracle oracle --freeze oracle/freeze.json

If scorer.py is itself in the agent write set, stop. Put the harness on the scoring host image. The patch under test is an argument, not a peer of the oracle.

Numbered workflow

1. Mint a logical test id when a property is accepted. Do not use the pytest node id. Record it in the pack, signed.
2. Canonicalize fixtures with sorted keys and no insignificant whitespace. Write fixture.sha256 beside the bytes. The digest in the freeze key is that file, not a runtime json.dumps default.
3. Normalize assertion source before hashing. Strip comments. Collapse whitespace. If an agent inserts a comment above the assert, the key must not change. If it changes the assert, the key must change.
4. Run properties on the scoring host. Feed (fixture, output). Collect one verdict per assertion, not one verdict per job.
5. Join with the freeze map. Apply the table above. QUARANTINE and BREAK both refuse merge. Only the reason string differs.
6. Refuse pack mismatches before code faults. A wrong HMAC is not a flake. Do not write it into freeze.json.
7. Promote a freeze only from the scorer’s output. A human (or a signed bot that is not the authoring agent) appends the key. The agent’s PR cannot add freeze keys for assertions it just broke.
8. Treat thaw as a human action. THAW-CANDIDATE means the assertion held while a freeze still exists. That is evidence, not a green light. Remove the key in a follow-up that does not contain behavior changes.

Step 7 is the one teams skip. Then the agent opens a PR that both breaks prop_create_user_status and adds its key to freeze.json. Hashing does not help if the map is writable by the same author.

Where a drafting model and a scoring host fit

Large diffs still need help proposing new properties. That is a drafting job, not a scoring job.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access is enough to emit unsigned property drafts from a patch diff: candidate predicates, suggested logical ids, and a list of fixtures the predicate would need. Those drafts land in review. They do not land in oracle/properties.py until a human copies them into the signed pack. The free server option is the scoring host in this layout: it fetches the pack, verifies the HMAC, and runs scorer.py against a checkout the agent cannot write. No model name, quota, or machine profile is required for the separation to work. If the draft model can push to oracle/, the split has already failed.

A minimal draft prompt, for operators who want the model to stay outside CI:

Read the unified diff. Propose pure properties as
(test_id, assertion_src, fixture_name, predicate_python).
Do not edit tests. Do not suggest pytest.mark.skip.
Do not invent fixture bytes. List missing fixtures as gaps.

The output is a review note. The scorer never reads it.

Limitations

Assertion hashing does not detect a tautology that was hashed on day one. assert True has a stable source. It will freeze and thaw like a real check. Classify predicates when they enter the pack, or this scheme will faithfully quarantine nonsense.

It also does not replace coverage or mutation scoring. An agent can add a new function and no properties. Every row in the table will say HOLD because nothing ran. Pair this with a rule that new exported symbols need at least one packed property. That rule is out of scope here; without it, silence still looks clean.

HMAC on the pack is only as strong as key handling. A key in the agent env is not a key. Rotate it outside the repo. If two scoring hosts use different keys, you will get PACK-REJECT that looks like flakiness. It is not.

Normalized source can collide. Two different asserts that collapse to the same tokens hash the same. Keep test_id in the key to bound that risk. Do not drop test_id to “make hashes portable across refactors.” Portability is how a freeze follows a rewrite you meant to catch.

Who should not use this

Do not use assertion-hash freezes on UI tests whose failure text is a screenshot path or a generated CSS selector. The source will churn every run. You will freeze noise.

Do not use it on tests that must hit live network. Fixture digests will not match, and the table will collapse into PACK-MISMATCH. Record the I/O first.

Do not use it if the same agent both authors the patch and is allowed to append freeze.json. The hash is then a speed bump. Skip marks were already a speed bump.

Small libraries with a handful of hand-written tests and no agent in CI gain little. The machinery costs a host, a key, and a pack pipeline. Apply it where patches are generated faster than reviewers can read assertion diffs.

The merge question is not “did pytest exit 0 in the workspace.” It is whether every packed assertion either held or sat in quarantine under the same hash it had yesterday. If an agent wants the flake gone, it has to change behavior until a human thaws the key. Renaming the test is no longer a strategy. If a freeze in your suite has already survived an assertion rewrite, that is the case this key is built to catch—drop the before/after sources in a comment if you have one.