Zero-Hit Properties Cannot Be Frozen. Score Agent Patches by Fixture Hits.

An agent patch that leaves the suite green has not been proven. The number that matters is how many locked fixture fields can still force a property to fail. If that count is zero, the check is decoration. A flake freeze on top of it only hides the gap.

This write-up is a scoring procedure, not a production study. The scorer perturbs fixtures, not program source. It then records flakes by cause class. Properties with a hit score of zero are rejected. They are not freeze candidates.

The wrong unit of evidence

CI reports pass or fail per test name. Agent patches exploit that grain. They keep names stable, weaken assertions, or ride a flake until someone adds skip. The suite stays green. The constraint surface shrinks.

A fixture-hit score answers a narrower question. Which fields in a frozen fixture does this property actually read in a way that can fail? A flake ledger answers a different one. When the same seed and the same fixture digest disagree across retries, what cause class explains it, and is that class allowed to freeze?

Those two numbers are independent. Do not mix them. Hits measure coupling. The ledger measures instability under an unchanged lock. Treating either as a skip list collapses the gate.

Artifact: hit matrix, retry log, cause-class ledger

The artifact is three files plus two small programs.

1. fixtures/lock.json — canonical inputs, with a digest file beside it.
2. hit_report.json — per-property hit counts, claimed fields, and false claims.
3. flake_ledger.json — freeze records keyed by cause class, not by pytest node id.
4. score_hits.py — perturbs one fixture field at a time and records which perturbations flip status.
5. check_ledger.py — admits only known cause classes, matching digests, and unexpired rows.

The listings below are a runnable sketch. They are not a benchmark. They do not claim a pass rate on any corpus.

1. Lock the fixture, not the test file

Keep the fixture outside the patch directory. Canonicalize JSON, then hash it. Refuse the patch if the digest changes in the same commit as the agent diff.

{
  "currency": "USD",
  "total_cents": 1500,
  "lines": [
    {"qty": 2, "unit_cents": 500},
    {"qty": 1, "unit_cents": 500}
  ]
}

# digest_lock.py
import hashlib, json, pathlib, sys

def digest(path: pathlib.Path) -> str:
    data = json.loads(path.read_text())
    canonical = json.dumps(data, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

if __name__ == "__main__":
    lock = pathlib.Path("fixtures/lock.json")
    expected = pathlib.Path("fixtures/lock.sha256").read_text().strip()
    actual = digest(lock)
    if actual != expected:
        sys.stderr.write(f"fixture digest mismatch\n{actual}\n")
        sys.exit(2)
    print(actual)

Test names can move. The digest cannot, not in the same change as the generated patch. If a human must extend the fixture, that is a separate commit with a separate review of the lock file.

2. Declare the fields each property claims to use

Each property lists fixture keys. The scorer verifies the list. A mismatch is a failure of test authoring, not of the patch under test.

# properties.py
from dataclasses import dataclass
from typing import Callable, Set

@dataclass(frozen=True)
class Property:
    name: str
    claimed_fields: Set[str]
    check: Callable[[dict], None]

def invoice_total_matches_lines(fx: dict) -> None:
    total = sum(line["qty"] * line["unit_cents"] for line in fx["lines"])
    if fx["currency"] != "USD":
        raise AssertionError("unsupported currency")
    if total != fx["total_cents"]:
        raise AssertionError(f"total {fx['total_cents']} != {total}")

PROPERTIES = [
    Property(
        name="invoice_total_matches_lines",
        claimed_fields={"lines", "total_cents", "currency"},
        check=invoice_total_matches_lines,
    ),
]

If a property claims currency but never fails when currency is perturbed, the claim is false. Drop the field or rewrite the check. False claims inflate coverage reports without adding a failure mode.

3. Score hits by single-field perturbation

Perturb one field per trial. Keep the unperturbed digest in the report. Do not perturb the program in this step. The question is whether the property is coupled to the fixture at all.

# score_hits.py
import copy, json, pathlib
from properties import PROPERTIES

PERTURB = {
    "currency": lambda v: "EUR" if v == "USD" else "USD",
    "total_cents": lambda v: v + 1,
    "lines": lambda v: v + [{"qty": 1, "unit_cents": 1}],
}

def run(check, fx) -> str:
    try:
        check(fx)
        return "pass"
    except Exception:
        return "fail"

def score(fx: dict) -> list[dict]:
    rows = []
    for prop in PROPERTIES:
        baseline = run(prop.check, fx)
        hits = []
        for field in sorted(fx):
            if field not in PERTURB:
                continue
            mutant = copy.deepcopy(fx)
            mutant[field] = PERTURB[field](mutant[field])
            if run(prop.check, mutant) != baseline:
                hits.append(field)
        claimed = set(prop.claimed_fields)
        rows.append({
            "property": prop.name,
            "baseline": baseline,
            "hit_count": len(hits),
            "hits": hits,
            "claimed": sorted(claimed),
            "false_claims": sorted(claimed - set(hits)),
            "unclaimed_hits": sorted(set(hits) - claimed),
        })
    return rows

if __name__ == "__main__":
    fx = json.loads(pathlib.Path("fixtures/lock.json").read_text())
    report = score(fx)
    pathlib.Path("hit_report.json").write_text(json.dumps(report, indent=2))
    bad = [r for r in report if r["hit_count"] == 0 or r["false_claims"]]
    if bad:
        raise SystemExit(f"reject {len(bad)} properties")

Gate rule: reject the patch when any property that is supposed to cover the diff has hit_count == 0. Also reject when false_claims is non-empty. A property that cannot be failed by its own fixture must not enter a freeze file.

4. Compare hit surface against the parent revision

Run the same properties on HEAD and on the agent worktree. Same fixture. Same digest. Compare hits, not traces. A shrunken hit surface is a regression even when every property still passes.

#!/usr/bin/env bash
set -euo pipefail
python digest_lock.py
git stash push -u -m "agent-worktree" -- .
python score_hits.py
cp hit_report.json /tmp/parent_hits.json
git stash pop
python digest_lock.py
python score_hits.py
python - <<'PY'
import json, sys
parent = {r["property"]: r for r in json.load(open("/tmp/parent_hits.json"))}
child = {r["property"]: r for r in json.load(open("hit_report.json"))}
for name in sorted(set(parent) | set(child)):
    p, c = parent.get(name), child.get(name)
    if p is None or c is None:
        print(f"PROPERTY_SET_CHANGED {name}")
        sys.exit(3)
    if c["hit_count"] < p["hit_count"]:
        print(f"HIT_SURFACE_SHRANK {name} {p['hits']} -> {c['hits']}")
        sys.exit(4)
print("hit surface held")
PY

Do not treat a newly added property as extra credit until it has a non-zero hit count on the locked fixture. Empty properties pad the suite. They do not pad the score.

5. Retry before any freeze exists

Disagreement under an unchanged digest is a flake. Name it only after retries. Three identical runs is a convention, not a measured optimum. Label it as such in CI.

#!/usr/bin/env bash
set -euo pipefail
digest=$(python digest_lock.py)
mkdir -p /tmp/retries
for i in 1 2 3; do
  python score_hits.py
  cp hit_report.json /tmp/retries/hit_$i.json
done
python - <<'PY'
import json, pathlib, sys
rows = [json.loads(p.read_text()) for p in sorted(pathlib.Path("/tmp/retries").glob("hit_*.json"))]
names = rows[0]
for r in rows[1:]:
    if r != names:
        print("FLAKE_UNCLASSIFIED")
        sys.exit(5)
print("retries agree")
PY

If retries disagree, fail with FLAKE_UNCLASSIFIED. Do not skip. A human then adds a ledger row. The agent patch must not add that row.

6. Freeze flakes by cause class, never by node id

Retries that disagree are flakes. Name the cause. Put the name in a ledger. Do not pytest.mark.skip the file. Node ids churn when the agent renames tests. Cause classes do not.

Allowed cause classes:

1. CLOCK — wall time or timezone in the property, not in the fixture.
2. HASH_ORDER — iteration over an unordered collection leaked into an assertion.
3. FLOAT_ULP — equality on binary floats without an ulp budget.
4. EXTERNAL_IO — network, clock daemon, or shared cache.

{
  "version": 1,
  "rules": [
    {
      "property": "invoice_total_matches_lines",
      "cause_class": "HASH_ORDER",
      "fixture_digest": "replace-with-lock-sha256",
      "owner": "maintainer",
      "expires": "2026-10-04",
      "action": "sort line ids in the property; do not skip",
      "freeze_hits": false
    }
  ]
}

CI rules for the ledger:

1. Unknown cause_class fails the build.
2. EXTERNAL_IO is invalid if every claimed field already lives in lock.json. The I/O does not belong in that property.
3. freeze_hits: true is rejected. Hits measure coupling to the fixture. The scorer is not freezable.
4. Expired rows fail closed. The property runs. It is not skipped.
5. The agent patch must not modify flake_ledger.json. Ledger edits are a human-only path.

# check_ledger.py
import datetime as dt, json, os, pathlib, sys

ALLOWED = {"CLOCK", "HASH_ORDER", "FLOAT_ULP", "EXTERNAL_IO"}
lock_sha = pathlib.Path("fixtures/lock.sha256").read_text().strip()
ledger = json.loads(pathlib.Path("flake_ledger.json").read_text())
today = dt.date.fromisoformat(os.environ.get("CI_DATE", "2026-09-20"))

for row in ledger["rules"]:
    if row["cause_class"] not in ALLOWED:
        sys.exit(f"unknown cause {row['cause_class']}")
    if row.get("freeze_hits"):
        sys.exit("cannot freeze the hit scorer")
    if row["fixture_digest"] != lock_sha:
        sys.exit("ledger digest does not match lock")
    if dt.date.fromisoformat(row["expires"]) < today:
        sys.exit(f"expired freeze {row['property']}")
print("ledger ok")

Inject CI_DATE from the CI clock. Do not let the patch supply today. A frozen clock in the repo is another oracle the agent can edit.

Where generation and scoring should run

Keep three processes separate: patch proposal, hit scoring, ledger review. Mixing them in one session is how lock files get rewritten to match the patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can propose a candidate patch. The free server option can run score_hits.py and check_ledger.py so the machine that holds fixtures/lock.json does not also host the generator.

The product is optional. The separation is not. A laptop, a VM, and a locked fixture directory implement the same control. A hosted runner does not make properties stronger. The hit matrix is the evidence.

Decision table

Observation	Action
hit_count == 0	Delete or rewrite the property. No freeze.
false_claims non-empty	Fix the claim set. Fail the patch.
Hit surface shrank vs parent	Fail the patch.
Status flips with digest unchanged	Classify cause. Human ledger only.
Ledger row expired	Run the property. Do not skip.
Patch touches flake_ledger.json or lock.json	Reject. Split the commit.

Limitations

Fixture perturbation is not mutation testing of the program. A property can have a high hit count and still miss a bug in an unmodeled branch. The scorer only proves coupling to the fixture you already have.

Single-field perturbation misses interactions. If a failure requires currency and lines to change together, the matrix under-counts. Extend the scorer only when a specific interaction is documented. Do not search the full product of fields by default. Combinatorial expansion is a different test, with a different budget.

Cause classes are coarse. CLOCK can hide a real race. Cap CLOCK freezes at one active row per property. If a second CLOCK event appears, delete the freeze and fix the property. HASH_ORDER is not a license to compare unsorted dumps forever. The action field is mandatory because a freeze without a next step is a skip.

The workflow assumes deterministic serialization of fixtures. Sets, maps without sort keys, and timestamps inside lock files will churn the digest and starve the gate. Put clocks in the property under test, or do not put them in the lock.

This is not a security boundary. An agent that can edit CI YAML can disable the scorer. Pin score_hits.py and check_ledger.py on a path the patch allowlist cannot touch. If the allowlist does not exist, the rest of this procedure is theater.

Who should not use this

Do not use this as the only gate if you have no gold fixtures. A hit score on synthetic JSON that the same agent just wrote is self-agreement.

Do not let the agent write flake_ledger.json. A freeze written by the process that failed the property is not a freeze. It is a self-issued waiver.

Do not apply EXTERNAL_IO freezes to unit properties. Move those checks to a contract suite with recorded responses, or drop them. Recorded I/O still needs a digest. It does not belong in the unit lock.

Safety-critical changes still need a human-owned oracle and a review of the patch. A green hit report is a merge input. It is not a certificate.

What to keep when you throw the rest away

Keep the hit count. Keep the rule that zero-hit properties cannot be frozen. Keep the ledger off the agent's edit path. If those three hold, a passing suite after an agent patch at least failed something on purpose.

Compute the hit report before review so it lands next to the diff, not after merge.