Freeze the Failure Signature, Not the Test Name

A green job is not merge evidence for an agent patch when property checks are unseeded, when fixtures sit in a writable tree, or when a flake freeze keys only on a pytest node id. Those three gaps let a patch look stable while the oracle moves. Score the patch against a seeded trial log, a hash-locked fixture corpus, and a freeze ledger bound to an exception digest.

Test names are labels. They are not oracles. An agent can rename test_retry_backoff, split a module, or wrap the same assertion in a helper. A freeze that stores only tests/test_retry.py::test_retry_backoff then expires against the wrong node, or never expires at all. The failure that justified the freeze is gone from the ledger.

Unseeded property trials have the same shape of false confidence. A pass on this run may be a miss on the next. If the scorer cannot replay the exact draw that produced a counterexample, a freeze cannot bind to anything real. Replay is the evidence. Color is not.

What the ledger must store

Store three artifacts next to the patch, not inside it. Keep them in a tree the authoring process cannot write.

1. A property trial log with an explicit RNG seed, trial count, hit count, and the minimized input if a check failed.
2. A fixture manifest that hashes every human-owned input file the scorer is allowed to read.
3. A freeze record that keys on failure signature: node id plus seed plus exception type plus a digest of the message body.

If any of those three is missing, the merge score is incomplete. Do not treat an incomplete score as a pass. Treat it as an advisory gap and leave the patch unmerged.

Layout the agent cannot write

Separate the writable source tree from the oracle tree. Agent-authored tests can exist. They must not count.

repo/
  src/                  # agent may patch
  tests/generated/      # agent-authored, never scored
  oracle/
    fixtures/           # human-owned, hashed
    properties/         # seeded checks
    freeze_ledger.json  # signature-bound
    manifest.sha256

The scorer reads oracle/. The agent process must not have write access to that path. A patch that touches oracle/ is a process failure, not a test failure. Reject it before scoring.

Step 1: Hash the fixture corpus

Compute a manifest from file contents, not from mtime. Content hashes survive checkout order. Timestamps do not.

# oracle_hash.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ORACLE = Path("oracle")
FIXTURES = ORACLE / "fixtures"
MANIFEST = ORACLE / "manifest.sha256"


def file_digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def build_manifest() -> dict[str, str]:
    rows: dict[str, str] = {}
    for path in sorted(FIXTURES.rglob("*")):
        if path.is_file():
            rel = path.relative_to(ORACLE).as_posix()
            rows[rel] = file_digest(path)
    return rows


def write_manifest() -> None:
    payload = json.dumps(build_manifest(), indent=2, sort_keys=True) + "\n"
    MANIFEST.write_text(payload)


def verify_manifest() -> list[str]:
    expected = json.loads(MANIFEST.read_text())
    actual = build_manifest()
    problems: list[str] = []
    for key in sorted(set(expected) | set(actual)):
        if expected.get(key) != actual.get(key):
            problems.append(f"hash mismatch: {key}")
    return problems


if __name__ == "__main__":
    import sys

    mode = sys.argv[1] if len(sys.argv) > 1 else "verify"
    if mode == "write":
        write_manifest()
    else:
        problems = verify_manifest()
        if problems:
            print("\n".join(problems))
            raise SystemExit(2)
        print("oracle fixture hashes match")

Run python oracle_hash.py write only on a human-owned checkout. CI on an agent patch runs python oracle_hash.py verify. A mismatch is a voided score, not a flake. Do not retry it. Do not freeze it.

Step 2: Seed every property trial

Do not call random without a seed recorded in the trial log. A property that cannot be replayed cannot be frozen. The runner below is a proposed example, not a measured production suite.

# oracle/properties/retry_budget.py
from __future__ import annotations

import json
import random
from dataclasses import asdict, dataclass
from pathlib import Path


@dataclass(frozen=True)
class TrialLog:
    property_id: str
    seed: int
    trials: int
    hits: int
    counterexample: dict | None


def retry_budget(delay_ms: list[int], cap_ms: int) -> bool:
    if not delay_ms:
        return False
    total = 0
    for delay in delay_ms:
        if delay < 0 or delay > cap_ms:
            return False
        total += delay
        if total > cap_ms:
            return False
    return True


def run(seed: int, trials: int = 200) -> TrialLog:
    rng = random.Random(seed)
    hits = 0
    counter = None
    for _ in range(trials):
        n = rng.randint(1, 8)
        cap = rng.choice([50, 100, 250, 1000])
        delays = [rng.randint(-5, cap + 40) for _ in range(n)]
        ok = retry_budget(delays, cap)
        # Count a hit only when the draw exercised a non-empty, mostly valid path.
        if any(delay >= 0 for delay in delays):
            hits += 1
        if not ok and counter is None:
            counter = {"delays": delays, "cap_ms": cap}
    return TrialLog("retry_budget", seed, trials, hits, counter)


if __name__ == "__main__":
    log = run(seed=20260921)
    Path("oracle/trial_retry_budget.json").write_text(
        json.dumps(asdict(log), indent=2) + "\n"
    )
    print(json.dumps(asdict(log)))

The fields that matter are seed, trials, hits, and counterexample. A log with hits == 0 is not a freeze candidate. A log with no seed is not a score. Archive the minimized input in the trial log, not in a pytest name.

Step 3: Bind the freeze to a failure signature

A node id is one field. It is not the key. Bind the skip to the failure you actually observed.

# freeze_bind.py
from __future__ import annotations

import hashlib
import json
import re
from pathlib import Path

LEDGER = Path("oracle/freeze_ledger.json")


def digest_message(message: str) -> str:
    normalized = re.sub(r"\d+", "<n>", message.strip())
    return hashlib.sha256(normalized.encode()).hexdigest()[:16]


def signature(nodeid: str, seed: int | None, exc_type: str, message: str) -> str:
    body = f"{nodeid}|{seed}|{exc_type}|{digest_message(message)}"
    return hashlib.sha256(body.encode()).hexdigest()[:24]


def load_ledger() -> dict:
    if not LEDGER.exists():
        return {"freezes": {}}
    return json.loads(LEDGER.read_text())


def freeze_applies(
    nodeid: str,
    seed: int | None,
    exc_type: str,
    message: str,
    today: str,
) -> bool:
    recs = load_ledger()["freezes"]
    sig = signature(nodeid, seed, exc_type, message)
    rec = recs.get(sig)
    if rec is None:
        return False
    return rec["expires_on"] >= today

Digits in messages are normalized so timestamps do not mint a new freeze every run. The seed stays in the key. If the property is reseeded, the old freeze does not apply. That is intentional. A freeze is replay insurance, not a permanent skip.

Proposed ledger shape:

{
  "freezes": {
    "a1b2c3d4e5f6a7b8c9d0e1f2": {
      "nodeid": "oracle/properties/test_retry_budget.py::test_replay",
      "seed": 20260921,
      "exc_type": "AssertionError",
      "message_digest": "9f3c1a2b4d5e6f70",
      "first_seen": "2026-09-21",
      "expires_on": "2026-10-05",
      "reason": "human: intermittent cap rounding on slow runner"
    }
  }
}

Human-authored reason is required. Agent output must not insert rows. Auto-minted freezes recreate the node-id problem with extra JSON.

Step 4: Reject oracle edits before scoring

# ci/reject_oracle_writes.sh
set -euo pipefail
BASE="${1:-origin/main}"
CHANGED="$(git diff --name-only "$BASE"...HEAD)"
if printf '%s\n' "$CHANGED" | grep -E '^oracle/'; then
  echo "reject: agent patch touched oracle/"
  printf '%s\n' "$CHANGED"
  exit 2
fi
python oracle_hash.py verify

Run this before pytest. A patch that "fixes" a flake by editing the ledger is not a fix. It is oracle mutation. Fail closed.

Step 5: Emit a score object, not a job color

Color collapses three questions into one bit. Keep them separate. The script below is a proposed scorer.

# score_patch.py
from __future__ import annotations

import json
from pathlib import Path

from freeze_bind import freeze_applies
from oracle_hash import verify_manifest
from oracle.properties.retry_budget import run


def score(today: str, seed: int = 20260921) -> dict:
    hash_problems = verify_manifest()
    if hash_problems:
        return {"result": "void", "reason": hash_problems}

    log = run(seed=seed)
    Path("oracle/trial_retry_budget.json").write_text(
        json.dumps(
            {
                "property_id": log.property_id,
                "seed": log.seed,
                "trials": log.trials,
                "hits": log.hits,
                "counterexample": log.counterexample,
            },
            indent=2,
        )
        + "\n"
    )
    if log.hits == 0 or log.seed is None:
        return {"result": "incomplete", "trial": log.property_id}

    if log.counterexample is not None:
        message = (
            f"total delay exceeded cap {log.counterexample['cap_ms']}"
        )
        skipped = freeze_applies(
            nodeid="oracle/properties/retry_budget.py::run",
            seed=log.seed,
            exc_type="AssertionError",
            message=message,
            today=today,
        )
        if skipped:
            return {"result": "advisory_skip", "seed": log.seed}
        return {"result": "fail", "counterexample": log.counterexample}

    return {"result": "pass", "seed": log.seed, "hits": log.hits}


if __name__ == "__main__":
    print(json.dumps(score(today="2026-09-21"), indent=2))

Pipe that JSON into the merge rule. Do not fold advisory_skip into pass. Do not fold incomplete into fail either. Incomplete means the oracle did not run. That is a scorer bug or an empty generator, not a product defect in src/.

Decision table

Observation	Score	Merge
Fixture hash mismatch	void	no
oracle/ in the diff	void	no
Property log missing seed	incomplete	no
Property hits == 0	incomplete	no
Failure signature matches unexpired freeze	advisory skip	not a pass
Failure signature matches expired freeze	fail	no
Seeded property fail, no freeze	fail	no
Seeded property pass, hashes match, no open freeze	pass	yes, with review

The advisory-skip row is the point of signature binding. The skip applies only to that exception digest and seed. A new exception type on the same node id still fails the job. A renamed test with the same digest still needs a new human freeze. Neither case is a silent green.

Worked failure signature

Suppose the runner reports a counterexample with cap_ms = 1000 and delays that sum above the cap, seed 20260921, exception type AssertionError. The ledger key is not test_replay. It is the signature of that node, that seed, that exception type, and the normalized message.

If the next run raises TimeoutError instead, the freeze does not apply. If the next run uses seed 20260922, the freeze does not apply. If the agent rewrites the assertion text so the digest changes, the freeze does not apply. The patch must either pass the replay or wait for a human to record a new signature.

That rule looks strict. It is. Flake freezes that outlive the failure they describe become skips with no forensic trail. Node-id freezes are how that happens. Signature freezes make the trail explicit.

Where a separate scorer runs

The agent that authors src/ should not be the process that writes oracle/trial_*.json. Split the two. A local workstation can author the patch. A second environment should check out the same commit, verify hashes, run seeded properties, and apply the ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option fit that split when you want authoring and scoring on different machines without standing up a private runner first. The scorer still needs the oracle tree, the seed, and the ledger. It does not need the model that proposed the patch. If you already have a locked CI worker, use that worker. The product is optional in this workflow; the hash, seed, and signature artifacts are not.

Limitations

Hash locks detect byte changes. They do not detect a human who later commits a weaker fixture. Signature freezes detect a matching exception. They do not detect a patch that swallows the exception and returns a default. Seeded properties detect replayable draws. They do not detect faults outside the generator's distribution.

This workflow also assumes you have human-owned fixtures and at least one property with a real hit rate. Empty corpora and tautological checks produce complete-looking logs that mean nothing. The trial log will still serialize. The score will still be a lie. Do not freeze a property that never hit. Do not hash a directory the agent is allowed to rewrite.

Signature normalization has a cost. Collapsing digits can merge two different bugs that only differ by an error code. If your messages carry enum values that distinguish faults, keep those tokens out of the digit scrub, or bind the freeze to a structured error code field instead of free text.

Who should not use this

Do not install a freeze ledger on a prototype with no flake history. Do not hash fixtures if the agent is expected to own the test tree. Do not bind signatures if your failures are non-deterministic at the exception-type level, such as races that raise either TimeoutError or ConnectionError for the same bug. In that last case, fix the race or record both signatures as a human-reviewed pair. Do not auto-mint freezes from agent output.

Teams that already reject any agent edit under tests/ can adopt the hash and seed steps without the ledger. The ledger is for known flakes you refuse to ignore and refuse to skip by name. If you cannot staff human review of freeze rows, skip the ledger. An unused ledger is worse than none. It looks like control and behaves like a silent allowlist.

The merge question is not whether CI went green. It is whether that green came from a replayable property, an untouched fixture hash, and a freeze that still describes the same failure. If you cannot point to those three artifacts, you do not have a score.