A Scorecard for Agent Diffs: Fixture Digests, Seed Replay, and Failure Signatures

A green unit-test job is not evidence that an agent patch is safe to merge. Agents can keep pytest quiet by rewriting golden files, dropping hard generator seeds, or renaming a flaky case. Score the patch against three files the agent does not own: fixture digests, a property seed log, and a freeze of failure signatures.

The rest of this article is a scoring plan you can run as a separate CI check. The scripts are a proposed harness. They are not production measurements.

What a quiet suite actually hides

Unit tests encode today's examples. They do not encode the distribution those examples came from. An agent that can edit tests/ and testdata/ can delete the awkward case, shrink a generator, or retitle a flake. Coverage often rises while doing it. Lines are easier to paint than invariants.

You need checks that fail when the test surface gets weaker. Silence is not strength. The scorecard below treats three artifacts as read-mostly contracts.

Artifact	Question it answers	Agent may	Agent may not
FIXTURE_LOCK.json	Did committed fixture bytes change?	Add a new path	Rewrite an existing digest without a human override
property_seeds.json	Do known counterexample seeds still hold?	Add a seed after a real find	Drop a seed or lower min_examples
flake_signatures.json	Is this failure a known signature?	Leave the file alone	Add, delete, or rename a signature row

If any row fails, the job is not green. A skip is a fail. Missing files are a fail.

1. Hash fixtures, do not trust golden rewrites

Committed testdata is an API. Changing bytes is a behavior change, even when assertions are updated in the same diff. Digest every file under testdata/ and store the map in git. Humans may rotate a digest with an override token. Agents do not get that token.

Proposed layout:

testdata/invoice_v3.json
FIXTURE_LOCK.json

Example lock file:

{
  "override_token_env": "FIXTURE_OVERRIDE",
  "files": {
    "testdata/invoice_v3.json": {
      "sha256": "e3b0c44298fc1c149afbf4c8996fb924",
      "owner": "payments"
    }
  }
}

Run this check before pytest. Order matters. If the lock is dirty, later green tests are untrusted.

1. Recursively list files under testdata/.
2. SHA-256 each file as raw bytes. Do not pretty-print JSON first.
3. Compare path → digest against FIXTURE_LOCK.json.
4. New paths are allowed only if the lock file in main does not already name them. Record them as added, not as pass.
5. Changed digests require FIXTURE_OVERRIDE to match a value injected by a human job, not by the agent workspace.

# proposed scorer fragment — not a shipped tool
from hashlib import sha256
from pathlib import Path
import json, os, sys

def digest_tree(root: Path) -> dict[str, str]:
    out = {}
    for p in sorted(root.rglob("*")):
        if p.is_file():
            out[p.as_posix()] = sha256(p.read_bytes()).hexdigest()
    return out

def score_fixtures(lock_path: Path, data_root: Path) -> dict:
    lock = json.loads(lock_path.read_text())
    current = digest_tree(data_root)
    expected = {k: v["sha256"] for k, v in lock["files"].items()}
    changed = [p for p, d in expected.items() if current.get(p) != d]
    added = sorted(set(current) - set(expected))
    missing = sorted(set(expected) - set(current))
    override = os.environ.get(lock["override_token_env"])
    ok = not missing and (not changed or bool(override))
    return {"ok": ok, "changed": changed, "added": added, "missing": missing}

Pretty-printing a golden JSON file will flip the digest. That is intentional. Canonicalization belongs in the writer, not in the scorer.

2. Replay seeds; do not re-roll the generator

Property tests without a seed tape are lottery tickets. A patch can lower max_examples, tighten a strategy, or filter the input that used to fail. Pytest still exits 0. The invariant did not get stronger. It got less often asked.

Keep a seed log next to the properties. Each row is a property id, a pytest node id, a floor on examples, and the seeds that once found a counterexample. Replay those seeds on every agent diff. If a recorded seed no longer runs, fail. If min_examples falls, fail.

{
  "properties": [
    {
      "id": "invoice_total_non_negative",
      "nodeid": "tests/test_invoice_properties.py::test_total_non_negative",
      "min_examples": 200,
      "seeds": ["1048291", "77", "9001"]
    }
  ]
}

Wiring Hypothesis (or any example database) is a local choice. The contract is the log, not the library. A minimal replay driver:

# proposed replay — adapt to your property runner
import json, subprocess, sys
from pathlib import Path

def replay_seeds(log_path: Path) -> dict:
    log = json.loads(log_path.read_text())
    failures = []
    for prop in log["properties"]:
        if int(prop["min_examples"]) < 1:
            failures.append((prop["id"], "min_examples_floor"))
            continue
        for seed in prop["seeds"]:
            cmd = [
                sys.executable, "-m", "pytest",
                prop["nodeid"],
                "-q",
                f"--hypothesis-seed={seed}",
            ]
            proc = subprocess.run(cmd, capture_output=True, text=True)
            if proc.returncode != 0:
                failures.append((prop["id"], seed, proc.returncode))
    return {"ok": not failures, "failures": failures}

Label the --hypothesis-seed flag as environment-specific. If your runner uses a different seed switch, put that switch in the log too. Do not let the agent edit the log to match a weaker command.

Two extra rules keep the tape honest:

1. Seeds only grow when a property test fails on main and a human copies the seed into the log.
2. min_examples may stay or rise. A diff that lowers it is scored reject, even when every replayed seed passes.

That second rule is the one agents hit first. Quiet generators look like refactors.

3. Freeze flakes by failure signature, not by test name

Test names are cheap. An agent can rename test_timeout_on_slow_provider to test_provider_latency_ok and mark the old node xfailed. Your flake list then points at a ghost. Hash the failure instead.

Build a signature from assertion kind plus a normalized message. Strip digits, hex addresses, and timestamps. Keep the template. Store SHA-256 of that template in a human-owned file.

import hashlib, re

def normalize_message(msg: str) -> str:
    msg = re.sub(r"\d+", "N", msg)
    msg = re.sub(r"0x[0-9a-fA-F]+", "HEX", msg)
    msg = re.sub(r"\d{4}-\d{2}-\d{2}T[^\s]+", "TS", msg)
    return " ".join(msg.split())

def signature(kind: str, msg: str) -> str:
    template = f"{kind}|{normalize_message(msg)}"
    return hashlib.sha256(template.encode()).hexdigest(), template

flake_signatures.json is a freeze, not a skip list. Humans add rows. Agents do not. If pytest emits a failure whose signature is absent, the scorecard fails. If a test is skipped or xfailed without a matching signature, the scorecard fails. If the patch deletes a row, the scorecard fails.

Collect signatures from a JUnit or pytest JSON report so the scorer does not parse stdout by folklore:

pytest tests/ --tb=no --junitxml=build/junit.xml
python scorecard.py --junit build/junit.xml --freeze flake_signatures.json

Name changes then become irrelevant. The same assertion template hashes to the same freeze row. That is the point.

Assemble one scorecard, then gate on it

Do not fold these checks into the agent's pytest command. Run them as a sibling job that reads the three files from the merge-base and the patched tree. Emit JSON. Humans read the table. Machines read ok.

# proposed CLI: python scorecard.py --base $BASE --head $HEAD
def score(base: dict, head: dict, junit_sigs: list[str]) -> dict:
    fixture = score_fixtures(...)  # as above, on head testdata vs base lock
    seeds = replay_seeds(Path("property_seeds.json"))
    freeze = json.loads(Path("flake_signatures.json").read_text())
    allowed = {row["sha256"] for row in freeze["signatures"]}
    new_sigs = sorted(set(junit_sigs) - allowed)
    freeze_ok = not new_sigs and freeze == json.loads(
        (base_dir / "flake_signatures.json").read_text()
    )
    ok = fixture["ok"] and seeds["ok"] and freeze_ok
    return {
        "ok": ok,
        "fixture": fixture,
        "seeds": seeds,
        "new_failure_signatures": new_sigs,
    }

A practical CI shape:

1. Checkout the merge-base and capture the three contract files.
2. Apply the agent patch in a second worktree.
3. Fail fast if the patch touched those three files.
4. Run fixture digest comparison.
5. Replay seeds.
6. Run the existing unit suite with JUnit output.
7. Score failure signatures against the freeze.
8. Publish one JSON artifact. Do not collapse it into a single emoji.

Step 3 is load-bearing. If the agent can rewrite the contracts, the other steps are theatre.

Where patch generation belongs

Keep the generate loop off the lock files. A separate workspace can propose diffs all day. The scorecard should still run against a tree that cannot write FIXTURE_LOCK.json, property_seeds.json, or flake_signatures.json.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you iterate candidate patches with MonkeyCode's free model access and free server option, point that loop at an isolated worktree and copy only the production sources plus tests into the scorer. The free server does not replace the lock files. It only makes retry cheaper while the contracts stay in your repo.

CODEOWNERS on those three paths is enough for most teams. No extra product is required for the gate itself.

Limitations, and who should not use this

This plan assumes you already have some property checks and some committed fixtures. If your suite is 100% example-based with no testdata directory, start by extracting two invariants, not by adding a freeze file. A freeze of nothing encodes nothing.

It also assumes failures can be normalized. Time-of-day clocks, live network errors, and unordered log lines will thrash signatures unless you stub them. Do not freeze raw integration noise. Stub first, then hash.

Do not use this as the only gate on safety-critical code. Seed replay catches regressions you have already seen. It does not invent the next counterexample. Humans still raise min_examples and still rotate fixtures.

Do not use it if agents are allowed to own tests/. Split write sets. Production code and new tests can be writable. Digests, seeds, and signatures stay read-only to the agent. If that split is politically impossible, the scorecard will be edited until it passes.

Finally, the snippets above are unlabeled as executed fleet data because they are not. Wire them to your runner, keep the JSON contracts in git, and treat ok: false as a merge block. The green unit job can remain a signal. It should not remain the only one.