Score Agent Diffs With a Claim Matrix, Not a Green Check

A green CI job is a count of passing functions. It is not a map of the claims an agent just made. Agent patches fail in a specific pattern: they add tests that restate the new code, they leave I/O unbounded, and they treat a flake as noise instead of a missing lock. The merge rule is therefore mechanical. Every behavioral claim in the diff must bind to three checks — a fixture, a property, and a freeze rule — or the patch stays red even if pytest prints a wall of dots.

Pass count hides that gap. A claim matrix makes it visible.

Why pass count is the wrong metric

Agent-authored tests optimize for green. They assert the return value the patch just hardcoded. They skip the boundary the model did not sample. They retry a network call until the suite lucks into 200. None of that shows up in 14 passed in 0.81s.

A claim matrix is a small table. Rows are claims extracted from the production delta. Columns are the three checks. Empty cells are merge blockers. The file is reviewable on one screen and strict enough for a script to fail the job.

Claim	Fixture	Property	Freeze rule
C1 refund never exceeds captured amount	fixtures/refund_cap.json	prop_refund_cap	freeze:refund_cap
C2 timeout path returns 504, not 200	fixtures/timeout_504.json	prop_timeout_status	freeze:timeout_status
C3 idempotency key is stable across retries	fixtures/idempotency.json	prop_idempotency_key	freeze:idempotency

If C2 has a fixture and no property, the scorer exits non-zero. A green example test does not fill the cell.

The three checks, defined

1. Fixture. A recorded input/output pair the production code must match. No live clock. No live network. No golden file generated from the same patch that is under review.
2. Property. An invariant that stays true across a generated input set. assert fn(2) == 4 is an example. assert fn(x) >= 0 for all x in the stated domain is a property.
3. Freeze rule. A named check that failed once in the recorded window cannot vote green until a human reopens it. The freeze is a ledger row, not a comment and not pytest.mark.skip.

The three checks cover different failure classes. Fixtures catch drift against known bytes. Properties catch the unsampled neighborhood. Freeze rules stop flakes from laundering a merge.

Artifact: claim matrix as YAML

Keep the matrix next to the patch, not in a wiki. The scorer does not parse English. It checks that each claim_id has three named artifacts on disk and that no freeze row is in blocked status.

# claims/refund-service.yaml
patch: "refund-cap-agent-diff"
claims:
  - id: C1
    statement: "refund amount never exceeds captured amount"
    fixture: tests/fixtures/refund_cap.json
    property: tests/properties/test_refund_properties.py::test_prop_refund_cap
    freeze: freeze/ledger.json#refund_cap
  - id: C2
    statement: "upstream timeout maps to 504, never 200"
    fixture: tests/fixtures/timeout_504.json
    property: tests/properties/test_refund_properties.py::test_prop_timeout_status
    freeze: freeze/ledger.json#timeout_status
  - id: C3
    statement: "idempotency key is stable across retries"
    fixture: tests/fixtures/idempotency.json
    property: tests/properties/test_refund_properties.py::test_prop_idempotency_key
    freeze: freeze/ledger.json#idempotency

Label the YAML as a contract, not as documentation. If a reviewer cannot point at a cell, the claim is unbound.

1. Extract claims from the production delta

Test names lie. test_handles_timeout can assert True. Claims come from the code that changed: new branches, new status codes, new persisted fields, new comparison operators.

git diff --stat origin/main...HEAD
git diff -U0 origin/main...HEAD -- '*.py' ':!tests/*'

Walk the hunks with a short checklist:

1. List every new return, raise, and persisted field.
2. Write one claim sentence per behavior, in the present tense, with a bound ("never exceeds", "maps to 504", "stable across retries").
3. Reject claims that only restate a function name.
4. Assign C1..Cn. Do not reuse IDs across patches.

A useful claim names a forbidden outcome. "Refunds work" is not a claim. "Refund amount never exceeds captured amount" is.

2. Lock I/O with human-owned fixtures

Fixtures are bytes the production path must reproduce. The agent may propose a fixture. A human writes it to tests/fixtures/ after checking it against a known source: a prior production log, a recorded HTTP cassette owned by the team, or a hand-built payload.

{
  "id": "refund_cap",
  "input": {"captured_cents": 1999, "refund_cents": 5000},
  "output": {"refunded_cents": 1999, "status": "capped"}
}

# tests/test_refund_fixtures.py
import json
from pathlib import Path
from refund_service import apply_refund

FIXTURE = json.loads(Path("tests/fixtures/refund_cap.json").read_text())

def test_refund_cap_fixture():
    got = apply_refund(FIXTURE["input"])
    assert got == FIXTURE["output"]

Two rejection rules keep this layer honest. A fixture generated by running the new function and dumping its output is a tautology; drop it. A fixture that hits the network is not a fixture; record it first, then point the test at the file.

3. Attach one property per claim

Properties quantify over a domain. They do not pin a single example. Hypothesis is sufficient for most numeric and string domains; the point is the invariant, not the library.

# tests/properties/test_refund_properties.py
from hypothesis import given, strategies as st
from refund_service import apply_refund, status_for_timeout, idempotency_key

@given(
    captured=st.integers(min_value=0, max_value=10_000_000),
    refund=st.integers(min_value=0, max_value=10_000_000),
)
def test_prop_refund_cap(captured, refund):
    got = apply_refund({"captured_cents": captured, "refund_cents": refund})
    assert 0 <= got["refunded_cents"] <= captured

@given(delay_ms=st.integers(min_value=0, max_value=60_000))
def test_prop_timeout_status(delay_ms):
    code = status_for_timeout(delay_ms)
    if delay_ms >= 10_000:
        assert code == 504
    else:
        assert code != 504 or delay_ms >= 10_000
    assert code != 200 or delay_ms < 10_000

@given(
    key=st.text(min_size=1, max_size=64),
    n=st.integers(min_value=2, max_value=8),
)
def test_prop_idempotency_key(key, n):
    issued = [idempotency_key(key) for _ in range(n)]
    assert len(set(issued)) == 1

If a property needs the live clock or a vendor sandbox, it does not belong in this layer. Split the impurity into a fixture. Keep the property pure.

A property that only calls the new helper and asserts result == helper(...) is another tautology. Delete it. The matrix cell stays empty until a real invariant exists.

4. Write freeze rules as a ledger

Skipping a flaky test is how agent diffs sneak through. The freeze ledger is the opposite: a failed check loses its vote until a human sets status back to open with a reason.

{
  "window_runs": 20,
  "entries": {
    "refund_cap": {"status": "open", "fails_in_window": 0},
    "timeout_status": {"status": "blocked", "fails_in_window": 3, "last_fail": "2026-09-12T21:14:00Z"},
    "idempotency": {"status": "open", "fails_in_window": 0}
  }
}

Update the ledger from CI, not from memory:

pytest tests/properties tests/test_refund_fixtures.py --junitxml=build/junit.xml
python tools/score_claims.py \
  --matrix claims/refund-service.yaml \
  --ledger freeze/ledger.json \
  --junit build/junit.xml

The scorer's freeze policy is narrow. One failure inside the window flips open to blocked. Blocked entries fail the patch. Retries do not clear the row. Only a human edit does.

5. Score the matrix before shared CI

The scorer is the merge gate. It is deliberately boring: path existence, node IDs, ledger status, and a non-empty claim list.

# tools/score_claims.py — example gate, not a published package
from __future__ import annotations

import argparse
import json
from pathlib import Path

import yaml


def load_matrix(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not data or not data.get("claims"):
        raise SystemExit("matrix has zero claims")
    return data


def score(matrix: dict, ledger: dict) -> list[str]:
    errors: list[str] = []
    entries = ledger.get("entries", {})
    for claim in matrix["claims"]:
        cid = claim["id"]
        for key in ("fixture", "property", "freeze"):
            if not claim.get(key):
                errors.append(f"{cid}: missing {key}")
        fixture = Path(claim["fixture"])
        if not fixture.is_file():
            errors.append(f"{cid}: fixture not on disk: {fixture}")
        freeze_id = claim["freeze"].rsplit("#", 1)[-1]
        row = entries.get(freeze_id)
        if row is None:
            errors.append(f"{cid}: freeze id {freeze_id!r} absent from ledger")
        elif row.get("status") == "blocked":
            errors.append(f"{cid}: freeze {freeze_id!r} is blocked")
    return errors


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--matrix", type=Path, required=True)
    p.add_argument("--ledger", type=Path, required=True)
    args = p.parse_args()
    errors = score(load_matrix(args.matrix), json.loads(args.ledger.read_text()))
    if errors:
        print("CLAIM MATRIX FAILED")
        print("\n".join(errors))
        raise SystemExit(1)
    print("CLAIM MATRIX OK", args.matrix)


if __name__ == "__main__":
    main()

Run order matters. Score the matrix first. Then run fixtures. Then run properties. Then write the ledger. A property suite that is green while C2 is unbound is a failed gate, not a partial success.

Draft properties in a scratch environment, accept them by hand

Generating candidate invariants is cheap. Accepting them is not. A local or remote scratch box is the right place to propose properties from the diff, because that work should not consume the shared runner and should not write into tests/properties/ until a reviewer keeps a candidate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is enough to draft candidate properties from a production diff. The free server option is enough to run score_claims.py plus the Hypothesis suite off the shared CI pool. Neither step makes the model an oracle. The matrix on disk is the oracle, and only human-accepted rows belong there.

A tight loop looks like this:

1. Paste the production hunks, not the agent-written tests, into the scratch session.
2. Ask for invariants that name a forbidden outcome and a domain.
3. Drop any candidate that references the new helper as its own expected value.
4. Run the survivors against the fixtures on the free server.
5. Copy only the passing, reviewer-owned properties into the repo.

If a candidate needs credentials, production data, or a live vendor, stop. That is a fixture problem, not a property problem.

Decision table

Symptom	Wrong response	Matrix response
Agent adds 12 tests, all examples of the new branch	Merge on green	Extract claims; examples do not fill property cells
Fixture dumped from the new function	Keep it as "regression coverage"	Delete; replace with bytes from a known source
Property calls the network	Raise retries, pin a seed	Split I/O into a fixture; keep the property pure
Same test failed 3 of last 20 runs	Rerun until green	Ledger blocked; patch cannot use that check to merge
Claim sentence restates a function name	Add more assertions	Rewrite the claim or drop the row
Scratch model proposes 9 properties	Commit all of them	Accept the ones that name a bound; discard the rest

Limitations

The matrix does not measure correctness in the abstract. It measures whether stated claims are bound. Garbage claims produce a green matrix. Review the sentences.

Hypothesis will not invent a domain you did not write. An unbounded st.text() on a parser that only accepts ISO-4217 codes wastes runs and hides the real invariant. Tighten the strategy to the contract.

The freeze ledger can stall a team that never reopens rows. Treat blocked as a bug, with an owner, not as a permanent skip list. A ledger that only grows is a graveyard.

This gate also assumes the production delta is small enough to enumerate. A 4,000-line generated rewrite will not yield a honest C1..Cn list in one sitting. Split the patch until claims fit on one screen.

Who should not use this

Do not install this gate on throwaway spikes, on UI copy tweaks, or on patches with no behavioral contract. Do not use it as a substitute for a security review. Do not let an agent edit freeze/ledger.json or claims/*.yaml. Those two files are human-owned, same as the fixtures.

Teams without a recorded I/O source should not fake fixtures. An empty tests/fixtures/ directory is better than a dump of the code under test.

If the only available execution environment cannot isolate network and clock, skip the property layer rather than lying about purity. A fixture-only matrix is weaker. It is still more honest than a green example suite.

The claim matrix is a scoring device. Use it to keep agent diffs from buying a merge with their own tests. When a scratch model and a free server help you draft and run that score off CI, they are optional tools around the same rule: unbound claims keep the patch red.