Run Only the Invariants the Diff Can Break

A full green suite is a weak merge signal after an agent patch. It is also a noisy one. Untested relations still break. Unrelated flakes still block the queue.

Scope a matrix of invariants to the files the diff touched. Lock the fixtures those checks read. Keep every property counterexample in an append-only seed ledger. Freeze flakes with an expiry instead of deleting them.

The rest of this article is a concrete CI gate. Examples are labeled workflow code, not production telemetry.

What “scoped” means

Most suites were built for human pull requests. An agent can touch three modules and still execute the entire suite, including tests that fail one run in twenty.

That waste hides the real miss. The patch can invert a round-trip, reorder a stable sort, or rewrite a parser span while every named test stays green.

A scoped invariant matrix answers one question: given this diff, which relations are now at risk, which inputs are allowed, and which failing tests are frozen rather than “fixed”?

The four files

Keep these next to the suite:

1. invariants.yaml — module glob to properties, fixture sets, and budgets
2. fixtures.lock — SHA-256 of every fixture file a property may read
3. seeds.jsonl — append-only counterexamples, one JSON object per line
4. flake_freeze.yaml — test node ids, reason, expiry date, owner

The gate reads git diff --name-only, selects rows, verifies hashes, replays seeds, then runs a time-bounded property loop. It fails closed on a missing row.

1. Classify the diff first

Do not start the suite yet. Name the paths.

git diff --name-only origin/main...HEAD > /tmp/touched.txt

Split them into three buckets:

1. Production code (src/, lib/, pkg/)
2. Tests (tests/, *_test.py)
3. Fixtures (testdata/, tests/fixtures/)

An agent patch that edits bucket 2 or 3 is not extra coverage by default. It is a change to the spec. Treat it as a second diff that needs its own verdict.

2. Select invariants for touched production files

invariants.yaml holds relations the named suite does not encode well.

version: 1
rows:
  - id: json_roundtrip
    globs: ["src/codec/**", "src/json_*.py"]
    properties: ["prop_roundtrip", "prop_reject_trailing"]
    fixtures: ["testdata/codec/"]
    budget:
      examples: 80
      seconds: 20
  - id: stable_sort_key
    globs: ["src/index/**"]
    properties: ["prop_sort_is_stable", "prop_key_is_total"]
    fixtures: ["testdata/index/keys.txt"]
    budget:
      examples: 50
      seconds: 15
  - id: parser_span
    globs: ["src/parse/**"]
    properties: ["prop_span_covers_lexeme", "prop_error_offset_in_range"]
    fixtures: ["testdata/parse/"]
    budget:
      examples: 100
      seconds: 25

If a production path matches no row, the gate fails. That is intentional. A module without invariants is not low risk. It is unmeasured.

Candidate rows can be drafted by a model. They still need a human to reject tautologies such as output == output.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is a workable place to draft those candidate rows from module docstrings and public function names. The free server option is a workable place to run the matrix on a box that is not your laptop. Neither step replaces the lockfiles or the merge rule below.

3. Lock fixtures before properties run

Properties that read mutable golden files are not properties. They are snapshots with extra steps.

# fixture_lock.py — labeled example, not a published benchmark
import hashlib, json, pathlib, sys

ROOT = pathlib.Path("testdata")
LOCK = pathlib.Path("fixtures.lock")

def digest(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def walk() -> dict[str, str]:
    out = {}
    for p in sorted(ROOT.rglob("*")):
        if p.is_file():
            out[p.as_posix()] = digest(p)
    return out

def main(argv: list[str]) -> int:
    current = walk()
    if argv[1:] == ["--write"]:
        LOCK.write_text(json.dumps(current, indent=2, sort_keys=True) + "\n")
        print(f"wrote {len(current)} fixture hashes")
        return 0
    locked = json.loads(LOCK.read_text())
    if current != locked:
        added = sorted(set(current) - set(locked))
        removed = sorted(set(locked) - set(current))
        changed = sorted(
            k for k in current if k in locked and current[k] != locked[k]
        )
        print("fixture lock mismatch")
        for label, items in (
            ("added", added),
            ("removed", removed),
            ("changed", changed),
        ):
            for item in items:
                print(f"  {label}: {item}")
        return 2
    print(f"ok {len(current)} fixtures")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Commands:

python fixture_lock.py --write   # only on an explicit fixture PR
python fixture_lock.py           # every agent-patch CI run

A fixture change must land in its own commit with a reason. Bundling fixture rewrites with a behavior patch is how an agent “fixes” a property: it edits the expected bytes.

4. Replay seeds, then generate

A property without memory repeats old misses. Store the shrinking result.

{"id":"json_roundtrip","prop":"prop_roundtrip","seed":"{\"a\":1,\"a\":2}","note":"duplicate keys","added":"2026-09-06","status":"active"}

Gate order:

1. Replay every seed whose id is in the selected rows. Failure is a regression.
2. Run new examples up to budget.examples or budget.seconds.
3. On a new counterexample, append one line to seeds.jsonl. Fail the job.
4. Reject a diff that deletes a seed line unless that seed is replayed and marked retired with a reviewer name.

# seed_ledger.py — labeled example
import json, time
from pathlib import Path

LEDGER = Path("seeds.jsonl")

def load_active():
    rows = []
    if not LEDGER.exists():
        return rows
    for line in LEDGER.read_text().splitlines():
        if not line.strip():
            continue
        row = json.loads(line)
        if row.get("status") != "retired":
            rows.append(row)
    return rows

def append_failure(inv_id: str, prop: str, seed: str, note: str) -> None:
    rec = {
        "id": inv_id,
        "prop": prop,
        "seed": seed,
        "note": note,
        "added": time.strftime("%Y-%m-%d"),
        "status": "active",
    }
    with LEDGER.open("a") as f:
        f.write(json.dumps(rec, ensure_ascii=False) + "\n")

Keep property bodies small. Prefer relations over oracles.

# props.py — labeled examples, not measured hit rates
def prop_roundtrip(encode, decode, obj):
    assert decode(encode(obj)) == obj

def prop_sort_is_stable(sort_fn, items):
    labeled = list(enumerate(items))
    out = sort_fn(labeled, key=lambda x: x[1])
    for i in range(1, len(out)):
        if out[i][1] == out[i - 1][1]:
            assert out[i][0] > out[i - 1][0]

def prop_span_covers_lexeme(parse, src: str):
    tree = parse(src)
    for node in tree.tokens:
        assert src[node.start:node.end] == node.lexeme

Metamorphic shapes help when you have no golden output. Permute equal keys. Encode twice. Parse concatenated documents. Compare two implementations of the same contract.

A cheap runner can honor the YAML budget without a framework:

# budget.py — labeled example
import time, traceback

def run_prop(prop, gen, examples: int, seconds: int):
    t0 = time.monotonic()
    for i in range(examples):
        if time.monotonic() - t0 > seconds:
            return {"status": "budget", "n": i}
        sample = gen(i)
        try:
            prop(sample)
        except AssertionError as exc:
            return {
                "status": "fail",
                "n": i,
                "sample": sample,
                "err": traceback.format_exc(limit=2),
            }
    return {"status": "pass", "n": examples}

Stop when the budget hits. Do not treat a budget stop as proof. Treat it as “no counterexample in this window.”

5. Freeze flakes with an expiry, do not delete them

Flakes destroy the matrix. They also invite the worst agent edit: pytest.mark.skip, a looser assertion, or deletion.

# flake_freeze.yaml
frozen:
  - nodeid: tests/test_index.py::test_build_large
    reason: "order depends on wall clock in tz=UTC-4"
    expires: "2026-09-20"
    owner: "platform"
    last_seen: "2026-09-05"

Rules:

1. A frozen node id is collected out of the agent merge job. It is not deleted from the repo.
2. Expiry is mandatory. A freeze without a date is a silent skip and fails the gate.
3. On expiry, the test returns to the suite. If it still flakes, retick with a new reason, not an infinite skip.
4. An agent patch that removes a freeze entry, adds skip, or weakens assert on that node id is a reject unless the commit contains a freeze-triage: trailer and a reviewer name.

Minimal check on the test diff:

# test_diff_policy.py — labeled example
import re, subprocess, sys

WEAKEN = re.compile(
    r"^\+.*(pytest\.mark\.(skip|xfail)|assert True|time\.sleep\()",
    re.M,
)
DELETE_TEST = re.compile(r"^-def test_", re.M)
TRAILER = "freeze-triage:"

def test_diff(base: str) -> str:
    return subprocess.check_output(
        ["git", "diff", f"{base}...HEAD", "--", "tests"],
        text=True,
    )

def trailers() -> str:
    return subprocess.check_output(
        ["git", "log", "-1", "--format=%B"],
        text=True,
    )

def main() -> int:
    diff = test_diff("origin/main")
    if not diff.strip():
        print("no test diff")
        return 0
    flagged = bool(WEAKEN.search(diff) or DELETE_TEST.search(diff))
    if flagged and TRAILER not in trailers():
        print("test diff changes skip/xfail/sleep/assert True or deletes test_*")
        print("reject unless freeze-triage is in the commit body")
        return 2
    print("test diff has no freeze-policy violations")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

This is a coarse net. It will flag legitimate refactors. That is cheaper than a skip that ships.

Load expiry in CI the same way you load the lockfile. A freeze past its date must re-enter the suite or the job fails.

# freeze_expiry.py — labeled example
from datetime import date
import sys, yaml

def main(path: str, today: str) -> int:
    doc = yaml.safe_load(open(path))
    today_d = date.fromisoformat(today)
    stale = []
    for row in doc.get("frozen", []):
        if "expires" not in row:
            print(f"missing expires: {row.get('nodeid')}")
            return 2
        if date.fromisoformat(row["expires"]) < today_d:
            stale.append(row["nodeid"])
    if stale:
        print("expired freeze entries, return them to the suite:")
        for nodeid in stale:
            print(f"  {nodeid}")
        return 2
    print("freeze dates ok")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], sys.argv[2]))

python freeze_expiry.py flake_freeze.yaml 2026-09-06

6. Assemble the gate

Numbered CI order:

1. git diff --name-only origin/main...HEAD
2. Fail if a production path has no invariants.yaml row
3. python fixture_lock.py
4. python test_diff_policy.py
5. python freeze_expiry.py flake_freeze.yaml "$TODAY"
6. Replay active seeds for selected rows
7. Run property budgets for selected rows only
8. Append new counterexamples and fail the job

python matrix_gate.py --base origin/main
python fixture_lock.py
python test_diff_policy.py
python freeze_expiry.py flake_freeze.yaml 2026-09-06
pytest -q tests

Keep the full suite on a slower schedule if you want. Do not use it as the agent merge gate. The merge gate is the matrix plus the three lockfiles.

Decision table

Observation	Verdict	Why
Production file with no matrix row	reject	unmeasured module
Fixture hash change in the same commit as behavior	reject	spec moved with code
Seed replay fails	reject	regression
New property counterexample	reject, append seed	new miss, now remembered
Seed line deleted without retired	reject	memory wipe
Flake deleted or skipped by the agent	reject	freeze policy
Freeze past expires still skipped	reject	freeze became a skip
Selected properties pass, fixtures match, seeds hold	accept	scoped evidence

Limitations

The matrix is not a proof. A passing prop_roundtrip does not imply parser safety, Unicode correctness, or performance.

Budgets hide rare bugs. Eighty examples will not find a 1-in-10,000 collision. Raise the budget on modules that already produced seeds.

Freeze files hide real races. An expiry exists so the hide is temporary. Teams that keep extending the same date are running a skip list.

Fixture locks fight legitimate format upgrades. Put those upgrades in a dedicated PR that rewrites hashes and retires related seeds in one review.

Models draft tautologies. len(x) >= 0 will pass forever. Review every new row as if it were production code. Discard any property that cannot fail on a wrong patch.

The gate assumes origin/main is the merge base and that tests live under a predictable prefix. Monorepos need per-package lockfiles. Binary fixtures need the same hash path; do not special-case them as “too large to lock.”

Who should not use this

Do not use this as the only control on safety-critical changes. Invariants here are tests, not formal proofs.

Do not use it on a repository where agents may rewrite tests in the same patch with no second review. The policy in test_diff_policy.py will become noise, then get disabled.

Do not use it if you cannot store seeds.jsonl in version control. A ledger that lives only on one runner is not a ledger.

Skip the matrix for throwaway prototypes with no merge queue. The overhead is for patches you might keep.

Close

Agent patches fail in the gaps between named tests. Name the gaps. Scope them to the diff. Lock the bytes they read. Remember the inputs that hurt. Freeze flakes in public, with a date.

If you already generate patches from a free model and run jobs on a free server, put the matrix on that path and keep the merge rule in CI. The useful part is the four files, not the model that drafted the first YAML row.