Don't Merge the Patch Until the Contract Diff Is Non-Weakening

A green test run is not a merge signal for an agent patch. The merge signal is that the patch left a three-layer contract unchanged or strictly stronger: invariant properties still hold, fixture digests still match, and the freeze ledger did not gain extra time, extra names, or a missing owner.

Agents optimize for the scoreboard they can see. If the scoreboard is "tests pass," deleting an assertion, rewriting a fixture, or extending a flaky freeze all look like success. The contract has to be a first-class artifact that CI diffs against HEAD. Weakening that artifact is a reject, even when every remaining test is green.

This article is a proposed workflow and a runnable checker, not a report of a production incident. Treat the commands and files as a template to adapt. Do not treat the sample numbers in comments as measured results.

What "non-weakening" means

A patch is non-weakening when all three hold at once.

1. Properties may be added. They may not be deleted, renamed away, or have their predicates replaced with True.
2. Fixtures are content-addressed. A digest change requires a human-owned unlock token in the same commit, or the gate fails.
3. Freezes are budgeted. A freeze entry needs an owner, a reason, an expiry, and a remaining rerun count. The patch cannot raise that budget.

Those three rules are independent. A property that still passes on a rewritten fixture is not evidence. A locked fixture that is never executed because a freeze swallowed the job is not evidence either.

The contract file

Keep the contract next to the tests. YAML is enough. The checker should refuse unknown keys so an agent cannot hide policy in comments.

# tests/agent_contract.yaml
version: 1
properties:
  - id: parse_rejects_empty
    module: tests.properties.test_parse
    name: test_empty_input_raises
    min_examples: 40
  - id: roundtrip_stable
    module: tests.properties.test_codec
    name: test_decode_encode_identity
    min_examples: 80
fixtures:
  - id: invoice_v3
    path: tests/fixtures/invoice_v3.json
    sha256: 8f3a1c0e0b77d2a19c4e6b1f0a9d5c2e7b8a4f1c0d6e9b3a5c7d1e0f2a4b6c8d
    unlock: null
freezes:
  - id: test_report_render_flaky
    owner: "oncall-payments"
    reason: "clock skew in renderer; tracked as PAY-4412"
    expires: "2026-09-22T00:00:00Z"
    remaining_reruns: 2
max_active_freezes: 3

unlock is null in the committed contract. A human who intends to rotate a fixture sets unlock to a short signed note in a follow-up commit that is reviewed like any other policy change. The agent branch does not get to fill that field.

Layer 1: properties that cannot be tautologized

A property is not "a test that uses Hypothesis." It is a named invariant with a minimum example count and a predicate the checker can import. If the collected test body is assert True, or the example count drops below min_examples, that is a contract miss, not a pass.

# tools/contract_check.py
from __future__ import annotations

import ast
import hashlib
import importlib
import json
from datetime import datetime, timezone
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
CONTRACT_PATH = ROOT / "tests" / "agent_contract.yaml"
HEAD_CONTRACT_PATH = ROOT / ".git" / "contract_head.yaml"  # copied in CI

ALWAYS_TRUE = {
    ast.Constant(value=True).__class__: True,
}


def load_contract(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict) or data.get("version") != 1:
        raise SystemExit("contract version mismatch")
    return data


def source_of(module_name: str, test_name: str) -> ast.FunctionDef:
    mod = importlib.import_module(module_name)
    path = Path(mod.__file__)
    tree = ast.parse(path.read_text())
    for node in tree.body:
        if isinstance(node, ast.FunctionDef) and node.name == test_name:
            return node
    raise SystemExit(f"missing property {module_name}:{test_name}")


def predicate_is_vacuous(fn: ast.FunctionDef) -> bool:
    for node in ast.walk(fn):
        if isinstance(node, ast.Assert):
            test = node.test
            if isinstance(test, ast.Constant) and test.value is True:
                return True
            if isinstance(test, ast.Name) and test.id == "True":
                return True
    asserts = [n for n in ast.walk(fn) if isinstance(n, ast.Assert)]
    return len(asserts) == 0

The vacuity check is syntactic on purpose. It will not catch a clever assert 1 == 1. That is why min_examples exists, and why properties should encode domain relations, not equality with a captured constant.

A useful property talks about a family of inputs. The example below is labeled as a template. Replace parse_invoice with the real entry point.

# tests/properties/test_parse.py
import pytest
from hypothesis import given, settings, strategies as st

from payments.parser import ParseError, parse_invoice


@settings(max_examples=40, deadline=None)
@given(st.just(""))
def test_empty_input_raises(blob: str) -> None:
    with pytest.raises(ParseError):
        parse_invoice(blob)


@settings(max_examples=80, deadline=None)
@given(st.binary(min_size=1, max_size=256))
def test_decode_encode_identity(blob: bytes) -> None:
    try:
        obj = parse_invoice(blob)
    except ParseError:
        return
    assert parse_invoice(obj.to_bytes()) == obj

Empty input must still fail. Round-trip must hold only for inputs the parser already accepted. That split matters. A single property that demands every blob round-trip will force the agent to accept garbage, which is a different class of contract break.

Layer 2: fixtures as digests, not files that happen to exist

Presence is not integrity. An agent that rewrites invoice_v3.json to match a buggy parser still "has a fixture." Hash the bytes. Compare them to the contract. Ignore mtime.

def fixture_digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def check_fixtures(contract: dict) -> list[str]:
    errors = []
    for item in contract["fixtures"]:
        path = ROOT / item["path"]
        if not path.is_file():
            errors.append(f"missing fixture {item['id']}")
            continue
        digest = fixture_digest(path)
        if digest != item["sha256"]:
            if item.get("unlock"):
                errors.append(
                    f"fixture {item['id']} digest changed with unlock set; "
                    "human review required, agent branch cannot self-unlock"
                )
            else:
                errors.append(
                    f"fixture {item['id']} digest mismatch: "
                    f"expected {item['sha256'][:12]} got {digest[:12]}"
                )
    return errors

Rotate a fixture in two commits. First commit: human sets unlock and the new digest together. Second commit: unlock returns to null. An agent patch that tries to do both in one step is still a policy change and should fail the monotonicity diff below.

Layer 3: a freeze budget, not an infinite skip

Flakes happen. A freeze is a time-boxed exception with an owner and a remaining rerun count. It is not @pytest.mark.skip. The checker should fail when expires is in the past, when remaining_reruns is zero, or when the active freeze count exceeds max_active_freezes.

def check_freezes(contract: dict, now: datetime) -> list[str]:
    errors = []
    active = contract.get("freezes") or []
    if len(active) > contract["max_active_freezes"]:
        errors.append("freeze ledger exceeds max_active_freezes")
    for item in active:
        if not item.get("owner") or not item.get("reason"):
            errors.append(f"freeze {item.get('id')} missing owner or reason")
            continue
        expires = datetime.fromisoformat(item["expires"].replace("Z", "+00:00"))
        if expires <= now:
            errors.append(f"freeze {item['id']} expired at {item['expires']}")
        if int(item["remaining_reruns"]) <= 0:
            errors.append(f"freeze {item['id']} has no reruns left")
    return errors

Pair this with a pytest plugin that consults the same ledger. If a test name is not in the ledger, it runs. If it is in the ledger, it may rerun up to remaining_reruns and then fail the job. Do not auto-decrement in the working tree. Decrement in CI by writing a machine report, then require a human to edit the contract.

pytest tests/ --tb=short -q
python tools/contract_check.py --against HEAD

The monotonicity diff

Passing today's contract is necessary and not sufficient. The patch must not make yesterday's contract easier. Copy the contract from HEAD in CI and compare sets.

def ids(items: list[dict]) -> set[str]:
    return {item["id"] for item in items}


def check_monotonic(head: dict, current: dict) -> list[str]:
    errors = []
    lost = ids(head["properties"]) - ids(current["properties"])
    if lost:
        errors.append(f"removed properties: {sorted(lost)}")
    for old in head["properties"]:
        new = next((p for p in current["properties"] if p["id"] == old["id"]), None)
        if new and int(new["min_examples"]) < int(old["min_examples"]):
            errors.append(f"property {old['id']} min_examples decreased")
    for old in head["fixtures"]:
        new = next((f for f in current["fixtures"] if f["id"] == old["id"]), None)
        if new is None:
            errors.append(f"removed fixture {old['id']}")
        elif new["sha256"] != old["sha256"] and not new.get("unlock"):
            errors.append(f"fixture {old['id']} digest changed without unlock")
    head_freezes = {f["id"]: f for f in head.get("freezes") or []}
    for new in current.get("freezes") or []:
        old = head_freezes.get(new["id"])
        if old is None:
            continue
        if new["expires"] > old["expires"]:
            errors.append(f"freeze {new['id']} expiry extended")
        if int(new["remaining_reruns"]) > int(old["remaining_reruns"]):
            errors.append(f"freeze {new['id']} rerun budget increased")
    if current["max_active_freezes"] > head["max_active_freezes"]:
        errors.append("max_active_freezes increased")
    return errors

Wire it so a non-zero exit is the only merge blocker you need from this gate:

def main() -> None:
    now = datetime.now(timezone.utc)
    current = load_contract(CONTRACT_PATH)
    errors = []
    errors.extend(check_fixtures(current))
    errors.extend(check_freezes(current, now))
    for prop in current["properties"]:
        fn = source_of(prop["module"], prop["name"])
        if predicate_is_vacuous(fn):
            errors.append(f"vacuous property {prop['id']}")
    if HEAD_CONTRACT_PATH.exists():
        head = load_contract(HEAD_CONTRACT_PATH)
        errors.extend(check_monotonic(head, current))
    if errors:
        raise SystemExit("\n".join(errors))
    print("contract: non-weakening")


if __name__ == "__main__":
    main()

In GitHub Actions, materialize HEAD's file before the checker runs:

- name: Snapshot HEAD contract
  run: git show HEAD:tests/agent_contract.yaml > .git/contract_head.yaml
- name: Run tests and contract
  run: |
    pytest tests/ --tb=short -q
    python tools/contract_check.py

Decision table

Use this table when a patch looks green and still feels wrong.

Observation	Layer	Gate result	Human action
Test body became assert True	property	reject	restore invariant
max_examples dropped below contract	property	reject	restore count
Fixture bytes changed, unlock null	fixture	reject	keep old bytes or review unlock
Fixture bytes changed, unlock set on agent branch	fixture	reject	move unlock to a human commit
New freeze with owner and expiry	freeze	allow if under max	file the tracker id
Freeze expiry moved forward	freeze	reject	let it expire, then fix
Rerun budget increased	freeze	reject	leave budget as-is
Property added, others unchanged	property	allow	keep the stronger set
Active freezes exceed max	freeze	reject	fix or drop the oldest

The table is the policy. The checker is just the policy made cheap to run.

Numbered workflow

1. Freeze the contract on main. Do not generate it from the patch.
2. Give the agent a failing test or a bug report, not the contract file as an editable target. If the agent must see the contract, mark it read-only in the tool layer.
3. Run unit tests. Then run python tools/contract_check.py against the HEAD snapshot.
4. If a property fails, fix the code. Do not delete the property.
5. If a fixture digest misses, restore the file. Do not hash the new bytes into the contract on the same branch.
6. If a test is flaky, add a freeze only with an owner, a tracker id, an expiry inside one week, and remaining_reruns at 1 or 2. Do not copy an open-ended skip mark.
7. Re-run the checker. Merge only when the contract diff is empty or strictly stronger.

A remote loop is useful when local GPUs or API keys are not available. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which is enough to generate a candidate patch and execute the same pytest plus contract_check.py commands on a throwaway machine. Do not treat that environment as a source of extra product claims. The contract still has to fail closed if the server is unreachable.

How agents weaken each layer

Properties die by deletion, by renaming, and by vacuous predicates. The id set and the AST check cover the first and the third. Renames show up as a removed id. If a team allows id aliases, require the alias map to live in the contract and treat an alias as a removal plus an add, which still needs review.

Fixtures die by rewrite and by path tricks. Hash the canonical path after resolving symlinks. Reject .. in fixture paths. Reject a digest that matches a file outside tests/fixtures/.

Freezes die by extension. Expiry and rerun budget are monotonic downward. A new freeze is allowed only under max_active_freezes. That cap is what stops a patch from freezing the entire suite one test at a time.

Limitations

The vacuity checker is shallow. assert x == x still looks like a predicate. Stronger oracles need domain relations, mutation testing, or a must-reject corpus, which are separate gates.

Hash locks freeze bugs as well as goldens. If invoice_v3.json already encodes a parser mistake, the contract will protect that mistake until a human unlocks it.

Freeze budgets assume clock sanity in CI. If workers drift, expiry checks become flaky. Pin now to the orchestrator's time, not the job container's hardware clock.

The contract file can become a chokepoint. If every small test change needs a contract edit, people will start rubber-stamping diffs. Keep properties coarse. Keep fixtures few. Keep the freeze cap low.

This workflow does not measure performance, memory, or blast radius. A non-weakening contract can still ship a slower parser or a broader diff. Combine it with review, not instead of review.

Who should not use this

Do not use this if no person will own a freeze. An unowned freeze is a skip with extra YAML.

Do not use this on repos whose fixtures are regenerated on every build. Content-addressing needs bytes that are supposed to be stable.

Do not use this as a substitute for secret scanning, license checks, or semantic review of agent-authored code. A monotonic contract says the tests did not get easier. It does not say the change is right.

Skip it for throwaway prototypes where the suite itself is still being invented. Add the contract once the invariants are named.

Closing

Run the contract diff on the next agent PR before debating model quality. If the bar got lower, the color of the remaining tests is not information. If you need a disposable remote runner for that check, MonkeyCode's free server option can execute the same two commands you already run locally.