Fail the Merge When an Agent Deletes or Weakens Tests

A greener suite is not a better suite. If an agent patch removes tests, drops assertions, or freezes flakes that no longer exist, CI reports success while the original contract quietly shrinks.

Score the test inventory first. Then score assertion strength. Only after those two diffs are clean should you run property oracles against locked fixtures, and only then may a dated freeze apply to a flake whose identity has not moved.

This is a merge gate you can run locally. It is a workflow, not a fleet study. The commands and scripts below are labeled proposals; they are executable as written and are not production metrics.

Why deletion looks like progress

Agent patches optimize for the signal you give them. If that signal is pytest exiting 0, deleting a test is a legal move. Weakening assert result == expected into assert result is also legal. Both reduce the failure surface.

The suite gets faster. The log gets quieter. Neither fact is a quality result.

A freeze list fails the same way. If a test is deleted and the freeze still names it, reviewers think the flake is contained. It is not. The check left the inventory.

What this gate measures

Capture three artifacts before the agent runs. Diff them after.

1. Inventory. Stable node ids from the test runner, not a file glob.
2. Assertion hashes. A digest of assertion-like statements per test file, ignoring comment-only edits.
3. Freeze eligibility. A flake may stay frozen only if its node id still exists, its assertion digest is unchanged, and its expiry is still in the future.

Property checks and fixture locks come after this diff. They answer whether remaining behavior still holds. They cannot answer whether you stopped checking.

Step 1 — Snapshot the inventory

Use the runner as the source of truth. Do not glob test_*.py and hope names match collection.

# proposal: capture node ids before the agent patch
python -m pytest --collect-only -q \
  | sed '/^$/d' \
  | grep '::' > /tmp/inventory.before

mkdir -p .gate
cp /tmp/inventory.before .gate/inventory.before

Pytest prints identities such as tests/test_ledger.py::test_refund_is_idempotent. That string is what you freeze and diff. If collection itself fails, stop. An uncollectable suite is not a baseline.

Keep the snapshot next to the patch review, not in the agent's scratch directory. Pin the pytest version and plugins so node ids do not flicker between runs.

Step 2 — Hash assertions, not whole files

Whole-file hashes fire on import reorder and comment edits. You want a narrower question: did the checks change?

The script below is a proposal. It is syntactic on purpose. assert True still hashes. Treat that as a known gap, not as semantic understanding.

# gate_assert_hash.py — proposal, run in CI
from __future__ import annotations

import ast
import hashlib
import json
import sys
from pathlib import Path

ASSERT_TYPES = (ast.Assert,)
CALL_NAMES = {"pytest.raises", "pytest.warns", "pytest.deprecated_call"}


def is_raises_like(node: ast.AST) -> bool:
    if not isinstance(node, ast.Call):
        return False
    name = ast.unparse(node.func)
    return name in CALL_NAMES or name.endswith(".raises")


def file_digest(path: Path) -> dict:
    tree = ast.parse(path.read_text(encoding="utf-8"))
    chunks: list[str] = []
    for node in ast.walk(tree):
        if isinstance(node, ASSERT_TYPES) or is_raises_like(node):
            chunks.append(ast.unparse(node))
    payload = "\n".join(sorted(chunks)).encode("utf-8")
    return {
        "path": str(path).replace("\\", "/"),
        "count": len(chunks),
        "sha256": hashlib.sha256(payload).hexdigest(),
    }


def walk(root: Path) -> list[dict]:
    return [file_digest(path) for path in sorted(root.rglob("test_*.py"))]


if __name__ == "__main__":
    root = Path(sys.argv[1] if len(sys.argv) > 1 else "tests")
    json.dump(walk(root), sys.stdout, indent=2)
    sys.stdout.write("\n")

Run it on both sides of the patch.

python gate_assert_hash.py tests > .gate/asserts.before.json
# ... agent applies a patch ...
python gate_assert_hash.py tests > .gate/asserts.after.json
python -m pytest --collect-only -q | sed '/^$/d' | grep '::' > .gate/inventory.after

If AST parse fails, fail closed. A test file the helper cannot read is not an empty assertion set.

Step 3 — Classify the test diff

Do not merge on exit code 0. Merge on a classified delta.

# gate_inventory_diff.py — proposal
from __future__ import annotations

import json
import sys
from pathlib import Path


def load_ids(path: Path) -> set[str]:
    return {
        line.strip()
        for line in path.read_text().splitlines()
        if "::" in line.strip()
    }


def load_asserts(path: Path) -> dict[str, dict]:
    rows = json.loads(path.read_text())
    return {row["path"]: row for row in rows}


def main() -> int:
    before_ids = load_ids(Path(".gate/inventory.before"))
    after_ids = load_ids(Path(".gate/inventory.after"))
    before_a = load_asserts(Path(".gate/asserts.before.json"))
    after_a = load_asserts(Path(".gate/asserts.after.json"))

    deleted = sorted(before_ids - after_ids)
    added = sorted(after_ids - before_ids)
    weakened = []
    for path, row in after_a.items():
        prev = before_a.get(path)
        if prev and row["count"] < prev["count"]:
            weakened.append(
                {
                    "path": path,
                    "before": prev["count"],
                    "after": row["count"],
                    "sha_changed": row["sha256"] != prev["sha256"],
                }
            )

    report = {
        "deleted_node_ids": deleted,
        "added_node_ids": added,
        "weakened_files": weakened,
        "ok": not deleted and not weakened,
    }
    Path(".gate/inventory.diff.json").write_text(
        json.dumps(report, indent=2) + "\n"
    )
    print(json.dumps(report, indent=2))
    if deleted:
        print("FAIL: tests left the inventory", file=sys.stderr)
        return 2
    if weakened:
        print("FAIL: assertion count dropped", file=sys.stderr)
        return 3
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Added tests are allowed. They are not trusted. They still need property oracles later.

Deleted tests and reduced assertion counts fail with distinct exit codes. CI can chart those classes separately instead of folding them into a single red bar.

Step 4 — Property checks on what remains

Inventory is necessary and not sufficient. After the diff is clean, check behavior that one-off examples under-specify.

Keep properties in a separate module. An agent rewriting tests/test_*.py should not be able to edit the oracle in the same patch. Label that split in review.

# properties/test_refund_properties.py — proposal
from decimal import Decimal

import pytest

from ledger import refund


@pytest.mark.property
@pytest.mark.parametrize(
    "paid,captured",
    [
        (Decimal("10.00"), Decimal("10.00")),
        (Decimal("10.00"), Decimal("3.50")),
        (Decimal("0.01"), Decimal("0.01")),
    ],
)
def test_refund_never_exceeds_captured(paid, captured):
    result = refund(paid=paid, captured=captured)
    assert result.amount >= 0
    assert result.amount <= captured
    assert result.amount <= paid

A property here is a parameterized invariant, not a proof. If you later want a generative runner with shrinking, add it as a second change. Start with invariants the domain already believes.

Fixture bytes that those properties read should be digested beside the inventory snapshot. If the agent rewrites a fixture to match a bug, the digest change is a fail even when tests pass.

# proposal: lock fixture bytes, not fixture meaning
find fixtures -type f -print0 | sort -z | xargs -0 sha256sum > .gate/fixtures.before
# after the patch
find fixtures -type f -print0 | sort -z | xargs -0 sha256sum > .gate/fixtures.after
diff -u .gate/fixtures.before .gate/fixtures.after

A fixture digest change needs a human-owned note in the patch. An agent comment is not that note.

Step 5 — Dated freeze, with eligibility rules

Flakes exist. Hiding them inside deleted tests is how freeze lists rot.

Use a freeze file with an expiry date. Then enforce eligibility against the inventory you already captured.

# .gate/flake_freeze.yaml — proposal
# Dates are inclusive UTC dates. Missing fields fail closed.
rules:
  - node_id: tests/test_ledger.py::test_refund_is_idempotent
    reason: "order-dependent cache on refund path"
    expires: "2026-10-08"
    assertion_sha256: "replace-with-digest-from-asserts.before.json"

The assertion_sha256 field is file-level in this proposal, because the hasher walks files. That is coarser than a per-test digest. It still blocks the common cheat: rewrite the checks, keep the freeze, ship a quieter suite.

# gate_freeze.py — proposal
from __future__ import annotations

import datetime as dt
import json
import sys
from pathlib import Path

import yaml  # pin PyYAML in CI


def main() -> int:
    today = dt.date.fromisoformat("2026-09-24")  # CI must inject real UTC date
    freeze = yaml.safe_load(Path(".gate/flake_freeze.yaml").read_text())
    inventory = {
        line.strip()
        for line in Path(".gate/inventory.after").read_text().splitlines()
        if "::" in line.strip()
    }
    asserts = {
        row["path"]: row
        for row in json.loads(Path(".gate/asserts.after.json").read_text())
    }
    failed: list[str] = []
    for rule in freeze.get("rules") or []:
        node = rule["node_id"]
        expires = dt.date.fromisoformat(rule["expires"])
        if node not in inventory:
            failed.append(f"freeze names missing test: {node}")
            continue
        if expires < today:
            failed.append(f"expired freeze: {node}")
            continue
        file_key = node.split("::", 1)[0]
        row = asserts.get(file_key)
        expected = rule.get("assertion_sha256")
        if not expected or not row:
            failed.append(f"freeze missing digest mapping: {node}")
            continue
        if expected != row["sha256"]:
            failed.append(
                f"freeze ineligible after assertion change: {node}"
            )
    if failed:
        print("\n".join(failed), file=sys.stderr)
        return 4
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Three rules matter.

1. You cannot freeze a node id that left the inventory.
2. You cannot freeze past expiry. The date is data, not a comment.
3. You cannot keep a freeze when assertion hashes moved. That change is a new test. Score it.

Decision table

Diff observed	Gate result	Next action
Node id missing	Fail (exit 2)	Restore the test or record a human-owned deletion
Assertion count down	Fail (exit 3)	Restore checks or add a reviewed property
Fixture digest changed	Fail	Revert the fixture or attach a human note
Freeze names a missing test	Fail (exit 4)	Drop the freeze rule
Freeze expired	Fail (exit 4)	Fix the flake or file a dated extension
Inventory stable, freeze eligible	Continue	Run properties, then the rest of CI
New tests added	Continue with distrust	Require at least one property on the same behavior

The table is the policy. The scripts only encode it. If a reviewer cannot point at a row, the gate is not doing its job.

Where generation fits, and where it does not

Candidate patches have to come from somewhere. A local checkout is enough. So is a remote job.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, which is enough to generate a candidate patch in isolation. That host proposes diffs. It does not own .gate/. Run gate_inventory_diff.py, the fixture digest, and gate_freeze.py in your CI against the branch you intend to merge. A green run on the machine that wrote the patch is not evidence.

Keep secrets and production fixtures off the proposal host. The gate above assumes the suite already runs where you merge.

Limitations

The assertion hasher is syntactic. It will not flag assert x == 1 rewritten as assert x != 0 when the count stays constant. Pair it with properties.

It will not flag a new tautology. Added tests pass the inventory gate by design. They still need oracles.

Collection order, plugin load, and skip marks can make node ids flicker. Pin pytest, plugins, and PYTEST_ADDOPTS. If collection is nondeterministic, the inventory diff is noise and should fail closed until you pin it.

The freeze date in the sample is pinned to 2026-09-24 so this article is reproducible. CI must inject the real UTC date. Do not let the agent edit .gate/flake_freeze.yaml without a human-owned review label.

This workflow does not measure performance, cost, or model quality. Faster tests after deletion will look like a win in wall-clock charts. Exclude deleted-node runs from those charts, or you will reward the failure mode the gate exists to stop.

Who should not use this

Do not install this gate on a one-file script with no CI. The inventory snapshot has nothing to diff.

Do not use it as a substitute for code review on security or payment paths. Assertion counts are not threat models.

Do not freeze the entire suite. A freeze is a per-node exception with an expiry. A global skip list is a disabled gate.

If your tests are generated and discarded every run, there is no inventory. Stabilize names before you diff them.

Order of operations

1. Snapshot inventory, assertion hashes, and fixture digests on main.
2. Apply the agent patch on a branch.
3. Diff inventory and assertion counts. Fail on deletion or weakening.
4. Reject freeze rules that name missing tests, expired dates, or moved hashes.
5. Run property modules the agent is not allowed to edit in the same patch.
6. Run the remaining example suite.
7. Store .gate/inventory.diff.json next to the merge, not only the pytest summary.

A quiet log after step 6 is not the artifact. The classified diff is.

If you already generate patches on a free server, add the inventory scripts to CI before you add more generation capacity. The cheapest correctness bug to catch is a test that is no longer there.