Reject Agent Diffs That Widen skip, xfail, or Timeouts

An agent that leaves every assert intact can still hollow out a suite. One new pytest.mark.skip, one raised timeout, one filterwarnings("ignore"), and CI stays green. Assertion-diff gates do not see that class of edit. The merge protocol that remains is a frozen test control plane, content-addressed fixtures, properties the agent cannot rewrite, and a quarantine job that still runs known flakes.

That is the claim. The rest is a workflow you can run on a laptop before anyone clicks merge.

The hole assertion gates miss

Most agent-review checklists score assertion text. They count deleted assert lines, weakened comparators, and dropped boundary cases. Those checks are necessary. They are not sufficient.

The test runner is also an oracle. Marks such as skip, xfail, skipif, plus retry decorators, warning filters, and timeout knobs, decide whether a check executes, how long it may run, and which exceptions count as failures. An agent patch that only edits those knobs never appears in an assertion-weakening score. The suite still "passes." The evidence is gone.

This protocol treats that control plane as first-class oracle surface. It does not replace property tests or fixture locks. It sits in front of them.

What the protocol freezes

Three files live at the repo root. None of them is a skip list.

1. properties.yaml — invariants that must hold after the patch. The agent may add rows. It may not delete or soften existing ones.
2. fixtures.lock — SHA-256 of every committed fixture blob, keyed by content, not by filename. Rename-to-dodge does not clear a missing hash.
3. flake.quarantine.json — known flakes with an expiry and a required shadow job. The tests still run. Failures there do not fail merge, and they do not vanish from inventory.

A fourth rule is not a file. It is a diff policy: the same change that claims to fix production code may not add or broaden skip, xfail, skipif, filterwarnings, retry wrappers, or timeout increases.

Pytest documents skip / xfail as explicit execution control, not as style (Skipping and xfail). Treat a widening of that surface the way you treat a deleted assertion.

Who this is for

Teams that already let a coding agent open pull requests against Python services, and that already run pytest in CI. The protocol assumes a human still owns merge. It is a classifier plus a checker, not a substitute for review.

Workflow

1. Snapshot the control plane before the agent runs

Record every test node id and its current marks. Store the snapshot outside the tree the agent is allowed to write, or hash it into CI as a protected artifact.

pytest --collect-only -q --disable-warnings \
  | awk 'NF && !/test session/' \
  > /tmp/pre-agent-collect.txt

python3 scripts/marks_snapshot.py --out control_plane.pre.json

The snapshot helper below is a proposed script, not a published plugin. Keep it read-only. If the agent can edit scripts/marks_snapshot.py, the probe is not an oracle.

# scripts/marks_snapshot.py — proposed helper, not a shipped plugin
"""Collect node ids and pytest marks. Run before and after an agent patch."""
from __future__ import annotations

import json
from pathlib import Path

import pytest

class MarkProbe:
    def __init__(self, out_path: Path) -> None:
        self.out_path = out_path

    def pytest_collection_modifyitems(self, items) -> None:
        rows = []
        for item in items:
            marks = sorted({m.name for m in item.iter_markers()})
            rows.append({
                "nodeid": item.nodeid,
                "marks": marks,
                "has_skip": "skip" in marks or "skipif" in marks,
                "has_xfail": "xfail" in marks,
                "timeout": _timeout(item),
            })
        self.out_path.write_text(json.dumps(rows, indent=2, sort_keys=True))

def _timeout(item):
    for m in item.iter_markers(name="timeout"):
        if m.args:
            return m.args[0]
    return None

2. Generate the candidate patch on a cheap loop, then stop

The agent is a hypothesis generator. It is not the judge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free model endpoint plus a free server option is enough for that first loop: propose a diff, apply it to a throwaway worktree, run the checker locally. MonkeyCode exposes both of those, so the generation step does not have to sit on a billed API. The protocol does not depend on that product. Any agent that emits a git diff can feed the same gate. Do not send the oracle files to the model if the model can write them back.

git worktree add /tmp/agent-hyp-$RANDOM HEAD
# apply the candidate diff inside the worktree only
python3 scripts/control_plane_gate.py \
  --pre control_plane.pre.json \
  --post control_plane.post.json \
  --properties properties.yaml \
  --lock fixtures.lock \
  --quarantine flake.quarantine.json

3. Reject control-plane widening

The gate fails on any of the following, even when pytest is green.

Diff class	Merge?	Why
New skip / skipif on an existing node	No	Execution removed
xfail added or strict dropped	No	Failure reclassified as expected
filterwarnings("ignore") added	No	Signal discarded
Timeout increased	No	Hang budget expanded
Retry / flaky plugin decorator added	No	Failures hidden by reruns
Test deleted and re-added with a new node id	No	Inventory evasion
Fixture bytes changed, filename unchanged	No	Lock mismatch
Fixture renamed, hash already in lock	Yes, lock rewrite by a human	Content still known
New property row, old rows intact	Yes	Surface grew
Quarantine expiry extended by the agent	No	Freeze became a skip
Production code only, control plane unchanged	Yes, if properties and locks hold	Normal path

Implement the first column as code, not as a reviewer memory aid.

# scripts/control_plane_gate.py — proposed checker
BLOCK_MARKS = {"skip", "skipif", "xfail", "flaky", "filterwarnings"}

def widenings(pre, post):
    pre_map = {r["nodeid"]: r for r in pre}
    findings = []
    post_ids = {r["nodeid"] for r in post}
    for row in post:
        old = pre_map.get(row["nodeid"])
        if old is None:
            if row["has_skip"] or row["has_xfail"]:
                findings.append(("new_node_suppressed", row["nodeid"]))
            continue
        if row["has_skip"] and not old["has_skip"]:
            findings.append(("skip_added", row["nodeid"]))
        if row["has_xfail"] and not old["has_xfail"]:
            findings.append(("xfail_added", row["nodeid"]))
        if _timeout_raised(old, row):
            findings.append(("timeout_raised", row["nodeid"]))
        extra = set(row["marks"]) - set(old["marks"])
        if extra & BLOCK_MARKS:
            findings.append(("blocked_mark", row["nodeid"], sorted(extra)))
    for nodeid in sorted(set(pre_map) - post_ids):
        findings.append(("node_removed", nodeid))
    return findings

def _timeout_raised(old, row) -> bool:
    before, after = old.get("timeout"), row.get("timeout")
    if before is None or after is None:
        return after is not None and before is None
    return after > before

A green pytest run with a non-empty findings list is a failed gate. Do not parse pytest exit codes alone.

Inline pytest.skip() is the same hole without a mark. A small AST pass closes the obvious case. It is still a proposal. It is also spoofable if the agent owns the file that runs the pass.

# scripts/forbid_runtime_skip.py — proposed, unexecuted example
import ast
from pathlib import Path

FORBIDDEN = {("pytest", "skip"), ("pytest", "xfail")}

def runtime_skips(root: Path) -> list[str]:
    hits = []
    for path in root.rglob("test_*.py"):
        tree = ast.parse(path.read_text(), filename=str(path))
        for node in ast.walk(tree):
            if not isinstance(node, ast.Call):
                continue
            fn = node.func
            if isinstance(fn, ast.Attribute) and isinstance(fn.value, ast.Name):
                if (fn.value.id, fn.attr) in FORBIDDEN:
                    hits.append(f"{path}:{node.lineno}")
    return hits

4. Run properties from a write-protected path

Property checks belong in a directory the agent job cannot write. CI should check out tests/properties from a second ref, or from a submodule with a pinned SHA. The production tree under test is the agent's worktree. The properties are not.

Hash that tree in CI. A properties-tree diff requires a human-owned commit. If the agent "fixes" a property by shrinking strategy ranges, that is assertion weakening by another name.

The example below is a template. Point the import at the real module. Hypothesis is the usual Python property library (Hypothesis docs).

# tests/properties/test_invoice_invariants.py — template
from decimal import Decimal
from hypothesis import given, strategies as st

from billing.invoice import apply_discount  # replace with the module under test

@given(
    amount=st.decimals(min_value="0.01", max_value="1e6", places=2),
    rate=st.decimals(min_value="0", max_value="0.50", places=2),
)
def test_discount_never_increases_total(amount: Decimal, rate: Decimal) -> None:
    out = apply_discount(amount, rate)
    assert out <= amount
    assert out >= Decimal("0")

Keep properties.yaml as the inventory the gate diffs, even if the executable checks live in Python. A row that disappears from the YAML is a failed gate, regardless of pytest.

# properties.yaml — proposed inventory
version: 1
invariants:
  - id: discount_never_increases_total
    module: tests.properties.test_invoice_invariants
    min_examples: 100
  - id: refund_bounded_by_capture
    module: tests.properties.test_refunds
    min_examples: 100

5. Content-address fixtures, ignore names

Filename locks fail when the agent renames case_12.json to case_12_legacy.json and writes a weaker blob under the old name. Hash the bytes.

find tests/fixtures -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum \
  > fixtures.lock

The checker loads the lock and the current tree. Every hash in the lock must still exist somewhere under tests/fixtures. New files are allowed. Missing hashes are not, unless a human removes the row.

import hashlib
from pathlib import Path

def missing_fixture_hashes(lock_path: Path, root: Path) -> list[str]:
    expected = {
        line.split()[0]
        for line in lock_path.read_text().splitlines()
        if line.strip()
    }
    present = {
        hashlib.sha256(p.read_bytes()).hexdigest()
        for p in root.rglob("*")
        if p.is_file()
    }
    return sorted(expected - present)

Pair the lock with a cheap usage check if it matters that production tests still open those files. A lock that only proves bytes exist on disk does not prove the patched code still reads them.

6. Quarantine flakes. Do not skip them.

A freeze file is not pytest.mark.skip. The quarantined node id still executes in a shadow job. The shadow job records pass/fail. Merge does not depend on it. Inventory does.

{
  "schema": 1,
  "entries": [
    {
      "nodeid": "tests/test_api.py::test_retry_after_429",
      "reason": "upstream 429 cadence is not deterministic in CI",
      "expires": "2026-10-01",
      "owner": "human",
      "shadow_job": "pytest-quarantine"
    }
  ]
}

Rules for the freeze file:

1. Only a human may add or extend an entry. The gate diffs flake.quarantine.json against origin/main and fails if the agent touched it.
2. expires is required. A missing date is a failed gate.
3. After expiry, the node must return to the main suite or be deleted in a human commit that also removes production callers, not in an agent cleanup.
4. The shadow job must collect the same node ids. A quarantine file whose tests are not collected is a failed gate.

# .github/workflows/quarantine.yml — proposed
name: flake-shadow
on: [pull_request]
jobs:
  shadow:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run quarantined node ids
        run: python3 scripts/run_quarantine.py flake.quarantine.json
      - name: Fail if quarantine is stale
        run: python3 scripts/quarantine_expiry.py --as-of today

run_quarantine.py should invoke pytest with an explicit --override-ini addopts= so repo-level retries do not apply. Quarantine that reruns until green is a skip with extra steps.

pytest --override-ini addopts= \
  --strict-markers \
  $(python3 scripts/quarantine_nodeids.py flake.quarantine.json)

What "pass" means after this

A mergeable agent patch has four independent greens:

1. Production tests run with the pre-patch control plane.
2. control_plane_gate.py reports zero widenings and zero disappeared node ids.
3. Write-protected properties pass against the patched tree.
4. Fixture lock hashes are a subset of the current fixture tree, and flake.quarantine.json is unchanged by the agent.

Pytest's exit code is one input. It is not the decision.

Limitations

The protocol does not see every runtime escape. if os.environ.get("CI"): pytest.skip() needs the AST pass, and that pass is gone if the agent owns it. Hypothesis ranges can still shrink if tests/properties is writable. Protect the tree.

Content-addressed fixtures do not help when the code stops reading the fixture. Timeout knobs measured in wall clocks stay noisy on shared runners. The gate blocks increases in configured timeouts. It does not prove the old timeout was correct.

Quarantine expiry is a calendar check. It does not repair the flake. If nobody owns the date, the file becomes a parking lot. That is worse than a visible skip, because the main job stays green.

Do not use this protocol as a reason to turn off human review, as a substitute for memory-safe languages or formal specs, or on suites where tests are generated at collect time with unstable node ids. Unstable node ids make disappearance look like a refactor. Stabilize names first.

Who should not adopt it: solo experiments with no CI; orgs that already forbid agent-authored test edits (that is a stronger rule); and safety-critical codebases that need independent V&V rather than a pytest plugin.

Generation can be cheap. Judgment should not be. Keep the model on the patch. Keep the three files and the property tree off the model's write path. If you generate candidates on a free model with a free server, run the gate in the same worktree before the next retry, or the retry will optimize for the previous pytest log, including any skip it just invented.