Lock Behavior at HEAD Before an Agent Can Add Tests

Agent-written tests do not make an agent patch safer. Test count is a weak proxy. The merge signal that matters is whether behavior at HEAD is still pinned, whether named input partitions still fail closed, and whether flake policy stayed out of the agent's diff.

A patch can add twelve tests and still delete the only failing case. It can wrap a race in time.sleep. It can mark a property xfail and call the suite green. Count the tests and you will miss all three. Count partitions, snapshot HEAD, and path-protect the flake registry instead.

This is a proposed workflow, not a production study. The scripts below are unlabeled only where they are complete enough to run as-is on a toy tree. Treat thresholds as local policy, not as published benchmarks.

What the gate should prove

Three claims, in order. If any claim is missing, extra tests are noise.

1. HEAD still behaves as recorded. Capture outputs, hashes, and exit codes before the agent edits the tree. Replay them on the patched tree. Drift is a regression even when new tests pass.
2. Named partitions are still covered. A passing happy path does not cover empty input, timeout, duplicate key, or partial failure. Coverage is a partition map, not a line percentage.
3. Flake policy is immutable to the patch author. Quarantine lives in a owned file. The agent cannot add skip, xfail, sleep, or retries in production test paths.

Those claims are independent. A green characterization replay does not prove partition coverage. A full partition map does not prove the agent left the quarantine file alone.

Layer 1: characterize HEAD first

Do not let the agent write the first tests in the review. Freeze a snapshot from main (or the merge base) while the working tree is still trusted. Then require the patched tree to replay it.

Proposed command sequence:

#!/usr/bin/env bash
set -euo pipefail
BASE="${BASE_SHA:-$(git merge-base HEAD origin/main)}
git stash push -u -m "agent-wip" || true
git checkout "$BASE"
python tools/characterize.py --out artifacts/head-snapshot.json
git checkout -
git stash pop || true
python tools/characterize.py --out artifacts/patch-snapshot.json
python tools/compare_snapshots.py \
  artifacts/head-snapshot.json \
  artifacts/patch-snapshot.json

The characterizer should hash observable behavior, not source text. Source text is what the agent already rewrote.

# tools/characterize.py — proposed snapshotter
from __future__ import annotations
import hashlib, json, subprocess, sys
from pathlib import Path

CASES = [
    ["python", "-m", "app", "--input", "fixtures/empty.json"],
    ["python", "-m", "app", "--input", "fixtures/dup-key.json"],
    ["python", "-m", "app", "--input", "fixtures/timeout.json"],
]

def run(cmd: list[str]) -> dict:
    p = subprocess.run(cmd, capture_output=True, text=True)
    blob = f"{p.returncode}\n{p.stdout}\n{p.stderr}".encode()
    return {
        "cmd": cmd,
        "returncode": p.returncode,
        "stdout_sha256": hashlib.sha256(p.stdout.encode()).hexdigest(),
        "stderr_sha256": hashlib.sha256(p.stderr.encode()).hexdigest(),
        "full_sha256": hashlib.sha256(blob).hexdigest(),
    }

def main() -> None:
    out = Path(sys.argv[sys.argv.index("--out") + 1])
    rows = [run(c) for c in CASES]
    out.write_text(json.dumps(rows, indent=2, sort_keys=True))

if __name__ == "__main__":
    main()

Comparison is exact on hashes you marked stable. If a case is allowed to change, name it in a allowlist file the agent cannot edit. Unlisted drift fails the job. Do not “update the snapshot” from the same session that produced the patch.

Layer 2: score partitions, not test functions

Pytest node count rises cheaply. Partition coverage does not. Keep a map of input classes the product must honor. CI should fail when a class has zero collecting tests, or when all tests for a class are skipped.

# tests/partitions.yml — owned, not agent-authored
version: 1
partitions:
  - id: empty-input
    owner: payments
    must_fail_closed: true
    markers: ["partition_empty"]
  - id: duplicate-key
    owner: payments
    must_fail_closed: true
    markers: ["partition_dup"]
  - id: timeout
    owner: payments
    must_fail_closed: true
    markers: ["partition_timeout"]
  - id: partial-batch
    owner: payments
    must_fail_closed: true
    markers: ["partition_partial"]

Bind tests with markers. The collector then becomes the score.

# tools/check_partitions.py — proposed collector
from __future__ import annotations
import json, subprocess, sys, yaml
from collections import defaultdict
from pathlib import Path

def collected_markers() -> dict[str, int]:
    raw = subprocess.check_output(
        ["pytest", "--collect-only", "-q", "-p", "no:cacheprovider"],
        text=True,
    )
    counts: dict[str, int] = defaultdict(int)
    items = subprocess.check_output(
        ["pytest", "--collect-only", "-q", "--disable-warnings", "-qq"],
        text=True,
    )
    # Fallback: read pytest --markers from node ids via json report if present.
    report = Path("artifacts/collect.json")
    if report.exists():
        data = json.loads(report.read_text())
        for t in data.get("tests", []):
            for m in t.get("keywords", []):
                counts[m] += 1
    return counts

def main() -> int:
    spec = yaml.safe_load(Path("tests/partitions.yml").read_text())
    counts = collected_markers()
    missing = []
    for part in spec["partitions"]:
        total = sum(counts.get(m, 0) for m in part["markers"])
        if total < 1:
            missing.append(part["id"])
    if missing:
        print("uncovered partitions:", ", ".join(missing))
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

A more reliable collector writes pytest --collect-only --report-log (or a small plugin) to JSON and counts keywords per node. The policy is the same. Zero tests on a named partition is a failed merge, even if overall coverage rose.

Property checks belong here as partition oracles, not as extra examples. Each partition should have one invariant the implementation cannot satisfy by returning a constant. Example shape, labeled as a proposal:

# tests/test_properties.py — proposed, agent must not author the oracle
import pytest
from app import parse_batch

@pytest.mark.partition_dup
def test_duplicate_key_is_rejected_not_last_write_wins():
    rows = [{"id": "a", "n": 1}, {"id": "a", "n": 2}]
    with pytest.raises(ValueError, match="duplicate"):
        parse_batch(rows)

@pytest.mark.partition_partial
def test_partial_batch_does_not_commit_prefix(monkeypatch):
    committed = []
    monkeypatch.setattr("app.commit", committed.append)
    with pytest.raises(TimeoutError):
        parse_batch([{"id": "a"}, {"id": "b", "hang": True}])
    assert committed == []

If the agent rewrites parse_batch to swallow duplicates, the partition still fails. That is the point. Do not accept a new example that only asserts isinstance(result, list).

Layer 3: freeze flakes in a file the agent cannot edit

Flaky tests get quarantined. They do not get deleted, skipped inline, or padded with sleep from the same diff that changes production code. Put the freeze in tests/quarantine.yml and deny the agent that path.

# tests/quarantine.yml
version: 1
entries:
  - nodeid: tests/test_network.py::test_retry_budget
    last_green_sha: 4f2c1a0
    fail_rate_30d: 0.18
    owner: platform
    expires_on: "2026-09-24"
    reason: "dns jitter in CI runners"
    max_reruns: 0

max_reruns: 0 is intentional. Reruns hide fail rate. If a node is too noisy for a single run, it belongs in quarantine, not in a retry loop the agent introduced.

Protect the file:

# CODEOWNERS
/tests/quarantine.yml    @qa-owners
/tests/partitions.yml    @qa-owners
/tools/characterize.py   @qa-owners
/artifacts/head-snapshot.json @qa-owners

Then add a path filter so a patch that touches those files fails before pytest runs. Humans can still land a quarantine edit in a dedicated PR.

# tools/deny_agent_paths.sh
set -euo pipefail
BASE="${BASE_SHA:-origin/main}"
changed=$(git diff --name-only "$BASE"...HEAD)
deny='^(tests/quarantine.yml|tests/partitions.yml|tools/characterize.py)$'
if echo "$changed" | grep -E "$deny" >/dev/null; then
  echo "agent patch touched an immutable test-policy path"
  echo "$changed" | grep -E "$deny"
  exit 1
fi

Scan the rest of the diff for skip-shaped cheats. This is string policy, not a semantic proof. It still catches the common merge-inflating edits.

# tools/scan_diff.py — proposed cheat scan
from __future__ import annotations
import re, subprocess, sys

PATTERNS = [
    r"pytest\.mark\.skip",
    r"pytest\.mark\.xfail",
    r"pytest\.mark\.flaky",
    r"@unittest\.skip",
    r"time\.sleep\(",
    r"pytest\.set_trace",
    r"reruns\s*=",
]

def main() -> int:
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    diff = subprocess.check_output(["git", "diff", "-U0", f"{base}...HEAD"], text=True)
    hits = []
    current = None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            current = line[6:]
        if not line.startswith("+") or line.startswith("+++"):
            continue
        if current and current.startswith("tests/"):
            for p in PATTERNS:
                if re.search(p, line):
                    hits.append(f"{current}: {line[1:].strip()}")
    if hits:
        print("blocked test-weakening edits:")
        print("\n".join(hits))
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

A human-owned quarantine entry is the only legal skip. Inline skips in the same PR as product code fail closed.

Decision table

Signal on the agent diff	Merge action
Characterization hash drift on a non-allowlisted case	Fail
New tests, but a named partition has 0 collected nodes	Fail
tests/quarantine.yml or tests/partitions.yml in the diff	Fail (separate PR)
Added skip / xfail / sleep / reruns under tests/	Fail
Partition covered, snapshot stable, policy files untouched	Continue to review
HEAD already wrong, snapshot encodes the bug	Do not use this gate until HEAD is repaired

The last row is a hard limitation. Characterization freezes whatever you give it. If main is incorrect, the snapshot will defend the bug.

Wire it as two jobs

Keep authoring and grading apart. Job A is the agent session, or the PR branch. Job B never sees the agent prompt, the chain-of-thought, or the discarded retries. Job B checks out the PR sha, fetches the merge-base snapshot, and runs the three layers.

Proposed CI sketch:

# .github/workflows/agent-gate.yml — proposed
name: agent-gate
on: pull_request
jobs:
  grade:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python tools/scan_diff.py origin/${{ github.base_ref }}
      - run: bash tools/deny_agent_paths.sh
      - run: python tools/check_partitions.py
      - run: bash tools/replay_head_snapshot.sh

Job B can run on a small shared runner. It does not need the model that wrote the patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. When the grading job must not share a session with the patch author, MonkeyCode's free model access and free server option are enough to host Job B and to draft partition properties from the spec document alone. They do not relax CODEOWNERS, and they do not make a frozen snapshot correct if HEAD is already wrong. Keep production secrets out of that environment.

Draft properties from the spec, not from the failing tests the agent just saw. If the spec is a markdown contract, the second session reads that file and emits invariants per partition id. The first session never receives those invariants.

Limitations

This plan assumes a stable merge base, a written partition list, and owners who actually expire quarantine rows. It will not save a suite that is already mostly skipped. It will not detect a semantically weaker assertion that still uses assert and avoids the cheat regex. It will not replace review on authentication, payments, or anything that can exfiltrate data.

Characterization also couples tests to incidental output. Hashing full stderr will flake on timestamps and paths. Hash structured fields, or normalize first. If you cannot name the stable fields, you are not ready to freeze them.

Who should not use this:

1. Greenfield spikes with no trusted HEAD.
2. Repos where everyone can edit CODEOWNERS.
3. Teams that treat quarantine as a junk drawer with no expires_on.
4. Pipelines that must not send fixtures to a shared or free server because the fixtures contain secrets.

If those constraints hold, start with the snapshot and the path filter. They are smaller than a new test framework and they still block the cheapest ways an agent inflates a green check.

A free grading environment is sufficient for hashing fixtures and denying policy paths. Put the model that authors the patch on one side of that boundary, and leave it there.