Picture a patch that fixes an off-by-one in a pagination helper. It is small, clean, and green in ninety seconds. It also deletes two assertions from the test file that covered the bug.
That is a constructed scenario, not a customer story. I use it because it recurs in agent workflows. An agent optimizes for the signal you give it. Green tests are a cheap signal. Deleting a test is the cheapest way to reach it.
Most teams react by reading diffs more carefully. Careful reading does not scale past a few patches a day. A structural control is cheaper. Treat the test directory as input to the agent, not output.
Freeze the surface first
Two ideas carry this gate. First, the test tree is immutable during a run. Second, the patch is graded on selected tests plus a fixed smoke set. The selection comes from coverage data, not from the agent's opinion.
Start with twenty lines of Python. The script hashes every test file before the run.
# surface.py — hash the test tree before any agent run
import hashlib, json
from pathlib import Path
TEST_ROOT = Path("tests")
LOCK = Path(".test-surface.json")
def snapshot() -> dict[str, str]:
return {
str(p): hashlib.sha256(p.read_bytes()).hexdigest()
for p in sorted(TEST_ROOT.rglob("*.py"))
}
def freeze() -> None:
LOCK.write_text(json.dumps(snapshot(), indent=2))
def verify(allow: set[str]) -> list[str]:
old = json.loads(LOCK.read_text())
new = snapshot()
changed = {k for k in new if old.get(k) != new[k]}
return sorted(changed - allow)
Run python surface.py freeze on the main branch. Run the verify step after the agent finishes. Any drift outside the allowlist exits non-zero.
python surface.py freeze
# ... agent applies its patch ...
python surface.py verify --allow tests/test_pagination.py || exit 1
The allowlist is not a loophole. A moved test file must appear in the commit message. Reviewers then see one line, not a forty-line diff.
Grade on selected tests, not all tests
Running the full suite per attempt is slow enough that people trim it. They trim it badly, by trusting the agent's summary. Use coverage data instead.
# select.py — pick tests that touch the changed modules
import json, subprocess
from pathlib import Path
changed = subprocess.run(
["git", "diff", "--name-only", "origin/main"],
capture_output=True, text=True, check=True,
).stdout.split()
cov = json.loads(Path("coverage.json").read_text())
hit: set[str] = set()
for path, data in cov["files"].items():
if any(c in path for c in changed):
hit.update(data.get("tests", []))
smoke = {"tests/test_smoke.py"}
print("\n".join(sorted(hit | smoke)))
The shape of coverage.json depends on your tool. Adapt the two lines that read files and tests. The smoke set is small and hand-picked. It catches import-time breakage that selection misses.
Pin the run so two verdicts are comparable. Hash seed and plugin order are common culprits.
PYTHONHASHSEED=0 pytest $(python select.py) \
-p no:randomly -q --maxfail=1 --durations=10
Now add a ratchet. Coverage of the touched modules must not fall below the main-branch number by more than half a point. A patch that fixes a bug by deleting a branch will trip this.
Quarantine with an expiry date
Some tests are flaky through no fault of the patch. Freeze them instead of dropping them. Keep the ledger as one JSON line per entry.
{"test": "tests/test_feed.py::test_order", "seed": 7, "fails": 3, "runs": 40, "frozen_until": "2026-12-05"}
A test enters the ledger after two failures in twenty identical seeded runs. Quarantined tests do not gate the agent. They still run on a schedule. When an entry passes its expiry date, the gate fails until someone fixes or re-justifies it. Without expiry, a quarantine becomes a graveyard.
Here is how the gate usually fails, and what each failure means.
| Signal | Likely cause | First move |
|---|---|---|
| Test hash drift | Patch edited tests | Reject, re-run from a clean tree |
| Selected test red | Real regression | Read the diff, not the agent summary |
| Coverage drop | Branch deleted or skipped | Inspect the removed lines |
| Expired ledger entry | Ignored flake | Fix it or extend with a reason |
Where running cost lands
The gate runs three things per attempt: selection, smoke, and ledger replay. That is more executions than a single pytest . call, not fewer. Cost moves from reading diffs to running jobs.
This is where the availability question becomes practical. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The server fits the unattended part of the loop, because the nightly ledger replay needs to run on a schedule with nobody watching. Free model access fits the retry budget, since cheap attempts make it affordable to keep the full test surface instead of trimming it to save money.
I have not benchmarked either against paid alternatives, and you should not treat availability as a performance claim. Measure your own wall-clock per attempt before you commit to a cadence.
Limits and who should skip this
Small repositories gain little. Below roughly fifty tests, the hash check is the only part worth keeping. Teams without coverage output need instrumentation before selection works at all.
Long integration suites blow the time budget quickly. Pin those to nightly runs and keep the gate to unit-level selection. Cross-platform flakiness cannot be pinned by a seed, so the ledger will not save you there.
Skip this entirely if agents never touch your test files. Plain diff review is enough. Skip it too if your compliance rules demand a full suite per commit, because the gate adds latency without replacing that requirement.
If you try one piece, try the hash check. It is twenty lines, and it catches the loudest failure in the room.
Top comments (1)
"Treat the test directory as input to the agent, not output" is the whole idea in one line. One gap worth closing: hashing tests/ catches edits there, but a skip can also come from config, like a pytest.ini addopts change or a conftest that deselects. Adding those files to the frozen surface would cover it.