An agent patch that turns CI green has not necessarily fixed the product. It may have deleted the failing test, marked it skip, widened a fixture, or replaced a precise assertion with a tautology. The decision that matters is a structured diff of the suite itself: collected node ids, assertion density, skip and xfail sets, and fixture signatures. Until that diff is clean, a zero exit code is evidence about the tests, not about the code.
This is a gate you can run before you trust a merge. It does not require a paid model. It does require a frozen snapshot of the suite taken before the agent is allowed to write.
Why a green job is an incomplete measurement
Most agent loops optimize for one bit: did pytest (or the equivalent) return 0? That bit is cheap to game. Deleting a test is a one-line edit. Broadening pytest.approx(rel=1e-1) can hide a numeric regression. Changing a fixture from a single-row record to an empty dict can make every consumer pass vacuously.
The product under test did not get safer. The measurement did.
A useful gate therefore treats the test tree as a second artifact under review. Code diffs still matter. Suite diffs catch a different failure mode: the agent repaired the oracle instead of the implementation.
What to census
Four quantities are enough for a first gate. Each is cheap to compute and hard to fake without leaving a trace in the snapshot.
- Collected node ids. The set of tests pytest (or your runner) would run. Additions can be legitimate. Silent deletions and renames that drop coverage are not.
-
Assertion density. Count of
assert/self.assert*nodes in test modules, via AST, not via grepping strings inside comments. -
Skip and xfail sets. Node ids marked skip, xfail, or wrapped in
pytest.importorskipafter the snapshot. A newly skipped test is a deleted test with better public relations. - Fixture signatures. Name, argument list, and a hash of the fixture function body for fixtures the collected tests actually use.
Flaky tests belong in a fifth, human-owned file: a quarantine manifest. The agent may not shrink it. A quarantined test that starts passing still needs a human unfreeze. That rule is separate from “ignore flaky tests,” which is how suites rot.
Artifact: a pre/post suite census
The script below is a proposal you can run locally. It is not a benchmark of any model. It snapshots collection, walks test files with ast, and exits non-zero when the post-patch suite is weaker than the pre-patch snapshot.
#!/usr/bin/env python3
"""suite_gate.py — compare two pytest suite snapshots.
Usage:
python suite_gate.py snapshot --out before.json
# agent writes the tree
python suite_gate.py snapshot --out after.json
python suite_gate.py diff before.json after.json --quarantine quarantine.txt
"""
from __future__ import annotations
import argparse, ast, hashlib, json, subprocess, sys
from pathlib import Path
from typing import Any
TEST_GLOBS = ("test_*.py", "*_test.py")
def collect_node_ids() -> list[str]:
proc = subprocess.run(
[sys.executable, "-m", "pytest", "--collect-only", "-q"],
check=False, capture_output=True, text=True,
)
ids = []
for line in proc.stdout.splitlines():
line = line.strip()
if line.startswith("====") or not line:
continue
if "tests collected" in line or "error" in line.lower():
continue
ids.append(line)
return sorted(set(ids))
class TestVisitor(ast.NodeVisitor):
def __init__(self) -> None:
self.asserts = 0
self.weak = 0
self.skips = 0
def visit_Assert(self, node: ast.Assert) -> None:
self.asserts += 1
if _is_weak_assert(node.test):
self.weak += 1
self.generic_visit(node)
def visit_Call(self, node: ast.Call) -> None:
name = _call_name(node)
if name in {"skip", "pytest.skip", "pytest.xfail", "unittest.skip"}:
self.skips += 1
if name.startswith("self.assert") or name.startswith("self.assert"):
self.asserts += 1
self.generic_visit(node)
def _call_name(node: ast.Call) -> str:
if isinstance(node.func, ast.Attribute):
base = node.func.value
if isinstance(base, ast.Name):
return f"{base.id}.{node.func.attr}"
return node.func.attr
if isinstance(node.func, ast.Name):
return node.func.id
return ""
def _is_weak_assert(test: ast.expr) -> bool:
if isinstance(test, ast.Constant) and test.value is True:
return True
if isinstance(test, ast.Compare) and len(test.ops) == 1:
if isinstance(test.ops[0], ast.IsNot) and isinstance(test.comparators[0], ast.Constant):
return test.comparators[0].value is None
return False
def walk_tests(root: Path) -> dict[str, Any]:
files: dict[str, Any] = {}
for glob in TEST_GLOBS:
for path in root.rglob(glob):
if "__pycache__" in path.parts:
continue
src = path.read_text(encoding="utf-8")
tree = ast.parse(src)
v = TestVisitor()
v.visit(tree)
digest = hashlib.sha256(src.encode()).hexdigest()[:16]
files[str(path)] = {
"sha16": digest,
"asserts": v.asserts,
"weak_asserts": v.weak,
"skip_calls": v.skips,
}
return files
def fixture_sigs(root: Path) -> dict[str, str]:
sigs: dict[str, str] = {}
for path in root.rglob("conftest.py"):
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if not isinstance(node, ast.FunctionDef):
continue
if not any(
(isinstance(d, ast.Name) and d.id == "fixture")
or (isinstance(d, ast.Attribute) and d.attr == "fixture")
for d in (node.decorator_list or [])
for d in [d]
):
# also match @pytest.fixture via attribute
deco_ok = False
for d in node.decorator_list:
if isinstance(d, ast.Attribute) and d.attr == "fixture":
deco_ok = True
if isinstance(d, ast.Call) and isinstance(d.func, ast.Attribute) and d.func.attr == "fixture":
deco_ok = True
if isinstance(d, ast.Name) and d.id == "fixture":
deco_ok = True
if not deco_ok:
continue
body = ast.dump(node)
sigs[f"{path}::{node.name}"] = hashlib.sha256(body.encode()).hexdigest()[:16]
return sigs
def snapshot(root: Path) -> dict[str, Any]:
return {
"node_ids": collect_node_ids(),
"files": walk_tests(root),
"fixtures": fixture_sigs(root),
}
def load_quarantine(path: Path) -> set[str]:
if not path.exists():
return set()
return {ln.strip() for ln in path.read_text().splitlines() if ln.strip() and not ln.startswith("#")}
def diff(before: dict[str, Any], after: dict[str, Any], quarantine: set[str]) -> list[str]:
findings: list[str] = []
lost = set(before["node_ids"]) - set(after["node_ids"])
lost -= quarantine
if lost:
findings.append(f"deleted_or_uncollected tests: {sorted(lost)[:20]}")
b_asserts = sum(f["asserts"] for f in before["files"].values())
a_asserts = sum(f["asserts"] for f in after["files"].values())
if a_asserts < b_asserts:
findings.append(f"assertion_count {b_asserts} -> {a_asserts}")
b_weak = sum(f["weak_asserts"] for f in before["files"].values())
a_weak = sum(f["weak_asserts"] for f in after["files"].values())
if a_weak > b_weak:
findings.append(f"weak_asserts {b_weak} -> {a_weak}")
b_skip = sum(f["skip_calls"] for f in before["files"].values())
a_skip = sum(f["skip_calls"] for f in after["files"].values())
if a_skip > b_skip:
findings.append(f"skip_or_xfail_calls {b_skip} -> {a_skip}")
for key, digest in before["fixtures"].items():
if key in after["fixtures"] and after["fixtures"][key] != digest:
findings.append(f"fixture_body_changed: {key}")
if key not in after["fixtures"]:
findings.append(f"fixture_removed: {key}")
still = [qid for qid in quarantine if qid not in after["node_ids"] and qid not in before["node_ids"]]
dropped_q = [qid for qid in quarantine if qid in before["node_ids"] and qid not in after["node_ids"]]
if dropped_q:
findings.append(f"quarantine_entries_removed_from_collection: {dropped_q}")
_ = still
return findings
def main() -> int:
p = argparse.ArgumentParser()
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("snapshot")
s.add_argument("--out", required=True)
d = sub.add_parser("diff")
d.add_argument("before")
d.add_argument("after")
d.add_argument("--quarantine", default="quarantine.txt")
args = p.parse_args()
root = Path(".")
if args.cmd == "snapshot":
Path(args.out).write_text(json.dumps(snapshot(root), indent=2))
return 0
before = json.loads(Path(args.before).read_text())
after = json.loads(Path(args.after).read_text())
findings = diff(before, after, load_quarantine(Path(args.quarantine)))
for f in findings:
print(f"FAIL {f}")
if findings:
print(f"{len(findings)} suite-weakening change(s)")
return 2
print("PASS suite census")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Label the weak-assert heuristic as incomplete. assert x is not None is sometimes the right contract. The gate only flags a net increase in that pattern across the suite, which is the direction agents take when they need a cheap green.
Decision table
Apply the census before you run the expensive suite. A patch can fail this table and still compile.
| Signal | Direction that fails the gate | Typical agent edit | Human override |
|---|---|---|---|
| Collected node ids | Net loss outside quarantine | Delete or rename the failing test | Add a signed note that coverage moved |
| Assertion count | Decrease | Comment out asserts; replace with prints | Allowed if a property test replaces N examples |
| Weak asserts | Increase |
assert True, is not None only |
Allowed if a typed round-trip is added in the same file |
| skip / xfail calls | Increase |
@pytest.mark.skip on the red test |
Quarantine file only, not inline skips |
| Fixture body hash | Change | Empty dict, broader approx, dropped seed |
Review as if it were production code |
| Quarantine manifest | Shrink or silent pass | Remove the flaky line so CI “recovers” | Human unfreeze with a new seed or a root-cause patch |
The last row is the flaky-test freeze. Put node ids in quarantine.txt. Do not let the agent write that file. A flaky test that starts passing is still quarantined until a person removes the line. Passing-by-accident is how flakes re-enter the merge path.
Numbered workflow
-
Freeze the tree. Commit or stash a clean baseline. Record
quarantine.txtif you already know flakes. Takebefore.jsonwithpython suite_gate.py snapshot --out before.json. -
Give the agent a bounded write set. Source under
src/and tests undertests/are enough. Excludequarantine.txt, CI config, and the census script. - Generate the patch on a disposable loop. A free model on a free server is sufficient for this step because the gate does not trust the model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option are one way to run that loop without standing up paid inference; the census still runs in your own checkout, and this article does not claim quotas, model names, or hardware.
-
Census the result.
python suite_gate.py snapshot --out after.jsonthenpython suite_gate.py diff before.json after.json. Exit 2 means stop. Do not run the full suite yet. You would only be measuring the weakened oracle. - Run tests only if the census passes. Same runner, same markers, same seed. If the suite is still red, the patch is a failed hypothesis about the code. Send it back. If it is green and the census is clean, review the source diff as you would any human PR.
- Unfreeze flakes off-band. A quarantined test that went green gets a ticket, not an automatic delete. Re-add it to the live set only after it passes N consecutive scheduled runs you actually record.
Keep the snapshots in CI as artifacts. A later patch that “accidentally” restores a deleted test will show up as a node-id addition. That is fine. The gate is asymmetric on purpose: additions are cheap, silent removals are not.
Property checks sit after the census
Property tests are still worth having. They are not a substitute for the census. An agent can satisfy a weak property (for all xs, len(xs) >= 0) the same way it satisfies a weak assert.
If you add properties, pin the generator seed and the example budget in a file the agent cannot edit. Then include those files in the fixture-signature set. A property that silently drops from 200 examples to 5 is another form of suite shrinkage. The census will not see example counts unless you put them in a fixture or a constant the AST hash covers. Put them there.
# tests/prop_config.py — agent write-deny this path in your sandbox
EXAMPLE_BUDGET = 200
RNG_SEED = 20260905
Wire the budget into Hypothesis or whatever library you use. The point is not the library. The point is that shrinking the budget is a detectable fixture change.
Limitations, and who should not use this
The census is a filter, not a proof of correctness. It will not catch a patch that keeps every test, keeps every assert, and still changes production behavior outside the collected examples. That is what the actual suite, plus review, is for.
Do not use this workflow when any of the following hold:
- There is no test runner that can emit a stable collection list. Untestable trees need characterization tests first, not an agent.
- Tests are generated at collection time from a non-deterministic plugin. The node-id set will thrash and the gate will page you forever.
- The language is not AST-parseable by this script and you have not replaced
walk_testswith an equivalent. Copying the Python visitor onto Go or C++ tests will silently under-count. - You need a legal or safety certification trail. A suite diff is not a formal method.
- The agent is allowed to rewrite CI. If it can disable the gate, the rest of the procedure is theater.
False positives will happen. Moving a test across files changes node ids. Replacing three example tests with one parameterized test drops file-level assertion counts even when coverage rises. Handle those with an explicit override file, not by lowering the gate to warnings. A warning-only census is how deletion becomes policy.
Cost is dominated by collection, not by the model. Collection on a large pytest tree can take tens of seconds. Snapshot twice anyway. The second snapshot is the one that tells you whether the agent edited the ruler.
What this does not claim
No timing, pass-rate, or model-quality numbers are attached to this gate. None should be. The only invariant is local and mechanical: after the agent writes, the suite must not be a weaker instrument than it was before. If you already generate patches in a free-model loop, put the census in CI first. The prompt can wait.
Top comments (0)