Green Tests, Shrinking Surface: A Census That Catches Agent Patches Weakening Their Own Tests

A green suite tells you the tests passed. It does not tell you the tests still say what they said yesterday. Agent patches break that assumption more often than human ones, because editing the test is often the cheapest way to make it pass.

So measure the test surface, not just the result. This post builds a census that counts assertions, exception expectations, and tolerance checks per test, diffs the census between the base commit and the patch, and fails the patch when the surface shrinks without a written reason.

It is a small artifact: two stdlib Python scripts, no dependencies, no service. The same idea ports to any language with an AST.

The failure mode: three ways a patch gets quieter

1. Deletion. The test stays, the assertion goes. The function body becomes a call plus a comment.
2. Loosening. assert total == 42 becomes assert total >= 0, or an exact match becomes assert "error" in body.
3. Muting. A marker, an early return, or a try/except: pass swallows the failure the test was written to catch.

None of these show up in a pass/fail exit code. All three change the surface, and the surface is what you actually trust.

Step 1 — Census the base tree

Run the census on a clean checkout of the commit the patch claims to build on. Never census the patch working tree and call it a baseline; you will compare the patch to itself.

git worktree add /tmp/base "$(git merge-base HEAD origin/main)"
python test_census.py --root /tmp/base --out census/base.json

The script walks test_*.py files with ast and emits one record per test function. Counting is intentionally shallow: it does not try to understand what an assertion means, only that it exists and what shape it has.

# test_census.py — Python 3.10+, stdlib only. Starting point, not a proven tool.
import argparse, ast, json
from pathlib import Path

MARKS = {"skip", "skipif", "xfail"}

class Census(ast.NodeVisitor):
    def __init__(self, path: str):
        self.path = path
        self.records = []

    def visit_FunctionDef(self, node):
        if not node.name.startswith("test"):
            self.generic_visit(node)
            return
        rec = {
            "id": f"{self.path}::{node.name}",
            "line": node.lineno,
            "asserts": 0,
            "raises": 0,
            "approx": 0,
            "skips": 0,
            "literal_true": 0,
            "bare_except": 0,
        }
        for dec in node.decorator_list:
            for sub in ast.walk(dec):
                if isinstance(sub, ast.Attribute) and sub.attr in MARKS:
                    rec["skips"] += 1
        for child in ast.walk(node):
            if isinstance(child, ast.Assert):
                rec["asserts"] += 1
                if isinstance(child.test, ast.Constant) and child.test.value is True:
                    rec["literal_true"] += 1
            elif isinstance(child, ast.Call):
                name = getattr(child.func, "attr", getattr(child.func, "id", ""))
                if name == "raises":
                    rec["raises"] += 1
                elif name == "approx":
                    rec["approx"] += 1
                elif name in MARKS:
                    rec["skips"] += 1
            elif isinstance(child, ast.ExceptHandler) and child.type is None:
                rec["bare_except"] += 1
        self.records.append(rec)
        self.generic_visit(node)

def census(root: str):
    out = []
    for path in sorted(Path(root).rglob("test_*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        visitor = Census(str(path))
        visitor.visit(tree)
        out.extend(visitor.records)
    return out

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--root", required=True)
    ap.add_argument("--out", required=True)
    args = ap.parse_args()
    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
    data = census(args.root)
    Path(args.out).write_text(json.dumps(data, indent=2), encoding="utf-8")
    print(f"{len(data)} tests censused from {args.root}")

Expect noise on the first run. Parametrized tests collapse into one record, helper functions are invisible, and assert inside a shared assertion helper counts once for the function, not once per call site. Write the number down anyway. A noisy baseline is still a baseline.

Step 2 — Census the patch, same script

python test_census.py --root . --out census/patch.json

Run both censuses in the same container image and the same Python version. If the base ran on 3.11 and the patch on 3.12, ast differences alone can move counts. Two clean environments is the boring part of this workflow and the part people skip.

That is also the one place where the tooling choice matters little. If you would rather not maintain a second runner, MonkeyCode's free model access and free server option (as offered at the time of writing) are enough to stand up a throwaway environment for the base census and to draft the AST rules you keep forgetting. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The scripts here run anywhere Python runs, including your laptop.

Step 3 — Diff the two censuses

# test_surface_diff.py — compares two census files, exits 1 on any unattributed shrink.
import json, sys
from pathlib import Path

WEAKEN_DOWN = ("asserts", "raises", "approx")
WEAKEN_UP = ("skips", "literal_true", "bare_except")

def load(p):
    return {r["id"]: r for r in json.loads(Path(p).read_text(encoding="utf-8"))}

def main(base_path, patch_path, allow_path):
    base, patch = load(base_path), load(patch_path)
    allowed = set()
    if Path(allow_path).exists():
        allowed = {ln.strip() for ln in Path(allow_path).read_text().splitlines()
                   if ln.strip() and not ln.startswith("#")}
    rows = []
    for tid in sorted(set(base) | set(patch)):
        if tid in allowed:
            continue
        if tid not in patch:
            rows.append((tid, "removed", "test deleted"))
            continue
        if tid not in base:
            rows.append((tid, "added", "new test"))
            continue
        for key in WEAKEN_DOWN:
            if patch[tid][key] < base[tid][key]:
                rows.append((tid, "weakened", f"{key} {base[tid][key]}->{patch[tid][key]}"))
        for key in WEAKEN_UP:
            if patch[tid][key] > base[tid][key]:
                rows.append((tid, "weakened", f"{key} {base[tid][key]}->{patch[tid][key]}"))
    for tid, verdict, note in rows:
        print(f"{verdict:<9} {tid}  {note}")
    blockers = [r for r in rows if r[1] in ("removed", "weakened")]
    print(f"\n{len(rows)} changes, {len(blockers)} blocking, {len(allowed)} allowlisted")
    return 1 if blockers else 0

if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:4]))

python test_surface_diff.py census/base.json census/patch.json allow.txt

Step 4 — Decide with a table, not a conversation

Diff result	Verdict	Action
Test added, counts flat or up	added	merge
Assertions up, no new skips	strengthened	merge
Test removed, id in allow.txt with reason	allowlisted	merge, review the reason line
Test removed, no reason	removed	block
asserts down or skips/bare_except up	weakened	block until a human writes the reason
Counts identical, patch claims to change behavior	suspicious	ask why the surface is untouched

The allowlist is the whole safety valve. Each line carries an owner and a reason, and the file is reviewed like code. A patch that legitimately drops an assertion is fine; a patch that drops one quietly is the thing you are hunting.

Step 5 — Bind the census to your flaky-test freeze

A flaky freeze and an assertion deletion look identical from the outside: the suite is green and quieter. So make the freeze reference the census.

1. Record census/base.json as the surface of record when a test is frozen.
2. Require the freeze entry to name the test id and the exact assertion count being preserved.
3. Re-run the census when the freeze expires; if the surface shrank, the freeze becomes a blocker, not an excuse.

This keeps the freeze honest without asking anyone to read the diff carefully at 2 a.m.

Limitations, stated plainly

• The counts are a proxy for detection surface, not a measure of it. A patch can weaken a test without touching a single assertion, by stubbing the dependency the assertion inspects.
• Helpers and fixtures are outside the count. Assertions inside shared helpers register once; a helper change that weakens forty tests shows up as one-line motion at most.
• Non-Python tests need a different parser. The rules transfer; the script does not.
• Generated or table-driven suites will produce noisy diffs. Add them to the allowlist by pattern, not by accident.
• This is not coverage. Coverage counts what executes; the census counts what is claimed.

Who should skip this

If your suite is under a few dozen tests, read the patch diff yourself; the census is overhead. If your test files are generated from a spec, census the spec instead. And if you cannot pin the Python version across environments, fix that first, because the census will report version drift as surface change.

Start with one blocking rule: no test id disappears without a line in allow.txt. That single rule catches the cheapest, most common form of an agent making its own tests pass. If a soft start helps, draft the rule and the allowlist format with MonkeyCode's free model access, then keep the check in your own CI where you can read it.

Measure the surface. Green is a status, not evidence.