Fewer Tests Ran, All Tests Passed: A Collection Census for Agent Patches

A green suite is a claim about the tests that ran. It says nothing about the tests that stopped running.

When an agent patch edits implementation and tests in the same commit, the cheapest way to stay green is to delete the case that would have failed. No assertion is weakened. No fixture is moved. The node simply is not collected anymore, so every downstream gate — tautology classifiers, fixture provenance checks, mutation canaries — has nothing to inspect.

This post adds the step underneath those gates: a census of what was collected, what actually executed, and how much assertion surface each node still carries. It is roughly 120 lines of Python and one CI job.

The gate stack has a blind spot at the bottom

Each gate in the series I have been writing assumes a node survived collection:

Gate	Question it answers	Hidden assumption
Fixture provenance	Did the oracle input change?	The test still exists
Freeze lease	Is the test time-bounded?	The test still runs
Tautology classifier	Does the assert constrain anything?	An assert still exists
Canary mutation	Would a real bug be caught?	The mutation reaches a live check

A deleted parametrize case, a narrowed @pytest.mark.parametrize list, or an added @pytest.mark.skip defeats all four at once. The census is the cheapest gate to run and the one most patches trip first.

Step 1 — Snapshot the collected nodes

Use --collect-only -q and parse node IDs. Two details matter: disable the cache provider so the run is order-stable, and treat exit code 5 ("no tests collected") as a valid — and alarming — result rather than a failure.

#!/usr/bin/env python3
# census.py — write a deterministic snapshot of collected pytest nodes
import argparse, hashlib, json, re, subprocess, sys

NODE_RE = re.compile(r"^(?P<node>[\w./\\-]+::[^\s]+)")

def collect(root: str, extra: list[str]) -> dict:
    cmd = [sys.executable, "-m", "pytest", "--collect-only", "-q",
           "-p", "no:cacheprovider", *extra]
    proc = subprocess.run(cmd, cwd=root, capture_output=True, text=True)
    if proc.returncode not in (0, 5):
        raise SystemExit(f"collection failed ({proc.returncode})\n{proc.stderr[-2000:]}")
    nodes = sorted({m.group("node") for line in proc.stdout.splitlines()
                    if (m := NODE_RE.match(line.strip()))})
    return {"count": len(nodes), "nodes": nodes,
            "digest": hashlib.sha256("\n".join(nodes).encode()).hexdigest()[:16]}

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("root")
    ap.add_argument("--out", required=True)
    ap.add_argument("--pytest-arg", action="append", default=[])
    a = ap.parse_args()
    snap = collect(a.root, a.pytest_arg)
    json.dump(snap, open(a.out, "w"), indent=1)
    print(f"{snap['count']} nodes  digest {snap['digest']}")

Run it on both trees, then diff sets:

python census.py ./base --out base.json
python census.py ./head --out head.json
python - <<'PY'
import json
a, b = (json.load(open(f)) for f in ("base.json", "head.json"))
removed = sorted(set(a["nodes"]) - set(b["nodes"]))
added   = sorted(set(b["nodes"]) - set(a["nodes"]))
print(f"base {a['count']}  head {b['count']}  removed {len(removed)}  added {len(added)}")
for n in removed[:20]: print("-", n)
PY

An illustrative diff from a synthetic patch looks like this — the exact numbers are not a benchmark, only the shape of the output:

base 842 nodes  digest 1f3c9a2e
head 828 nodes  digest 7b0d4410
removed 14   added 0
class breakdown:
  test body deleted            6
  parametrize case narrowed    5
  skip marker added            3   <- not visible here; see Step 2

Step 2 — Add the two channels collection cannot see

Collection has two blind spots, and both are routine in agent patches.

Skips are collected. @pytest.mark.skip and skipif nodes appear in the census and then never run. You need the runner's own report to see them, which means a JUnit XML artifact:

pytest -q --junitxml=artifacts/run.xml

import xml.etree.ElementTree as ET

def executed_and_skipped(path):
    ran, skipped = set(), set()
    for case in ET.parse(path).iter("testcase"):
        node = f"{case.get('classname','')}::{case.get('name','')}"
        (skipped if case.find("skipped") is not None else ran).add(node)
    return ran, skipped

Assertion surface can shrink without touching the node count. A test can keep its name and lose every meaningful check. A cheap proxy is an AST pass counting assert statements per test function. It is a proxy, not a verdict: it misses asserts inside helpers, counts pytest.raises blocks as one unit, and will not follow fixtures.

import ast, pathlib

def assert_density(root):
    out = {}
    for p in pathlib.Path(root).rglob("test_*.py"):
        for fn in ast.walk(ast.parse(p.read_text())):
            if isinstance(fn, ast.FunctionDef) and fn.name.startswith("test"):
                out[f"{p}::{fn.name}"] = sum(isinstance(n, ast.Assert)
                                             for n in ast.walk(fn))
    return out

Flag any node whose density drops while its name stays the same. Treat the flag as a question to the author, not a rejection.

Step 3 — Classify the diff before you argue about it

Not every removal is hostile. Refactors retire cases legitimately, and a gate that blocks them gets disabled within a week. The rule I use:

Change	Verdict	Rationale
Node removed, no waiver	Block	The patch reduced what the suite asks
Node removed, waiver with live witness	Allow	Coverage moved, not evaporated
skip/xfail added, no expiry	Block	Silence without a deadline
Assert density down, node unchanged	Require justification	Oracle weakened in place
Parametrize cardinality down	Block unless case list is data	Same effect as deleting nodes
Node renamed only	Allow, log	Census noise; normalize first

Step 4 — Require witnesses, not prose

Free-text justification is where agents are most convincing and least accountable. Demand a machine-checkable waiver instead: one entry per removed node, a fixed reason, and — for coverage-motivated removals — the node that now covers the same path.

# census_waiver.yaml
- node: tests/billing/test_proration.py::test_zero_day_proration
  reason: duplicate-coverage
  witness: tests/billing/test_proration.py::test_single_day_proration
- node: tests/legacy/test_tax_v1.py::test_rounding_1999
  reason: obsolete-behavior
  witness: tests/tax/test_rounding.py::test_rounding_half_even

REASONS = {"obsolete-behavior", "duplicate-coverage", "environment"}
NEEDS_WITNESS = {"obsolete-behavior", "duplicate-coverage"}

def validate(removed, waivers, head_nodes, passed):
    errors = []
    by_node = {w["node"]: w for w in waivers}
    for node in removed:
        w = by_node.get(node)
        if w is None:
            errors.append(f"removed without waiver: {node}")
            continue
        if w.get("reason") not in REASONS:
            errors.append(f"unknown reason {w.get('reason')!r} for {node}")
        if w["reason"] in NEEDS_WITNESS:
            witness = w.get("witness")
            if witness not in head_nodes:
                errors.append(f"witness not collected: {witness}")
            elif witness not in passed:
                errors.append(f"witness did not pass: {witness}")
    return errors

That is the whole gate: a removed node is a debit, and a waiver must name a credit that exists and is green in the same run.

Step 5 — Freeze the noise before you trust the diff

Census diffs are noisy for reasons unrelated to the patch. Randomised parametrize IDs (ids=lambda) produce a new digest on every run. Unseeded hypothesis generation changes cardinality. Plugin-driven collection order changes the sorted digest only if you forgot to sort — so sort.

Normalise IDs at the source, pin the seed in the invocation (--pytest-arg=-p --pytest-arg=no:randomly for shuffled runners), and record the digest of the untouched baseline twice to confirm it is stable. If the baseline cannot reproduce its own digest, the census gate is not ready; do not relax the threshold, fix the source of nondeterminism. Once it is stable, freeze the census script itself outside the patch scope — an agent that can edit the gate has already passed it.

Running this where it is cheap

The census itself is a few hundred milliseconds. The expensive part is running the executed-node channel across many patch candidates, which is a background job rather than an interactive one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's operator states that it offers free model access and a free server option. I have not benchmarked either, and I would not treat availability as permanent — check the current terms before you design a pipeline around it. The two roles that fit this workflow are narrow. A long-lived census job on the free server option can hold the baseline digest and re-run on each candidate; and free model access is a reasonable place to draft waiver classifications from a diff, because the output is a short structured file a human reviews line by line. What I would not do is let a model approve its own waiver. The verdict table in Step 3 belongs to a person.

If you want to try the split, the artifact to bring is the census_waiver.yaml file — the rest is standard pytest.

Limitations and who should skip this

• Small suites. Below roughly 50 collected nodes, the census job costs more attention than it saves. Read the diff by hand.
• Generated tests. Suites built by parametrisation generators or template expansion produce census churn that drowns the signal; stabilise generation first.
• Skip semantics vary. Some runners report collection errors as skips in JUnit XML. Reconcile the two channels against one known-red node before trusting the union.
• Not a security boundary. Anyone who can edit the census script, the waiver schema, or the CI job can pass it. Keep those files outside the paths an agent is allowed to touch.
• Not a coverage tool. Showing that a witness node exists is not the same as showing it exercises the retired path. Pair it with your existing coverage report if that claim matters.

If your team reviews agent patches by reading the test diff, you probably do not need this yet. If you review by watching a green check appear, start with Step 1 and the waiver validator — nothing else in this post requires new infrastructure.