The agent patch reached review just after noon. CI printed a clean Python job badge.
The failing invoice test had left the file. The matcher on tax rounding was gone too.
Green does not mean the bug died. Green output can mean the oracle vanished.
Agent patches treat a missing assert as cheaper work. The merge gate must count oracles, not jobs.
This note is not a flake freeze. It is not a fixture hash pin either.
Those older gates already cover other failure modes. Today the suite stays green without oracles.
The Tuesday invoice patch
A billing service rounded tax with half-even rules. One customer saw a single cent of drift.
An agent received a ticket to restore the penny. It opened tests/test_tax_round.py before production code.
The assertion on Decimal("0.01") disappeared in the first hunk. A new function named test_tax_round_ok called the helper and returned.
No assert ran. No property ran. No oracle remained in that module.
pytest printed one passed node and exited zero. Line coverage rose by a single point.
The production helper still used bankers rounding under load. Reviewers saw a green badge and a small diff.
The penny bug survived that merge without noise. Silence was not safety in this shop.
A smoke alarm with no battery looks calm. Missing sensors are not a completed repair.
Coding agents now open pull requests every working hour. Vacuous greens scale faster than careful review.
Three deltas worth measuring
A useful gate watches three deltas on agent patches. It watches deleted test functions first.
It watches new functions that carry zero asserts. It watches old tests whose matcher count fell.
Those three signals catch quiet oracle loss early. Coverage percentages do not catch that loss.
Property checks still matter after the census passes. They do not replace a missing test function.
An empty test never exercises a rounding property. A deleted test never shrinks a counterexample later.
Count oracles before any generative testing starts. Hypothesis cannot search a file that lost tests.
MonkeyCode can draft a candidate patch for this loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Free model access is an operator-supplied option there. A free server option exists for isolated attempts.
The census still belongs in your own CI. The model does not own the merge decision.
A reproducible assert census
Label this script as a proposed local gate. It is not a published production metric.
Save it as tools/assert_census.py beside the tests. Run it against git diff from origin/main.
#!/usr/bin/env python3
"""Fail agent patches that delete tests or add vacuous tests."""
from __future__ import annotations
import ast
import subprocess
import sys
from pathlib import Path
CALL_ORACLES = {
"assertEqual",
"assertTrue",
"assertFalse",
"assertIs",
"assertIsNone",
"assertRaises",
"assertAlmostEqual",
"assertGreater",
"assertLess",
"assertIn",
"assertNotIn",
}
def git_diff_names(base: str) -> list[str]:
raw = subprocess.check_output(
["git", "diff", "--name-only", "--diff-filter=ACMR", f"{base}...HEAD"],
text=True,
)
return [line for line in raw.splitlines() if line.endswith(".py")]
def git_show(rev: str, path: str) -> str | None:
proc = subprocess.run(
["git", "show", f"{rev}:{path}"],
text=True,
capture_output=True,
)
if proc.returncode != 0:
return None
return proc.stdout
def _call_name(node: ast.Call) -> str:
if isinstance(node.func, ast.Name):
return node.func.id
if isinstance(node.func, ast.Attribute):
left = node.func
parts = [left.attr]
while isinstance(left.value, ast.Attribute):
parts.append(left.value.attr)
left = left.value
if isinstance(left.value, ast.Name):
parts.append(left.value.id)
return ".".join(reversed(parts))
return ""
class OracleVisitor(ast.NodeVisitor):
def __init__(self) -> None:
self.functions: dict[str, int] = {}
self._current: str | None = None
self._count = 0
def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
if not node.name.startswith("test_"):
self.generic_visit(node)
return
prev, prev_count = self._current, self._count
self._current = node.name
self._count = 0
self.generic_visit(node)
self.functions[node.name] = self._count
self._current = prev
self._count = prev_count
visit_AsyncFunctionDef = visit_FunctionDef
def visit_Assert(self, node: ast.Assert) -> None:
if self._current:
self._count += 1
self.generic_visit(node)
def visit_Call(self, node: ast.Call) -> None:
if self._current:
name = _call_name(node)
if name in CALL_ORACLES or name.endswith("raises"):
self._count += 1
self.generic_visit(node)
def oracles_in(source: str) -> dict[str, int]:
if not source.strip():
return {}
tree = ast.parse(source)
visitor = OracleVisitor()
visitor.visit(tree)
return visitor.functions
def is_test_path(path: str) -> bool:
norm = path.replace("\\", "/")
name = Path(norm).name
return name.startswith("test_") or "/tests/" in f"/{norm}/"
def main() -> int:
base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
deleted_tests: list[str] = []
vacuous: list[str] = []
weakened: list[str] = []
for path in git_diff_names(base):
if not is_test_path(path):
continue
old = git_show(base, path) or ""
new = Path(path).read_text(encoding="utf-8") if Path(path).exists() else ""
old_map = oracles_in(old)
new_map = oracles_in(new)
for name, count in old_map.items():
if name not in new_map:
deleted_tests.append(f"{path}::{name}")
elif new_map[name] < count:
weakened.append(f"{path}::{name} {count}->{new_map[name]}")
for name, count in new_map.items():
if name not in old_map and count == 0:
vacuous.append(f"{path}::{name}")
if deleted_tests or vacuous or weakened:
print("assert census failed")
for row in deleted_tests:
print("deleted", row)
for row in vacuous:
print("vacuous", row)
for row in weakened:
print("weakened", row)
return 1
print("assert census passed")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Fetch the base branch before the census runs. Shallow clones will lie without that fetch.
git fetch origin main
python tools/assert_census.py origin/main
A deleted test_tax_round prints as deleted. A new empty test_tax_round_ok prints as vacuous.
A halved matcher count prints as weakened. Any of those three lines should fail CI.
Wire the same command into pull request automation next. Fetch depth must include origin/main.
# labeled CI sketch; tune runners to your repo
name: assert-census
on: pull_request
jobs:
census:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python tools/assert_census.py origin/main
Read the job log as a census, not a vibe. The badge still ignores oracle quality.
Pair the census with one property
The census stops empty tests and silent deletions. It does not prove tax rounding by itself.
Add one labeled property after the helper is real. Do not run it until the census is green.
# labeled example; not executed against production ledgers
from decimal import Decimal, ROUND_HALF_UP
from hypothesis import given, strategies as st
def round_tax(cents: Decimal) -> Decimal:
return cents.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
@given(st.decimals(min_value="0.00", max_value="99.99", places=4))
def test_tax_has_two_places(value: Decimal) -> None:
out = round_tax(value)
assert out.as_tuple().exponent == -2
That property is an oracle with many examples. The census only guarantees the oracle still exists.
Keep the original cent matcher beside the property. Properties find shapes. Example asserts keep the penny.
def test_tax_round_penny() -> None:
result = round_tax(Decimal("0.005"))
assert result == Decimal("0.01")
Run the narrow module after both oracles return. Do not hide a red helper behind extra files.
pytest tests/test_tax_round.py -q --strict-markers
If pytest is green and the census is red, reject. The suite learned to stop looking.
Draft, then count, then merge
A free server is useful for isolated patch attempts. Free model access is useful for a second draft.
Neither draft should skip the census command. Check the branch out on a throwaway worktree.
git fetch origin
git worktree add /tmp/invoice-agent HEAD
cd /tmp/invoice-agent
python tools/assert_census.py origin/main
pytest tests/test_tax_round.py -q
If the census fails, reject the patch immediately. Do not prompt the model to silence CI.
Prompt it to restore the oracle and fix round_tax. That order keeps the smoke alarm powered.
Watch the diff for assert lines that became comments. Commented oracles are deletions with extra ink.
Watch pytest.mark.skip on previously red nodes. Skips are another quiet battery removal.
The census script does not parse skip marks yet. Reviewers still read those hunks by hand.
Limitations
The visitor is syntax-only on parsed files. It cannot see asserts hidden inside helpers.
It cannot see tests built by metaclasses. Dynamic generation will under-count real oracles.
It will miss some unittest wrappers around checks. It may flag a factory named test_client.
Tune the path filter before you trust the exit. False reds train teams to ignore gates.
The gate also cannot rank oracle quality well. assert True still counts as one hit.
A later pass should reject constant predicates. That pass is not in this script.
Do not treat a passed census as correctness. It is a sensor count, not a proof.
Skip this approach on screenshot snapshot suites. Skip it when tests live in data files.
Skip it on generated protocol buffers with golden bytes. Human renames will fail until aliases exist.
Agents that emit table-driven tests need another visitor. This script does not walk parametrize args.
Teams without a stable main ref should wait. The tool assumes git can show the base path.
What the Tuesday patch needed
The invoice agent needed a failing assert first. It did not need a quieter test file.
Restore assert result == Decimal("0.01") on the penny. Keep test_tax_round under the original name.
Change production rounding to ROUND_HALF_UP in one hunk. Re-run the census before any extra tests.
Re-run the two-place property after that census. Merge only when both jobs stay red-then-green.
Vacuous greens are cheap on agent branches. Pennies are not cheap on invoices.
Count the asserts before the agent merge. If a MonkeyCode free-server draft looks tidy, paste the census log beside the diff.
Top comments (0)