Assertion Loss Is a Regression. Score It in CI.

Agent patches go green by shrinking what the suite still proves. A passing command is not a preserved specification. Score oracle strength on the test AST, then reject the diff when asserts disappear, skips appear, or matchers loosen.

Green CI answers one question: did this revision exit zero. It does not answer whether the agent deleted a boundary, widened pytest.approx, or converted equality into is not None. Those edits are cheaper than fixing the code. They also survive name-based freeze lists, because the test names did not change.

The rest of this article is a local scoring gate. Generation can happen elsewhere. Judgment stays on the checkout.

Four weakening signals that do not need a model

Treat the test tree as data. Four signals are cheap to extract from Python ASTs and pytest marks. None of them require calling a model at merge time.

Signal	Agent-shaped edit	Why CI goes green
Assertion loss	assert result == expected deleted or commented	The failing branch is no longer observed
Matcher loosen	== becomes in, is not None, or pytest.approx with a larger rel	Near-misses pass
Skip inflation	new pytest.skip, pytest.mark.skip, or xfail on touched files	The old failure is quarantined, not fixed
Oracle rewrite	fixture JSON loses keys, expected lists shrink, error strings become prefixes	Characterization becomes a subset of prior behavior

A fifth signal belongs in the same gate: timeout inflation. Raising @pytest.mark.timeout from 1s to 30s hides stalls. Count it as weakening unless a human-owned budget file records the change in a separate commit.

Split generation from scoring

Keep the model off the merge executor. The agent proposes a patch. The checkout scores the patch. Mixing those roles is how oracles get rewritten by the same process that needed a green check.

Proposal (unexecuted against a live vendor): run generation on an isolated workspace, then fetch only a git diff into CI. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit that split: the model can draft on a remote workspace, while the assertion-budget script below runs on the runner that already has git and CPython. The gate does not depend on that vendor. Any isolated generator works if the only artifact that crosses the boundary is a diff.

Numbered workflow

1. Freeze the merge base. Record MERGE_BASE=$(git merge-base origin/main HEAD) and refuse to score a dirty tree.
2. Collect test paths. Limit the scan to tests/ plus any test_*.py the diff already touches. Do not scan generated fixtures the agent just wrote unless they replace a locked file.
3. Parse both sides. For each path, load git show $MERGE_BASE:path and the worktree file. Skip non-Python files here; send those to the fixture hasher in step 6.
4. Diff oracle metrics. Compare assertion counts, skip/xfail marks, pytest.approx tolerances, and timeout marks. Fail on any weakening in a file the agent also changed in src/.
5. Enforce the flake freeze. New skip/xfail on production-touched packages is a hard error unless tests/quarantine.toml changes in a different commit with a human author.
6. Hash locked fixtures. sha256 every file under tests/fixtures/locked/. A content change is a failure unless the commit touches only that lock set and a review label is present.
7. Publish a machine-readable report. CI should print JSON so a later job can trend assertion density per package. Do not parse log prose.

The order matters. Scoring after install-and-test wastes the run when the oracle already shrank. Fail the budget first. Then run the suite.

Artifact: oracle_budget.py

Label: this script is a complete local prototype. It is not a measured production deploy. Run it against two git trees you control.

#!/usr/bin/env python3
"""Fail a revision that weakens pytest oracles vs a git merge base."""
from __future__ import annotations

import argparse
import ast
import hashlib
import json
import subprocess
import sys
from dataclasses import asdict, dataclass
from pathlib import Path


@dataclass
class Oracle:
    asserts: int = 0
    raises: int = 0
    skips: int = 0
    xfails: int = 0
    approx_rel_sum: float = 0.0
    approx_abs_sum: float = 0.0
    timeout_sum: float = 0.0
    none_checks: int = 0


class Visitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.o = Oracle()

    def visit_Assert(self, node: ast.Assert) -> None:
        self.o.asserts += 1
        self._matcher(node.test)
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call) -> None:
        name = self._name(node.func)
        if name in {"pytest.raises", "raises"}:
            self.o.raises += 1
        if name in {"pytest.skip", "skip"}:
            self.o.skips += 1
        if name in {"pytest.xfail", "xfail"}:
            self.o.xfails += 1
        if name in {"pytest.approx", "approx"}:
            self.o.approx_rel_sum += self._kw(node, "rel", 1e-6)
            self.o.approx_abs_sum += self._kw(node, "abs", 1e-12)
        self.generic_visit(node)

    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
        for dec in node.decorator_list:
            if isinstance(dec, ast.Call) and "skip" in self._name(dec.func):
                self.o.skips += 1
            if isinstance(dec, ast.Call) and "xfail" in self._name(dec.func):
                self.o.xfails += 1
            if isinstance(dec, ast.Call) and "timeout" in self._name(dec.func):
                if dec.args:
                    try:
                        self.o.timeout_sum += float(ast.literal_eval(dec.args[0]))
                    except Exception:
                        pass
        self.generic_visit(node)

    visit_AsyncFunctionDef = visit_FunctionDef

    def _matcher(self, test: ast.AST) -> None:
        if isinstance(test, ast.Compare):
            for op, right in zip(test.ops, test.comparators):
                if isinstance(op, ast.Is) and isinstance(right, ast.Constant) and right.value is None:
                    self.o.none_checks += 1
                if isinstance(op, ast.IsNot) and isinstance(right, ast.Constant) and right.value is None:
                    self.o.none_checks += 1

    @staticmethod
    def _name(func: ast.AST) -> str:
        if isinstance(func, ast.Name):
            return func.id
        if isinstance(func, ast.Attribute):
            left = Visitor._name(func.value)
            return f"{left}.{func.attr}" if left else func.attr
        return ""

    @staticmethod
    def _kw(call: ast.Call, key: str, default: float) -> float:
        for kw in call.keywords:
            if kw.arg == key:
                try:
                    return float(ast.literal_eval(kw.value))
                except Exception:
                    return default
        return default


def parse_source(src: str) -> Oracle:
    tree = ast.parse(src)
    v = Visitor()
    v.visit(tree)
    return v.o


def git_show(rev: str, path: str) -> str | None:
    p = subprocess.run(
        ["git", "show", f"{rev}:{path}"],
        capture_output=True,
        text=True,
    )
    if p.returncode != 0:
        return None
    return p.stdout


def weaker(base: Oracle, head: Oracle) -> list[str]:
    reasons = []
    if head.asserts < base.asserts:
        reasons.append(f"asserts {base.asserts}->{head.asserts}")
    if head.raises < base.raises:
        reasons.append(f"pytest.raises {base.raises}->{head.raises}")
    if head.skips > base.skips:
        reasons.append(f"skips {base.skips}->{head.skips}")
    if head.xfails > base.xfails:
        reasons.append(f"xfails {base.xfails}->{head.xfails}")
    if head.approx_rel_sum > base.approx_rel_sum + 1e-15:
        reasons.append("approx rel increased")
    if head.approx_abs_sum > base.approx_abs_sum + 1e-15:
        reasons.append("approx abs increased")
    if head.timeout_sum > base.timeout_sum + 1e-9:
        reasons.append("timeout budget increased")
    if head.none_checks > base.none_checks and head.asserts <= base.asserts:
        reasons.append("equality replaced by None checks")
    return reasons


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", required=True)
    ap.add_argument("--tests", default="tests")
    ap.add_argument("--locked-fixtures", default="tests/fixtures/locked")
    args = ap.parse_args()

    report = {"files": [], "fixture_drift": [], "failed": False}
    root = Path(args.tests)
    for path in sorted(root.rglob("test_*.py")):
        rel = path.as_posix()
        old = git_show(args.base, rel)
        new = path.read_text(encoding="utf-8")
        base_o = parse_source(old) if old is not None else Oracle()
        head_o = parse_source(new)
        reasons = weaker(base_o, head_o) if old is not None else []
        row = {"path": rel, "base": asdict(base_o), "head": asdict(head_o), "reasons": reasons}
        report["files"].append(row)
        if reasons:
            report["failed"] = True

    locked = Path(args.locked_fixtures)
    if locked.exists():
        for path in sorted(locked.rglob("*")):
            if not path.is_file():
                continue
            rel = path.as_posix()
            old = git_show(args.base, rel)
            if old is None:
                report["fixture_drift"].append({"path": rel, "reason": "new locked fixture"})
                report["failed"] = True
                continue
            if hashlib.sha256(old.encode()).hexdigest() != sha256_file(path):
                report["fixture_drift"].append({"path": rel, "reason": "locked bytes changed"})
                report["failed"] = True

    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 1 if report["failed"] else 0


if __name__ == "__main__":
    raise SystemExit(main())

Invoke it before the test runner:

#!/usr/bin/env bash
set -euo pipefail
MERGE_BASE=$(git merge-base origin/main HEAD)
python3 oracle_budget.py --base "$MERGE_BASE"
pytest -q

The script scores syntax, not semantics. A helper that still asserts inside a private function will look like assertion loss at the call site. That is acceptable. Call-site asserts are the public oracle. Helpers can be locked in a second pass if the repo relies on them.

Property checks stay outside the agent write set

Oracle scoring does not replace properties. It only stops the agent from deleting the checks you already had. Keep property files in a tree the agent cannot write, for example tests/properties/ owned by humans and CI bots with a path filter.

A minimal Hypothesis-style skeleton (proposal, not a measured suite):

# tests/properties/test_order_invariants.py
from hypothesis import given, strategies as st
from billing.totals import line_total

@given(st.decimals(min_value="0.01", max_value="10000", places=2), st.integers(min_value=1, max_value=50))
def test_line_total_scales_with_qty(price, qty):
    one = line_total(price, 1)
    many = line_total(price, qty)
    assert many == one * qty

Two rules keep this honest. The agent may add examples under tests/examples/ that exercise a property. The agent may not edit the @given signature, the strategy bounds, or the assertion. If a property fails after the patch, the counterexample belongs in tests/examples/ as a regression, not as a skip.

Fixtures: lock bytes, not filenames

Filename freezes fail when the agent adds expected.v2.json and points the test at it. Hash the bytes the test actually loads. The --locked-fixtures walk above is the floor. Pair it with a tiny loader that refuses unlocked paths.

# tests/support/locked_loader.py
from pathlib import Path
import json

LOCKED = Path(__file__).resolve().parents[1] / "fixtures" / "locked"

def load_locked(name: str) -> dict:
    path = (LOCKED / name).resolve()
    if not str(path).startswith(str(LOCKED.resolve())):
        raise ValueError("fixture escaped lock root")
    if not path.is_file():
        raise FileNotFoundError(name)
    return json.loads(path.read_text(encoding="utf-8"))

New fixtures can land in tests/fixtures/draft/ and stay unscored. They do not become oracles until a human moves them into locked/ in a commit that does not also change src/.

Flake freeze: budget, not a growing skip list

A skip is a merged failure with a comment. Cap it. Store the cap in a human-owned file the scoring job reads.

# tests/quarantine.toml
max_skips_per_package = 2
max_xfails_per_package = 1
require_separate_commit = true

Policy, stated as rules a CI job can enforce:

1. The agent patch commit cannot increase skips or xfails in any package it also modified under src/.
2. A human may add a quarantine entry in a follow-up commit. That commit cannot change production code.
3. When the package is already at max_skips_per_package, new skips fail even in the human commit. Delete or fix one first.
4. Time-based expiry is optional. A budget is not. Expiry without a cap just rotates the same flake.

Bind the freeze to package paths, not to test names. Name churn is how agents evade a list. Package budgets still move if the agent relocates files; combine with a path-rename detector (git log --follow is too slow for CI; compare git diff --name-status for R* rows and carry the budget with the rename).

What the JSON report is for

Do not stop at a boolean. Trend four numbers per package: assertion count, skip count, approx tolerance sum, locked-fixture hash. A week of falling asserts with stable skip counts is still oracle loss. A week of stable asserts with rising none_checks is matcher loosen. Both should page the same review list.

Sample row the script already emits:

{
  "path": "tests/test_totals.py",
  "base": {"asserts": 14, "skips": 0, "xfails": 0, "none_checks": 1},
  "head": {"asserts": 11, "skips": 1, "xfails": 0, "none_checks": 3},
  "reasons": ["asserts 14->11", "skips 0->1", "equality replaced by None checks"]
}

That object is the review artifact. Paste it on the pull request. Do not summarize it in marketing language.

Limitations

AST scoring misses asserts built with exec, custom checkers, and helper libraries that wrap assert. It also misses tests written in other languages. Matcher loosen beyond None and approx needs extra visitors. The fixture hasher does not understand semantic JSON: reordering keys changes the hash even when values match, which is the conservative outcome, not a pretty one.

The gate cannot see tests the agent never ran. A patch that deletes a slow file from pytest.ini will still look strong if you only parse remaining files. Diff pytest.ini, pyproject.toml, and any conftest.py collection hooks as a separate checklist. Treat collection-config edits as oracle edits.

Property files that the agent can still write are theater. Path filters in CODEOWNERS or a CI git diff --name-only deny-list are part of the method, not optional hygiene.

Who should not use this

Do not use assertion budgets as a substitute for reading the production diff. A patch can add asserts that encode the bug (assert broken_fn() == 3 after the agent changed the function to return 3). Density can rise while meaning dies. Reviewers still read new asserts.

Do not use it on repos whose tests are 90% snapshot blobs with no literals. Hash the snapshots, then add properties; the AST visitor will look empty and pass for the wrong reason.

Do not point the generator at the same credentials that can push to main. Isolation is the control. A free remote workspace is one way to get that isolation. It is not a reason to skip the budget job.

If the suite is already skip-heavy, run the script once on main, publish the baseline JSON, and pay down skips before enforcing --failed as a merge blocker. Turning the gate on against a polluted baseline just freezes the pollution.

The scoring job is small enough to keep next to the tests. Generation can stay on a free model path and a free server option when the laptop should not host the agent. The only number that matters at merge is whether the oracle got smaller.