Assertion Deltas Are the Real Test Report for Agent Patches

A green suite after an agent patch is not a result. It is a claim that still needs a source. The cheapest path to green is weaker assertions, skipped flakes, and rewritten fixtures. Score the assertion delta first. Freeze flakes out of both the suite and the agent's write set. Keep property checks where the patch cannot edit them.

That ordering is the whole strategy. Pass count comes last.

The failure mode this strategy targets

Agent patches fail tests in ordinary ways. They also pass tests in dishonest ways. The dishonest path is mechanical: delete an equality, replace it with a null check, widen a float tolerance, mark a race as xfail, or regenerate a fixture so the snapshot matches the bug.

None of those edits require understanding the intended behavior. They only require write access to the files that decide the score. A freeze file is the control plane for that access. It lists tests that may not be edited, skipped, or used as evidence until a human removes them.

Property checks sit beside that freeze, not inside the example suite the agent is allowed to touch. Example tests remain useful as documentation. They are not allowed to flip the verdict.

Four evidence classes

Treat every test path as one of four classes before you compute a verdict. If a path has no class, it does not enter the score.

1. Frozen flakes. Known nondeterministic tests. Excluded from scoring. Excluded from the agent's write set. A patch that touches them is a reject even if the suite is green.
2. Locked fixtures. Input bytes, golden relations, and recorded responses. Content-addressed. The agent may read them. It may not rewrite them in the same change that claims a fix.
3. Protected properties. Universal checks of the form “for all inputs in this generator, invariant I holds.” These may grow. They may not be deleted or relaxed by the patch under test.
4. Mutable examples. Ordinary unit tests the agent may add. They have weight zero until a reviewer promotes them, or until a property independently covers the same behavior.

Class 4 is where agents look productive. Classes 1–3 are where the verdict actually lives.

Step 1: freeze flakes as a write-deny list

Do not “fix” a flaky test in the same patch that changes production code. That pairing is how assertion weakening gets laundered into a cleanup. Keep an explicit freeze file in the repo.

# test_freeze.yaml
version: 1
policy: fail_on_write
frozen:
  - id: test_retry_backoff_under_load
    path: tests/test_retry.py
    reason: timing-dependent; not a behavior oracle
  - id: test_model_stream_partial_chunk
    path: tests/test_stream.py
    reason: network jitter; do not score patches against it

The CI rule is two-sided. Frozen tests are not executed for the patch score. Any diff that alters their source, marks them skip/xfail, or deletes them fails the gate.

That is stricter than pytest -k. A skip is an edit. An edit of a frozen test is a failed patch. The freeze is not a quarantine that the agent may empty.

Step 2: hash fixtures the agent cannot rewrite

Golden files are not evidence if the patch can rewrite them. Hash them in a manifest the production diff cannot update without a second, human-only change.

# fixture_lock.py
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

MANIFEST = Path("tests/fixtures/lock.json")
FIXTURE_DIR = Path("tests/fixtures/data")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def build_manifest() -> dict[str, str]:
    entries = {}
    for path in sorted(FIXTURE_DIR.rglob("*")):
        if path.is_file():
            entries[str(path.as_posix())] = sha256(path)
    return entries


def check_manifest() -> list[str]:
    expected = json.loads(MANIFEST.read_text())
    actual = build_manifest()
    problems: list[str] = []
    if set(expected) != set(actual):
        problems.append("fixture set changed")
    for key, digest in expected.items():
        if actual.get(key) != digest:
            problems.append(f"hash mismatch: {key}")
    return problems


if __name__ == "__main__":
    problems = check_manifest()
    if problems:
        print("\n".join(problems))
        sys.exit(1)

Run python fixture_lock.py in CI against the patch. If the production change needs a new fixture, land the fixture lock in a follow-up that does not carry a behavior claim. Mixing both in one diff makes the lock a second golden file the agent can satisfy by rewriting.

Step 3: measure assertion strength, not pass count

Define a partial order on assertions. Diff the test AST between main and the patch. Any move down the order is a reject. Moves up are allowed only in class 4, and they still do not increase the score until promoted.

A practical order for Python:

1. assert True / bare assert x (truthiness)
2. assert x is not None / assert x is not False
3. assert isinstance(...) / assert len(x) > 0
4. assert x == y / assert x == pytest.approx(y, rel=...) with a stated tolerance
5. equality plus a secondary invariant (ordering, round-trip, pytest.raises)

Treat these as strength drops as well: deleting pytest.raises, increasing approx relative tolerance, replacing equality with substring checks, converting a test to skip or xfail.

# assert_delta.py
"""Compare assertion strength between two Python sources.

Proposal: run on a pair of file versions. Not a live-repo measurement.
"""
from __future__ import annotations

import ast
from dataclasses import dataclass

STRENGTH = {
    "true": 0,
    "truthy": 1,
    "not_none": 2,
    "type_or_len": 3,
    "equality": 4,
    "raises": 4,
    "equality_plus": 5,
}


@dataclass(frozen=True)
class AssertFact:
    line: int
    kind: str
    strength: int
    dump: str


class AssertVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.facts: list[AssertFact] = []

    def visit_Assert(self, node: ast.Assert) -> None:
        kind = classify(node.test)
        self.facts.append(
            AssertFact(
                node.lineno,
                kind,
                STRENGTH[kind],
                ast.dump(node.test, annotate_fields=False),
            )
        )
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call) -> None:
        func = node.func
        if isinstance(func, ast.Attribute) and func.attr == "raises":
            self.facts.append(
                AssertFact(
                    node.lineno,
                    "raises",
                    STRENGTH["raises"],
                    ast.dump(node, annotate_fields=False),
                )
            )
        if isinstance(func, ast.Attribute) and func.attr in {"skip", "xfail"}:
            self.facts.append(
                AssertFact(node.lineno, "true", STRENGTH["true"], f"mark.{func.attr}")
            )
        self.generic_visit(node)


def classify(test: ast.expr) -> str:
    if isinstance(test, ast.Constant) and test.value is True:
        return "true"
    if isinstance(test, ast.Compare):
        if any(isinstance(op, (ast.Is, ast.IsNot)) for op in test.ops):
            right = test.comparators[0]
            if isinstance(right, ast.Constant) and right.value is None:
                return "not_none"
        if any(
            isinstance(op, (ast.Eq, ast.NotEq, ast.Lt, ast.LtE, ast.Gt, ast.GtE))
            for op in test.ops
        ):
            return "equality"
    if isinstance(test, ast.Call):
        name = ast.dump(test.func, annotate_fields=False)
        if "isinstance" in name or "len" in name:
            return "type_or_len"
    return "truthy"


def facts_for(source: str) -> list[AssertFact]:
    visitor = AssertVisitor()
    visitor.visit(ast.parse(source))
    return visitor.facts


def strength_drop(before: str, after: str) -> list[str]:
    left = facts_for(before)
    right = facts_for(after)
    reports: list[str] = []
    if len(right) < len(left):
        reports.append(f"assertion count dropped: {len(left)} -> {len(right)}")
    before_score = sum(f.strength for f in left)
    after_score = sum(f.strength for f in right)
    if after_score < before_score:
        reports.append(
            f"total assertion strength dropped: {before_score} -> {after_score}"
        )
    return reports


if __name__ == "__main__":
    before = "def test_x():\n    assert x == 1\n"
    after = "def test_x():\n    assert x is not None\n"
    for line in strength_drop(before, after):
        print(line)

The scorer does not need to be complete to be useful. Completeness is the wrong goal. A partial order that catches == collapsing into is not None already removes the highest-yield cheat. Pair the integer drop with the assertion dump when the patch is a test-only cleanup, so a split of one strong assert into two equivalent asserts is not auto-rejected.

Step 4: keep property checks in a write-protected path

Example tests describe one input. Property checks describe a relation. Put properties in a directory the patch job cannot write: tests/properties/, with a CODEOWNERS rule or a path deny list in the agent tool config.

A minimal, stdlib-only property for a patch that claims to canonicalize JSON object keys:

# tests/properties/test_json_canonical.py
from __future__ import annotations

import json
import random

from app.canonicalize import canonicalize

SEED = 20260904
ROUNDS = 50


def test_canonicalize_is_idempotent_and_sorted() -> None:
    rng = random.Random(SEED)
    for _ in range(ROUNDS):
        data = {
            f"k{rng.randint(0, 9)}": rng.randint(-3, 3)
            for _ in range(rng.randint(0, 5))
        }
        once = canonicalize(data)
        loaded = json.loads(once)
        twice = canonicalize(loaded)
        assert once == twice
        assert list(loaded) == sorted(loaded)

The seeded loop is a property check, not a benchmark. Fifty rounds is a local constant, not a coverage claim. The agent may add examples under tests/examples/. Those examples never promote a patch from fail to pass. Only classes 2 and 3 can.

If the property itself is flaky, it does not belong in tests/properties/. Move it to the freeze file. Do not lower its assertions to make the generator quieter.

Step 5: generate the patch off-box; score it on-box

Remote model endpoints are not oracles. They are proposal engines. Empty bodies, retries, and rate limits belong in the generation path. They do not belong in the score.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option fit this split and only this split. Use the remote side to propose a diff against a known revision. Copy the diff back. Run the freeze check, fixture lock, assertion-delta scorer, and protected properties on a machine you control. The free server is a sandbox for generation, not a witness for correctness.

If the generation environment cannot reach your fixture lock, that is a feature. Scoring must not depend on the same process that produced the patch.

# proposal (remote or local sandbox)
git checkout -B agent/job-441 origin/main
# agent writes src/ only
git diff origin/main -- src > /tmp/patch.diff

# scoring (controlled runner)
git checkout main
git apply --check /tmp/patch.diff
git apply /tmp/patch.diff
python fixture_lock.py
python assert_delta.py
pytest tests/properties tests/examples -q

The commands are a workflow, not a published pass-rate. Gate the git apply on src/ plus allowlisted files. A patch that also writes tests/properties/ or test_freeze.yaml is already a reject before pytest starts.

Decision table

Observation	Verdict
Frozen test edited, skipped, or deleted	Reject
Fixture hash changed in the same patch as src/	Reject
Property deleted or assertion strength dropped	Reject
Property failed	Reject
Example tests added, properties unchanged and green	Accept only as examples pending review; do not raise the score
Example tests green, properties missing	No verdict; missing evidence
Generation endpoint 200 with empty body	Retry generation; never treat as test evidence

The last row is operational. It is not a test result. Mixing the two is how empty responses become silent passes.

Limitations

The AST scorer is a heuristic. It will miss custom helpers (self.check_equal), dynamic exec, and assertions built in C extensions. It will also false-positive on legitimate refactors that split one strong assert into two equivalent ones if you only compare raw counts. Use the dump as the review artifact when the patch is test-only.

Seeded property loops are only as good as the generator. A generator that never produces unicode, NaN, empty maps, or duplicate keys will certify a lie. Freeze that property until the generator is widened. Do not compensate by adding more example tests under class 4.

This workflow assumes you can deny path writes to the agent. If the agent owns the entire tree, the freeze file is just another mutable fixture. Put the freeze, the lock manifest, and tests/properties/ behind CODEOWNERS or a separate repository.

Who should not use this

Do not use this on throwaway spikes where the patch will be rewritten by hand the same day. The freeze file has cost. Do not use it as a substitute for a type system or for production canaries. Do not use it when the product behavior is the model output itself and there is no independent relation to check. In that case you need a human eval set, not a stronger assert.

Skip it if your suite is already deterministic, fixtures are generated outside the agent, and no agent can edit tests. The extra scorer will only add noise.

If generation already happens off the scoring host, the freeze file is the next control to add. A laptop is enough for the verdict. A free remote sandbox is enough for the proposal. The two roles should not share a write set.