Tautologies Are Not Tests. Classify Agent Patches Before Merge

Test count is a vanity metric for agent patches. A diff that adds twelve tests and zero failure modes has not increased the suite's power. Classify every new test as tautology, characterization, negative-path, or property, and refuse the merge when production code grows while the last two classes do not.

Agents are fluent at restating the implementation. They are weak at naming what must not happen. The gate has to score that gap, not the file list.

What a tautology looks like

A tautology is a test whose expected value is computed by the same code path under review. It cannot fail unless the process crashes. It is not a flake. It is not an oracle leak. It is a statement that f(x) == f(x).

Three shapes show up in agent diffs. The first inlines the production function on both sides of assert. The second asserts only that a mock was called. The third commits a golden file in the same patch that created the output.

Same-PR goldens are not characterization. Characterization needs a corpus frozen before the patch exists. If the agent writes both the function and the snapshot, the snapshot is a tautology with extra steps.

Four labels, one density number

Give every new test exactly one label. Do not allow “unit” or “e2e” as substitutes. Those describe where the test runs, not what it constrains.

1. Tautology — expected value is derived from the code under review, or the test only inspects mock call counts.
2. Characterization — expected value comes from a corpus or lockfile that predates the patch.
3. Negative-path — the test forces a named error, a rejected input, a timeout, or a permission miss.
4. Property — the test asserts an invariant across generated inputs, not a single example.

Compute one ratio on the production side of the diff:

negative_path_density =
  (new_negative_path_tests + new_property_tests)
  / max(1, added_production_loc)

A useful default: density must be greater than 0 whenever added_production_loc >= 20, and at least one negative-path test must exist whenever the patch introduces a new except, catch, or HTTP error branch. Tune the threshold per repo. Do not tune it to zero.

Decision table

Patch signal	Required labels	Gate
Production LOC up, tests only tautology or same-PR golden	none accepted	fail
New except Exception / bare catch	negative-path on a named error	fail until the bare catch is gone
Pure refactor, behavior lockfile unchanged	characterization replay	pass if replay matches
New public function	one negative-path and one property or characterization	fail if either missing
Test-only patch	any non-tautology	pass; still reject tautology-only

The table is the policy. The classifier below is a helper. Humans still own the labels when the helper is unsure.

Workflow

1. Freeze a characterization corpus on main before the agent runs. Hash it. Do not let the agent rewrite the hash in the same PR.
2. Require pytest markers on every new test: tautology, characterization, negative_path, or property.
3. Run a diff classifier in CI. Fail closed on unlabeled new tests.
4. Reject bare exception handlers. Named errors only.
5. Replay the frozen corpus against the patched code. A mismatch is a behavior change, not a test failure to delete.
6. Publish the density number on the PR. Do not publish raw test counts without it.

The order matters. If you classify after the agent “fixes” failing tests, you will label tautologies as characterization because the snapshot already moved.

Marker contract

# pytest.ini
[pytest]
markers =
    tautology: restates the implementation; does not count toward density
    characterization: locked corpus that predates this patch
    negative_path: named failure mode
    property: invariant over generated inputs

# tests/test_ledger.py
import pytest
from ledger import Ledger, Insufficient, UnknownAccount

@pytest.mark.negative_path
def test_debit_unknown_account_raises():
    book = Ledger.from_pairs([("a", 10)])
    with pytest.raises(UnknownAccount):
        book.debit("missing", 1)

@pytest.mark.negative_path
def test_debit_insufficient_raises():
    book = Ledger.from_pairs([("a", 10)])
    with pytest.raises(Insufficient):
        book.debit("a", 11)

@pytest.mark.property
def test_debit_credit_round_trip_preserves_sum():
    # Proposal: replace with Hypothesis in repos that already depend on it.
    pairs = [("a", 4), ("b", 6), ("c", 0)]
    book = Ledger.from_pairs(pairs)
    before = book.total()
    book.debit("a", 3)
    book.credit("b", 3)
    assert book.total() == before

A tautology the gate should refuse:

@pytest.mark.tautology
def test_total_matches_implementation():
    pairs = [("a", 4), ("b", 6)]
    book = Ledger.from_pairs(pairs)
    assert book.total() == sum(v for _, v in pairs)  # production uses the same sum

If the production total() is sum(...), this test cannot detect a change to grouping, currency, or withheld holds. Delete it or replace it with a property that includes holds.

Classifier (proposed, unexecuted)

Treat the following as a proposed CI helper, not as a measured production run. It reads a unified diff from stdin and exits non-zero when the policy fails.

#!/usr/bin/env python3
"""classify_agent_tests.py — proposed gate, not a published score."""
from __future__ import annotations

import re
import sys
from collections import Counter

MARKERS = ("tautology", "characterization", "negative_path", "property")
MARKER_RE = re.compile(
    r"@pytest\.mark\.(" + "|".join(MARKERS) + r")"
)
EXCEPT_RE = re.compile(r"^\+\s*except\s+(Exception|err|e)?\s*:\s*$")
PROD_RE = re.compile(r"^\+.*", re.M)
TEST_FILE_RE = re.compile(r"(?:^|/)(tests?/.*test_.*\.py|.*_test\.py)$")

def parse(diff: str) -> tuple[int, Counter, int, int]:
    prod_loc = 0
    markers: Counter[str] = Counter()
    unlabeled = 0
    bare_except = 0
    current_is_test = False
    pending_def = False

    for line in diff.splitlines():
        if line.startswith("+++ "):
            path = line[4:].strip()
            current_is_test = bool(TEST_FILE_RE.search(path))
            continue
        if line.startswith("+") and not line.startswith("+++"):
            body = line[1:]
            if not current_is_test:
                if body.strip() and not body.startswith("#"):
                    prod_loc += 1
                if EXCEPT_RE.match(line):
                    bare_except += 1
                continue
            if re.match(r"\s*def test_", body):
                pending_def = True
            hit = MARKER_RE.search(body)
            if hit:
                markers[hit.group(1)] += 1
                pending_def = False
            elif pending_def and body.strip().startswith("def test_"):
                unlabeled += 1
                pending_def = False
    return prod_loc, markers, unlabeled, bare_except

def main() -> int:
    diff = sys.stdin.read()
    prod_loc, markers, unlabeled, bare_except = parse(diff)
    useful = markers["negative_path"] + markers["property"]
    density = useful / max(1, prod_loc)
    print(f"prod_loc={prod_loc} markers={dict(markers)} "
          f"unlabeled={unlabeled} bare_except={bare_except} density={density:.4f}")
    if unlabeled:
        print("fail: unlabeled new tests")
        return 1
    if bare_except:
        print("fail: unnamed except in production diff")
        return 1
    if prod_loc >= 20 and useful < 1:
        print("fail: production grew without negative-path or property tests")
        return 1
    if markers["tautology"] and useful == 0 and prod_loc > 0:
        print("fail: tautology-only test delta")
        return 1
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Wire it as a command, not as a dashboard widget.

git fetch origin main
git diff origin/main...HEAD | python classify_agent_tests.py

The parser is greedy and line-based. It will mis-count multi-decorator tests if the marker sits below def. Put markers above the function. That is a style rule, not a suggestion.

Fixtures without moving the oracle

Characterization fixtures belong on main. Hash the directory in CI. A patch may add rows only through a separate, reviewed corpus PR. Agent patches may read the corpus. They may not rewrite it to match a new bug.

# tools/corpus_hash.py — proposed
from pathlib import Path
import hashlib, sys

def digest(root: Path) -> str:
    h = hashlib.sha256()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        h.update(path.relative_to(root).as_posix().encode())
        h.update(b"\0")
        h.update(path.read_bytes())
    return h.hexdigest()

if __name__ == "__main__":
    expected = Path("tests/corpus.sha256").read_text().strip()
    actual = digest(Path("tests/corpus"))
    if actual != expected:
        print(f"corpus hash mismatch {actual} != {expected}")
        sys.exit(1)

If the agent “updates fixtures to match new behavior,” the hash check fails. Behavior changes then go through an explicit corpus PR, not through the patch that introduced the change.

Where a free agent session fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are enough to draft marker placements and to run the classifier against one branch without attaching the job to a billed runner. They do not label tests for you, and they do not prove density. The gate above still runs in your CI. Use the free session to propose markers; keep merge authority on the density number and the corpus hash.

Limitations

The classifier does not understand semantics. A test marked negative_path can still assert the wrong exception. A property can still use a single example. Marker lying is cheaper than writing a real failure mode, so review a sample of labels on every PR until the team stops lying.

Line-based LOC is a blunt denominator. Generated code, import reshuffles, and comment-only lines will move density. Exclude generated paths in the diff filter before you argue about the threshold.

Characterization corpora freeze bugs as well as behavior. If main is wrong, the hash will protect the wrongness. Do not use this workflow as a substitute for a failing production repro.

Who should not use this

Skip the density gate on throwaway scripts, on spikes with no merge, and on repos that do not yet have a named error type. Skip it if the only tests are manual. Skip it if you cannot stop the agent from editing tests/corpus.sha256 in the same PR — the hash is then theater.

Do not replace code review with the script. The script catches unlabeled tests, bare except, and tautology-only deltas. It will not catch a well-marked test that asserts the wrong business rule.

The merge question is simple. Did production code grow a failure mode the suite can name? If the agent cannot name one, the patch is not tested. It is narrated.

Comments

[Reid Marlow]

Same-PR goldens are the failure I keep seeing in agent PRs. The model writes the function, then writes a snapshot that matches it, then the suite stays green while grouping or withheld holds change. Freezing the corpus hash on main before the agent runs is the part that actually bites. I would also fail a patch that adds a new except branch without a named negative-path test, even if density is technically above zero from an unrelated property test on a helper.

[ANP2 Network]

The four labels classify test shape. Provenance is enforced for exactly one of them, characterization, and then it disappears for the two labels that actually feed negative_path_density. So the gate ends up counting assertions whose authority can come entirely from the patch being judged.

A negative-path test written by the agent that wrote the implementation inherits that agent's model of what can fail. If the patch introduces raise UnknownAccount and then adds pytest.raises(UnknownAccount), the assertion is f(x) == f(x) at the specification level. The syntax passes. The specification is still circular. It is a same-PR golden wearing a different label. Your own principle applied consistently would say a named error counts toward density only if the symbol resolves on the pre-patch tree, or the failure mode traces to something outside the patch, such as an incident or a spec clause. Otherwise naming an exception supplies its own justification.

Of the four labels, property is the most exposed. It reads as the strongest. Generated inputs multiply the checks without saying where the invariant came from. "Debit then credit preserves the sum" is recoverable by reading the two methods, and generating a thousand amounts does not make that expectation independent of them. The invariants worth demanding are the ones nobody can reconstruct from the code, which is exactly the region your line about agents being weak at naming what must not happen predicts they will miss. Generation does not supply provenance.

The bare-catch row also creates a measurable incentive to make the code worse. The cheapest way past it is to narrow except Exception to an error invented in the same patch, then assert that error. Both cells go green. Density goes up. A failure the broad handler used to swallow now escapes as an uncaught crash. What is worth scoring instead is whether an input the suite can actually construct reaches the named error, and whether the narrowing changed which inputs arrive at the handler at all. A test that raises the invented error directly answers neither.

For a genuinely new subsystem, where every error symbol is new by definition and there is no pre-patch tree to resolve against, is provenance checkable in CI at all, or does the gate fall back to treating an externally supplied spec as the authority?