Record the Public Surface, Then Count Unique Predicates

Agent patches fail after a green pipeline more often than they fail in review. The suite grew. The predicates did not. Order-dependent leaks were filed as flakes instead of being classified as shared-state bugs. Merge on a hash-stable public-surface corpus and a uniqueness count of assertions. Do not merge on test cardinality.

This is a proposed gate, not a war story. Every script below is labeled as an example. Run it on a throwaway tree before you attach it to a required check.

The signal that inflates

Test count is easy to game. An agent can add twenty functions that all assert result == expected for the same equality. CI reports more coverage. The oracle did not get stronger.

A second failure mode is quieter. Shared clocks, module-level caches, and insertion-ordered sets make a test pass alone and fail after a shuffle. Quarantining that test hides the leak. The public surface still drifts under the next patch.

Green is necessary. It is not sufficient.

Three gates, in order

Stop at the first failure. Do not skip ahead because the later gate is cheaper.

1. Replay a recorded behavior corpus against the public surface. Any hash change outside an explicit allowlist is a merge block.
2. Diff the test tree and count unique predicates. Files and test names do not count.
3. Run a shuffle-repeat campaign. Failures that depend on order are environment leaks. They are not freeze candidates.

The gates are independent on purpose. A corpus can stay stable while the test file doubles with duplicate asserts. A unique-predicate count can rise while a global cache poisons later cases. You need all three.

Gate 1: freeze the public surface, not a flake

Pick the functions callers actually import. Record canonical JSON for a fixed input set. Store hashes, not pretty-printed dumps, so whitespace noise cannot pass as a behavior change.

Proposed example: a tiny registry of public callables and a JSONL corpus.

# record_corpus.py — proposed example, not production telemetry
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any, Callable

# Replace with the real public surface you are willing to freeze.
PUBLIC: dict[str, Callable[..., Any]] = {}


def canonical(value: Any) -> str:
    return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)


def record(path: Path, cases: list[dict[str, Any]]) -> None:
    rows = []
    for case in cases:
        fn = PUBLIC[case["name"]]
        got = fn(*case.get("args", []), **case.get("kwargs", {}))
        payload = canonical(got)
        rows.append({
            "name": case["name"],
            "args": case.get("args", []),
            "kwargs": case.get("kwargs", {}),
            "sha256": hashlib.sha256(payload.encode()).hexdigest(),
        })
    path.write_text("\n".join(json.dumps(r, sort_keys=True) for r in rows) + "\n")

Replay is the gate. The allowlist is a reviewed file, not a comment in the PR.

# replay_corpus.py — proposed example
import hashlib
import json
import sys
from pathlib import Path

ALLOW = Path("corpus.allowlist")  # one sha256 per line, reviewed


def replay(corpus: Path) -> int:
    allowed = {line.strip() for line in ALLOW.read_text().splitlines() if line.strip()}
    failures = 0
    for line in corpus.read_text().splitlines():
        row = json.loads(line)
        fn = PUBLIC[row["name"]]
        got = fn(*row["args"], **row["kwargs"])
        digest = hashlib.sha256(canonical(got).encode()).hexdigest()
        if digest != row["sha256"] and digest not in allowed:
            print(f"BLOCK {row['name']} expected={row['sha256'][:12]} got={digest[:12]}")
            failures += 1
    return failures


if __name__ == "__main__":
    sys.exit(1 if replay(Path("corpus.jsonl")) else 0)

Record the corpus on main before the agent starts. Replay after the patch. If a behavior change is intentional, the allowlist row must name the case and the reviewer. An empty allowlist is a valid policy.

Gate 2: count unique predicates

A predicate is the comparison the test actually makes. assert add(2, 2) == 4 and assert add(3, 1) == 4 share a shape. They are not the same predicate once you include the left-hand call and the constant. Duplicate shapes with identical constants are the inflation to reject.

Proposed example: walk assert nodes and hash a normalized dump. Extend the visitor if your suite uses pytest.raises or custom helpers. Until you do, treat helper-only tests as unparsed and fail closed.

# unique_predicates.py — proposed example
import ast
import hashlib
from pathlib import Path

class Predicates(ast.NodeVisitor):
    def __init__(self) -> None:
        self.found: list[str] = []
        self.unparsed = 0

    def visit_Assert(self, node: ast.Assert) -> None:
        dump = ast.dump(node.test, include_attributes=False)
        self.found.append(hashlib.sha256(dump.encode()).hexdigest())
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call) -> None:
        name = ast.dump(node.func, include_attributes=False)
        if "raises" in name or "warns" in name:
            self.unparsed += 1
        self.generic_visit(node)


def scan(root: Path) -> tuple[int, int, int]:
    visitor = Predicates()
    files = 0
    for path in root.rglob("test_*.py"):
        files += 1
        visitor.visit(ast.parse(path.read_text()))
    unique = len(set(visitor.found))
    return files, unique, visitor.unparsed

Wire it to the test diff, not the whole tree. A useful floor is local: unique_after >= unique_before, and unparsed == 0 on the added lines. A global floor is optional. It will punish deletions of dead tests, which you may want.

Command-level check against main:

git fetch origin main
git diff --name-only origin/main...HEAD -- 'tests/**/*.py' '**/test_*.py'
python unique_predicates.py --base origin/main --head HEAD --fail-on-unparsed

If unique predicates did not rise and the corpus did not change, the agent only restated existing oracles. That is not a merge.

Gate 3: shuffle-repeat classifies leaks

Do not freeze a test because it failed once. Reorder the suite. Repeat it. If failure tracks order, you have a leak: leftover files, cached modules, unseeded time, or a singleton.

Proposed example: a small runner that executes collected node ids in shuffled permutations.

# shuffle_repeat.py — proposed example
import itertools
import random
import subprocess
import sys

def nodeids() -> list[str]:
    out = subprocess.check_output([sys.executable, "-m", "pytest", "--collect-only", "-q"], text=True)
    return [line.strip() for line in out.splitlines() if "::" in line]


def run(order: list[str]) -> int:
    return subprocess.call([sys.executable, "-m", "pytest", "-q", *order])


def campaign(rounds: int, seed: int) -> int:
    ids = nodeids()
    rng = random.Random(seed)
    leaks = 0
    for i in range(rounds):
        order = ids[:]
        rng.shuffle(order)
        code = run(order)
        if code != 0:
            print(f"LEAK round={i} seed={seed} exit={code}")
            leaks += 1
    return leaks

Keep rounds small and fixed in CI. The goal is classification, not exhaustion. A test that fails in isolation stays a functional bug. A test that fails only after another test ran is a leak. Fix the shared state. Do not write a skip.

Pair this with process isolation when the suite is cheap enough:

python -m pytest -q --forked   # only if pytest-forked is already a dependency

If you cannot fork, shuffle-repeat is the cheaper detector. It will miss leaks that require a specific pair of tests in a specific order. Document that bound. Do not pretend it is a proof.

Decision table

Observation	Classification	Merge action
Corpus hash changes, not in allowlist	Public behavior drift	Block
Corpus hash changes, allowlist row reviewed	Intentional contract change	Allow that case only
Test files added, unique predicates unchanged	Duplicate oracle	Block
Unique predicates up, unparsed > 0 on added lines	Opaque helper asserts	Block until parsed or rewritten
Failure in isolation	Functional regression	Block
Failure only after shuffle	Shared-state leak	Block; do not skip
All three gates pass	Candidate merge	Review the allowlist and the predicate list

Print the table in the CI log. Reviewers should not have to reconstruct the class from a red X.

Where a remote runner helps

Shuffle-repeat is sensitive to the machine. Laptop thermal throttling, dirty tmp, and a running Docker daemon all inject noise that looks like a flake. A dedicated runner removes that class of false leaks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a machine that is not your laptop, MonkeyCode's free server option is a place to park the shuffle-repeat loop, and free model access can draft candidate predicates from a corpus file. A reviewer still accepts or rejects each predicate. The model does not write the allowlist.

Treat model output as a diff, not as an oracle. Paste candidates into unique_predicates.py coverage only after a human has deleted tautologies and helper-only asserts.

Limitations

The corpus is only as honest as the surface you registered. Private helpers can rot while public hashes stay stable. That is acceptable if your merge rule is about callers. It is not acceptable if the agent was asked to refactor internals that have no recorded cases.

The AST visitor misses unittest helpers, custom assert_* wrappers, and property libraries that hide comparisons. Fail closed on unparsed. Do not invent a 100% parse rate.

Shuffle-repeat is probabilistic. Four rounds will not enumerate n! orders. Raise rounds only when the suite is fast. If a leak needs a rare pair, you will miss it until a later patch.

Hashing canonical JSON will break on unordered sets that serialize through default=str, on timestamps, and on object ids. Those types do not belong in a behavior corpus until you define a canonical form. Exclude them. Do not weaken sort_keys.

None of the three gates measure performance, accessibility, or security. Do not overload them.

Who should not use this

Skip the corpus gate if the public surface is a GUI, a model checkpoint, or a distributed consensus protocol. Those need different oracles.

Skip shuffle-repeat if tests already require a live network you do not control. Fix hermeticity first. A remote runner will not save a suite that calls production.

Skip unique-predicate counting if the language has no cheap AST and the team will not fund a parser. A weak visitor that you ignore is worse than no visitor.

Solo scripts and one-off migrations do not need this contract. Required checks have a cost. Spend it on trees that agents touch more than once.

What to log on every agent PR

Keep the log short. Reviewers read it once.

corpus_delta: 0
allowlist_rows_added: 0
unique_predicates_before: 41
unique_predicates_after: 47
unparsed_added_lines: 0
shuffle_rounds: 4
shuffle_leaks: 0

If corpus_delta is zero and unique predicates did not move, the patch did not earn a merge. If shuffle_leaks is nonzero, the next edit is the leak, not another test file.

The procedure is boring on purpose. Agent patches become reviewable when the public surface is hashed, the predicates are unique, and order is part of the oracle. Count those. Stop counting tests.