Intersect, Don't Union: Three Trust Classes for Agent Patch Tests

Stop merging because the whole suite is green. An agent that writes both the patch and the tests can add passing cases that never discriminate a fault. Merge only when three trust classes independently agree: human-owned oracles still fail known-bad fixtures, property checks pass on recorded seeds, and flaky tests are frozen out of the verdict.

That is an intersection, not a union. A union lets the largest, noisiest layer hide a silent oracle.

Why suite-green is the wrong aggregator

Agent patches usually land with extra tests. Those tests are not automatically more evidence. They are additional votes from a writer that already believes the patch is correct.

If you count every passing test equally, you reward volume. Agents are good at volume. They are weaker at naming an oracle they did not invent.

A practical gate therefore classifies tests first, then ANDs the class verdicts. Unknown files default to fail-closed. They may run, but they do not make the merge green.

Three trust classes

Treat the suite as an inventory, not a scoreboard.

1. oracle — human-owned examples and fixtures. The agent may not add files to this class in the same change that edits production code.
2. property — parameterized or seeded checks. Allowed from an agent only after a human accepts the invariant text and the seed policy.
3. frozen_flake — tests with unstable pass/fail history. They execute in quarantine, but their status is excluded from the merge bit.

Everything else is untrusted. Untrusted tests can still fail the job. They cannot pass it.

Artifact: a trust manifest and a fail-closed classifier

The following files are a proposal you can drop next to pytest. They are not a production SLA. Treat paths and markers as local convention.

# tests/trust_manifest.yaml
version: 1
classes:
  oracle:
    require_human_owner: true
    globs:
      - "tests/oracles/**/*.py"
    markers:
      - oracle
  property:
    require_accepted_invariant: true
    globs:
      - "tests/properties/**/*.py"
    markers:
      - property
  frozen_flake:
    freeze_file: "tests/flakes/freeze.yaml"
    globs:
      - "tests/**/*.py"
    markers:
      - flake
fail_closed_untrusted: true
oracle_must_fail_known_bad: true
known_bad_dir: "tests/oracles/known_bad"

# tests/flakes/freeze.yaml
# Proposal: expire freezes so they cannot hide a permanent race.
freezes:
  - id: "test_retry_backoff_under_load"
    reason: "timing depends on shared runner load"
    first_seen: "2026-09-01"
    expires: "2026-09-29"
    owner: "platform-tests"

Classifier (proposal, Python 3.11+). Run it before pytest, not after. Classification is a precondition. If the freeze file is stale, the gate stops.

# tools/classify_trust.py
from __future__ import annotations

import argparse
import fnmatch
import json
import sys
from datetime import date, datetime
from pathlib import Path

import yaml

ALLOWED = {"oracle", "property", "frozen_flake", "untrusted"}


def load_yaml(path: Path) -> dict:
    with path.open() as fh:
        return yaml.safe_load(fh)


def expired(expires: str, today: date) -> bool:
    return datetime.strptime(expires, "%Y-%m-%d").date() <= today


def classify_file(rel: str, manifest: dict, freeze_ids: set[str]) -> str:
    for cls in ("oracle", "property"):
        for glob in manifest["classes"][cls].get("globs", []):
            if fnmatch.fnmatch(rel, glob):
                return cls
    stem = Path(rel).stem
    if stem in freeze_ids or rel in freeze_ids:
        return "frozen_flake"
    return "untrusted"


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--manifest", default="tests/trust_manifest.yaml")
    parser.add_argument("--root", default=".")
    parser.add_argument("--today", default=date.today().isoformat())
    args = parser.parse_args()

    root = Path(args.root)
    manifest = load_yaml(root / args.manifest)
    freeze = load_yaml(root / manifest["classes"]["frozen_flake"]["freeze_file"])
    today = datetime.strptime(args.today, "%Y-%m-%d").date()

    freeze_ids: set[str] = set()
    expired_ids: list[str] = []
    for item in freeze.get("freezes", []):
        if expired(item["expires"], today):
            expired_ids.append(item["id"])
            continue
        freeze_ids.add(item["id"])

    inventory = []
    for path in (root / "tests").rglob("test_*.py"):
        rel = str(path.relative_to(root)).replace("\\", "/")
        inventory.append({"path": rel, "class": classify_file(rel, manifest, freeze_ids)})

    report = {
        "today": today.isoformat(),
        "expired_freezes": expired_ids,
        "counts": {k: sum(1 for row in inventory if row["class"] == k) for k in ALLOWED},
        "inventory": inventory,
    }
    print(json.dumps(report, indent=2))
    if expired_ids:
        print("expired freezes must be triaged before merge", file=sys.stderr)
        return 2
    if report["counts"]["oracle"] == 0:
        print("no oracle tests classified; fail closed", file=sys.stderr)
        return 3
    return 0


if __name__ == "__main__":
    sys.exit(main())

python tools/classify_trust.py --today 2026-09-15 > artifacts/trust_inventory.json

Numbered merge protocol

Work the layers in this order. Skipping a layer turns the AND back into a union.

1. Lock authorship. In the same pull request, reject new files under tests/oracles/ if production code also changed, unless a human owner is on the review. Agents propose oracle diffs in a follow-up change.
2. Prove oracles still bite. Apply each fixture in tests/oracles/known_bad/ to a clean tree and assert the oracle class fails. If those fixtures go green, the oracle class has been neutralized.
3. Accept invariants in text, not in review theater. A property test needs a one-line invariant in tests/properties/INVARIANTS.md. No invariant, no property class.
4. Record seeds. Property runs write artifacts/property_seeds.json. The merge artifact is the seed list plus pass/fail, not a screenshot of a green check.
5. Quarantine flakes by identity. Frozen tests run with -m flake on an isolated pytest node. Their exit code is logged and ignored for the merge bit, until the freeze expires.
6. AND the three bits. oracle_ok AND property_ok AND freeze_file_valid. Untrusted failures still fail the job. Untrusted passes do not satisfy property_ok or oracle_ok.

# tools/intersection_gate.sh
set -euo pipefail
mkdir -p artifacts
python tools/classify_trust.py --today "${TODAY:-2026-09-15}" > artifacts/trust_inventory.json

pytest tests/oracles -m oracle --maxfail=1 --junitxml artifacts/oracle.xml
python tools/replay_known_bad.py --dir tests/oracles/known_bad --pytest-args "-m oracle"

pytest tests/properties -m property --junitxml artifacts/property.xml
python tools/record_seeds.py --from-properties tests/properties --out artifacts/property_seeds.json

set +e
pytest -m flake --junitxml artifacts/flake.xml
set -e

python tools/and_verdicts.py \
  --inventory artifacts/trust_inventory.json \
  --oracle-junit artifacts/oracle.xml \
  --property-junit artifacts/property.xml \
  --property-seeds artifacts/property_seeds.json \
  --flake-junit artifacts/flake.xml

replay_known_bad.py is a local helper. The contract is small: each known-bad fixture must make the oracle class fail, and an empty seed file is not a pass.

# tools/replay_known_bad.py
"""Proposal: apply each known-bad fixture and require oracle tests to fail."""
from __future__ import annotations

import argparse
import subprocess
import sys
from pathlib import Path


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--dir", default="tests/oracles/known_bad")
    parser.add_argument("--pytest-args", default="-m oracle")
    args = parser.parse_args()

    fixtures = sorted(Path(args.dir).glob("*.patch"))
    if not fixtures:
        print("no known-bad fixtures; fail closed", file=sys.stderr)
        return 3

    failures = 0
    for patch in fixtures:
        subprocess.check_call(["git", "stash", "push", "-u", "-m", "known-bad-replay"])
        try:
            subprocess.check_call(["git", "apply", str(patch)])
            result = subprocess.run(["pytest", *args.pytest_args.split()], check=False)
            if result.returncode == 0:
                print(f"oracle stayed green under {patch.name}", file=sys.stderr)
                failures += 1
        finally:
            subprocess.call(["git", "reset", "--hard"])
            subprocess.call(["git", "stash", "pop"])
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Label the git apply path as a lab workflow. Do not point it at a dirty tree you cannot restore. The point is the expect-fail contract, not the VCS details.

Decision table

Oracle class	Property class	Freeze file	Untrusted tests	Merge
pass, known-bad still fail	pass, seeds recorded	valid, none expired	pass	allow
pass, known-bad still fail	pass, seeds recorded	valid	fail	block
pass, but known-bad now pass	any	any	any	block (oracle neutralized)
pass	no seeds / empty class	valid	pass	block (fail closed)
fail	pass	valid	pass	block
pass	pass	expired freeze	pass	block (triage freeze)
no oracle files	pass	valid	pass	block

Untrusted failures are fail-closed by design. They are noise that still has to be inspected. They are not votes you can ignore.

Property checks without letting the agent own the invariant

Keep the invariant in a file a classifier can grep. Keep the seed list explicit so a rerun is deterministic.

# tests/properties/INVARIANTS.md
- applying the patch twice to the same fixture byte-equals one apply

# tests/properties/test_patch_preserves_idempotency.py
import pytest

pytestmark = pytest.mark.property

# INVARIANT: applying the patch twice to the same fixture byte-equals one apply.
SEEDS = [0, 1, 7, 13, 99]


@pytest.mark.parametrize("seed", SEEDS)
def test_double_apply_equals_single_apply(seed, workspace_factory):
    ws = workspace_factory(seed=seed)
    once = ws.apply_patch()
    twice = ws.apply_patch(ws.apply_patch())
    assert once.digest == twice.digest

If an agent adds assert True or an empty parametrize list, step 4 fails. Empty seeds are not a pass. A property file without a matching line in INVARIANTS.md stays untrusted even if the glob matches.

# tools/record_seeds.py
from __future__ import annotations

import ast
import json
from pathlib import Path


def seeds_in(path: Path) -> list[int]:
    tree = ast.parse(path.read_text())
    for node in tree.body:
        if isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name) and target.id == "SEEDS":
                    return ast.literal_eval(node.value)
    return []


def main() -> None:
    root = Path("tests/properties")
    payload = {
        path.as_posix(): seeds_in(path)
        for path in sorted(root.glob("test_*.py"))
    }
    if any(len(v) == 0 for v in payload.values()) or not payload:
        raise SystemExit("empty property seeds; fail closed")
    Path("artifacts/property_seeds.json").write_text(json.dumps(payload, indent=2))


if __name__ == "__main__":
    main()

Fixture locks the agent cannot silently rewrite

Store oracle fixtures as content-addressed files. Hash them in CI. A patch that “fixes tests” by rewriting expected bytes should show up as a lockfile diff, not as a quieter suite.

# tests/oracles/test_fixture_lock.py
import hashlib
from pathlib import Path

import pytest

pytestmark = pytest.mark.oracle
LOCK = Path("tests/oracles/locks.sha256")


def test_fixture_bytes_match_lock():
    recorded = {
        line.split()[1]: line.split()[0]
        for line in LOCK.read_text().splitlines()
        if line and not line.startswith("#")
    }
    root = Path("tests/oracles/fixtures")
    current = {
        str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest()
        for p in sorted(root.rglob("*"))
        if p.is_file()
    }
    assert current == recorded

This is a lock, not a snapshot test the agent is invited to regenerate in the same change. Regenerating locks.sha256 belongs in a follow-up with a human owner, the same rule as new oracle files.

Where a free model endpoint and a free server fit

Generating candidate invariants is cheap. Accepting them is not. A free model endpoint is useful as a proposer: it drafts invariant sentences and known-bad sketches. A human still promotes those drafts into tests/oracles/ or INVARIANTS.md.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option can sit in that split. The model proposes; the free server runs intersection_gate.sh so the AND of trust classes is not computed only on a laptop. Neither layer replaces the oracle class. If the remote job cannot classify files, fail closed.

Do not treat a green remote run of untrusted tests as merge evidence. That is a union in disguise.

Limitations

The classifier only knows globs, markers, and a freeze file. It cannot detect a tautology that lives inside a well-named oracle file. Humans still read invariants.

Known-bad fixtures rot. If nobody adds a bad case for a new subsystem, step 2 becomes theater. Refresh the directory when a production path grows, not when the suite feels noisy.

Freeze files hide flakes and also hide races that only appear under merge traffic. Expiry is mandatory. Infinite freezes are skipped tests with better stationery.

Seed logs are merge artifacts, not proofs of functional correctness. They prove you ran the properties you claimed, with the seeds you claimed.

This protocol assumes pytest-style collection. Other runners need the same algebra, not these exact flags. replay_known_bad.py as written is unsafe on a dirty worktree; isolate it.

Who should not use this

Do not use intersection gating if you have no human-owned oracle directory. You would AND against an empty set and either merge nothing or cheat by promoting agent tests into oracle.

Do not use it on a safety-critical path as the only review. Trust classes are an admission-control filter, not a hazard analysis.

Do not use it if your team will not expire freezes. A freeze file without owners and dates is a skip list.

Skip the remote proposer if your patches include secrets or proprietary fixtures you cannot send to a hosted model. Run classification and pytest locally, and keep oracles off the wire.

What to implement first

Start with the manifest, the classifier, and the known-bad replay. Add property seeds second. Add freeze expiry last, because it is the easiest layer to game.

The merge question is not “did the suite pass?” It is “did every trusted class pass, and did the untrusted class refuse to fail-open?” If you cannot answer from trust_inventory.json, the gate is not done. Point the classifier at your existing pytest markers before standing up another runner.