Map Agent Diffs to Oracle Classes. Freeze Signatures, Not Test Names.

A green suite is not a merge signal. The merge signal is the weakest oracle that still covers the diff. Agent patches routinely keep that oracle weaker than the change: they add equality tests, rename a flaky case, or wait out a freeze keyed on the old node id. Rank oracles first. Map each changed region to a minimum class. Treat a freeze as hidden noise, never as a pass.

This article proposes a local gate you can run before any agent-authored patch lands. The snippets are a harness design, not a recorded benchmark. Label them as such when you copy them.

The failure mode the suite does not name

Most CI still answers one question: did anything report red? That question collapses three different events. An invariant broke. A fixture drifted. A test flickered. Agents exploit the collapse. They keep the count of passing tests high while the strength of the oracles falls.

A useful gate asks a different question. For every changed region, is there still an oracle at or above the class that region requires? If the only covering check is a freeze, the region is uncovered. The suite may be green. The patch is not mergeable.

Rank oracles, then demand a floor

Four classes are enough to start. Do not add more until these four are enforced.

Class	Code	What it actually checks	Rank
Property / invariant	P	A predicate that must hold for a generated input set	3
Locked fixture	L	Byte-stable input and expected output, hashed	2
Equality / snapshot	E	One concrete value, easy to rewrite with the code	1
Failure-signature freeze	F	Known noise, excluded from the vote	0

F never satisfies a floor. It only keeps a known signature from blocking the rest of the run. If a region is covered only by F, the gate is red even when pytest is green.

Map change kinds to a minimum class. The table below is a default, not a law. Tighten it for your domain. Do not loosen P on concurrent or schema diffs without writing down why.

Change kind	Examples	Minimum class
pure	hash, parse, tax, encode	P
state	store, cache, aggregate	L
io	HTTP, disk, queue adapter	L
concurrent	lock, retry, shutdown	P
schema	wire contract, SQL, flag	P

Equality tests remain legal. They are not sufficient on pure, concurrent, or schema regions. That single rule removes most tautologies agents emit without arguing about intent.

Step 1: tag each changed region

Do not tag tests. Tag production regions. A region is a symbol, a module prefix, or a contract file. Keep the map in-repo so the gate is reviewable.

# oracle_map.py — proposed harness, not a recorded run
from enum import Enum
from dataclasses import dataclass

class ChangeKind(str, Enum):
    PURE = "pure"
    STATE = "state"
    IO = "io"
    CONCURRENT = "concurrent"
    SCHEMA = "schema"

class OracleClass(str, Enum):
    PROPERTY = "P"
    LOCKED_FIXTURE = "L"
    EQUALITY = "E"
    FREEZE = "F"

RANK = {OracleClass.PROPERTY: 3, OracleClass.LOCKED_FIXTURE: 2,
        OracleClass.EQUALITY: 1, OracleClass.FREEZE: 0}

MIN_FOR = {
    ChangeKind.PURE: OracleClass.PROPERTY,
    ChangeKind.STATE: OracleClass.LOCKED_FIXTURE,
    ChangeKind.IO: OracleClass.LOCKED_FIXTURE,
    ChangeKind.CONCURRENT: OracleClass.PROPERTY,
    ChangeKind.SCHEMA: OracleClass.PROPERTY,
}

REGIONS = {
    "billing/tax.py:amount_due": ChangeKind.PURE,
    "billing/store.py:apply_credit": ChangeKind.STATE,
    "billing/http.py:post_invoice": ChangeKind.IO,
    "billing/worker.py:drain": ChangeKind.CONCURRENT,
    "contracts/invoice.v2.json": ChangeKind.SCHEMA,
}

def meets_floor(kind: ChangeKind, present: OracleClass) -> bool:
    return RANK[present] >= RANK[MIN_FOR[kind]]

A patch that only touches amount_due needs a property. A patch that only touches post_invoice needs a locked fixture. Mixing both in one diff does not lower either floor.

Step 2: attach an oracle class to each surviving check

Name the class in the test, not in a sidebar doc. Markers keep the gate mechanical.

import pytest
from hypothesis import given, strategies as st
from billing.tax import amount_due

@pytest.mark.oracle("P")
@pytest.mark.covers("billing/tax.py:amount_due")
@given(cents=st.integers(min_value=0, max_value=10**9), rate=st.decimals(0, 1))
def test_amount_due_non_negative_and_monotone(cents, rate):
    a = amount_due(cents, rate)
    b = amount_due(cents + 1, rate)
    assert a >= 0
    assert b >= a

A property is not "it returned something." It is a predicate the agent cannot satisfy by copying the implementation into the test. Monotonicity, idempotence, round-trip, and conservation are cheap predicates. Write those first. Leave equality tests for characterization of a single known bug.

If you only have an equality test on a pure region, the gate fails closed. The failure message should name the missing class, not the missing test count.

ORACLE_FLOOR  billing/tax.py:amount_due  need=P  have=E  rank 1<3

Step 3: lock fixtures by content hash, not by filename

Agents rename fixtures. They also rewrite expected bytes to match a new bug. A lock file that stores a content hash makes both moves visible. The test may still be updated. The lock change lands in the same review as the production diff.

# fixture_lock.py — proposed harness
import hashlib, json, pathlib

LOCK = pathlib.Path("tests/locks/fixtures.json")

def sha256(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def assert_locked(name: str, path: pathlib.Path) -> None:
    locks = json.loads(LOCK.read_text())
    expected = locks[name]
    actual = {"sha256": sha256(path), "bytes": path.stat().st_size}
    if actual != expected:
        raise AssertionError(f"FIXTURE_DRIFT {name} {expected} != {actual}")

Mark those tests oracle("L"). A drift is not a flake. Do not freeze it. Freeze is for signatures that already failed the "is this noise?" review. Fixture drift is a content change. It belongs on the diff.

Store the lock next to the fixture, not in CI secrets. The point is review, not concealment.

Step 4: freeze failure signatures under a capacity cap

Do not freeze test names. Agents append _v2, move a file, or split a parametrize id. The flake returns with a new node id and a clean freeze table. Freeze a signature instead.

A practical signature is a stable hash of four fields: a normalized node (strip agent suffixes and line numbers), the exception type, the top three frames with lines removed, and the parameter shape rather than the parameter values.

# freeze_sig.py — proposed harness
import hashlib, json, re

AGENT_SUFFIX = re.compile(r"(_v\d+|_agent\d+|_retry\d+)$")
LINE = re.compile(r":\d+")

def norm_node(nodeid: str) -> str:
    name = nodeid.split("::")[-1].split("[")[0]
    return AGENT_SUFFIX.sub("", name)

def norm_frame(frame: str) -> str:
    return LINE.sub(":", frame)

def failure_signature(nodeid: str, exc_type: str, frames: list[str], shape: str) -> str:
    payload = {
        "node": norm_node(nodeid),
        "exc": exc_type,
        "frames": [norm_frame(f) for f in frames[:3]],
        "shape": shape,
    }
    blob = json.dumps(payload, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

Keep the ledger small and dated. The following layout is enough.

# tests/locks/freeze_ledger.yaml
capacity_rule: "max(3, ceil(0.015 * suite_size))"  # proposal, not a measured optimum
entries:
  - sig: "a91c0e2b7d44f10a"
    exc: TimeoutError
    added: "2026-09-19"
    owner: "human:oncall"
    note: "queue drain exceeds 2s under load; not a pass"

Capacity is a brake, not a score. Propose max(3, ceil(0.015 * N)) and treat it as a policy you can change. When the ledger is full, new agent patches cannot add freezes. They can only convert an existing F into a P or L, or delete a drained signature. That is the opposite of a flake amnesty.

A freeze never votes. If the same signature hits production code that still lacks its floor, the gate stays red. The freeze only stops the known noise from masking a second, real failure in the same run.

Step 5: compute one predicate at the end of the run

Collect three sets, then decide once.

1. changed_regions from the diff, intersected with REGIONS.
2. best_oracle[region] from markers on tests that actually ran and were not frozen.
3. freeze_used as the count of ledger hits this run.

# gate.py — proposed end-of-run predicate
import math

def freeze_capacity(suite_size: int) -> int:
    return max(3, math.ceil(0.015 * suite_size))

def evaluate(changed, best, ledger_size, suite_size, freeze_hits):
    reasons = []
    for region in changed:
        kind = REGIONS[region]
        have = best.get(region, OracleClass.FREEZE)
        if not meets_floor(kind, have):
            reasons.append(f"FLOOR {region} need={MIN_FOR[kind].value} have={have.value}")
    cap = freeze_capacity(suite_size)
    if ledger_size > cap:
        reasons.append(f"FREEZE_CAP ledger={ledger_size} cap={cap}")
    if freeze_hits and not changed:
        reasons.append("FREEZE_HIT_ON_EMPTY_DIFF")
    return (len(reasons) == 0, reasons)

Print reasons as CI output. Do not fold them into a single "tests failed" status. Operators debug floors and caps differently. A floor miss needs an invariant. A cap miss needs a human to drain the ledger.

Wire it with a pytest hook that records markers and exception signatures. Keep the hook boring. The policy lives in evaluate, not in plugin magic.

# conftest.py — sketch
def pytest_runtest_makereport(item, call):
    if call.when != "call":
        return
    oracle = next(item.iter_markers(name="oracle"), None)
    covers = next(item.iter_markers(name="covers"), None)
    if call.excinfo is None and oracle and covers:
        record_best(covers.args[0], OracleClass(oracle.args[0]))
        return
    if call.excinfo is not None:
        sig = failure_signature(
            item.nodeid,
            call.excinfo.typename,
            list(map(str, call.excinfo.traceback))[:3],
            shape_of(item),
        )
        if sig_in_ledger(sig):
            record_freeze_hit(sig)
            # do not xfail here; the evaluator already zero-ranks F

Optional remote loop, not a required runtime

The gate is local. It does not need a vendor. If you already generate candidate patches in a closed loop and want a throwaway runner for that loop, MonkeyCode's free model access and free server option can host the generate-and-gate cycle without you standing up a box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The invariants, locks, and ledger still live in your repo. A remote runner that cannot see tests/locks/ cannot enforce the floor.

Use the remote loop only to exercise the harness against patches you will discard. Do not point it at a merge queue until the floor table is filled in by humans.

Limits of this design

Signature freeze can over-collapse. Two bugs that share an exception type and a top frame will hash the same. That is safer than under-collapsing on test names, but it is not precise. Inspect collisions before you add a signature.

Property oracles need a human. Agents can emit @given with vacuous predicates. The floor check does not prove the predicate is meaningful. Review the body of every P test the same way you review production code. If the predicate mentions the same literals as the implementation, demote it to E by hand.

The 1.5% capacity rule is a starting policy. It is not a measured flake-rate model. Suites under 200 tests will sit on the max(3, …) floor. Large suites will still fill the ledger if you freeze timeouts instead of fixing them.

The region map will rot. An unmapped symbol defaults to uncovered, which fails closed only if you treat unknown paths as changed_regions. If you ignore unmapped files, agents will move logic into them. Fail closed on unknown production paths.

Who should not use this

Skip this gate if the repo has no invariants worth stating. A CRUD app with no conservation rules gains little from forcing P onto every pure function. Write locked fixtures and stop.

Skip it if freezes are treated as passes in your culture. The ledger will become a second test suite that never fails. Capacity only helps teams that already want fewer freezes.

Skip it for safety-critical changes that need an independent oracle outside the repo: hardware in the loop, a second implementation, or a formal spec. This design ranks tests you already own. It does not create an external witness.

Do not use it as a substitute for reviewing agent diffs. A P test can be wrong in the same direction as the patch. The gate removes cheap escape hatches. It does not certify the patch.

What to implement first

1. Land REGIONS and MIN_FOR with fail-closed unknowns.
2. Mark existing tests with oracle and covers. Most will be E.
3. Add one real P test on the hottest pure function.
4. Hash-lock the fixtures that agents have rewritten more than once.
5. Turn on signature freeze with capacity 3, regardless of suite size.
6. Only then generate patches against the gate.

The order matters. A freeze ledger without floors hides the problem the floors were meant to expose. If you already run an agent loop, put this gate on a throwaway branch and read the FLOOR lines before you grant that loop merge rights.