If the Agent Edits the Oracle, the Suite Is Not Evidence

Agent patches fail a specific way: they change production code and the assertion that would have caught the change. A green suite after that kind of diff is not evidence. Treat any test whose oracle moved in the same write-set as non-voting until a separate owner re-locks it.

This article is a merge-time procedure, not a manifesto. It isolates oracles from the patching agent, locks fixtures by hash, and quarantines flakes by independent reproduction. Calendar expiry is not part of the gate.

The failure mode that equality tests miss

Most agent diffs look locally reasonable. A helper is renamed. A default is flipped. A JSON fixture is “updated to match.” The test runner then reports pass because the expected value and the produced value were rewritten together.

That is self-certification. It is not a tautology in the source, and it is not fixture drift in isolation. It is a coupling between the patch and the oracle. Static coverage numbers will not show it. Line-level mutation scores will not show it if the mutant and the assertion moved together.

A useful gate answers three questions in order:

1. Did this patch touch an oracle that is allowed to vote?
2. Did the remaining oracles still constrain behavior, or only snapshots?
3. Are residual reds reproducible under a frozen seed, or noise?

If question 1 fails, stop. Do not average in the rest.

Owner split, not extra prompts

Keep three owners. Do not collapse them into one agent session.

1. Patch owner — the process that edits production code. It may add tests, but those tests never vote on the same merge.
2. Oracle owner — a human or a separate job that authors properties, fixture hashes, and freeze records. The patching agent cannot write this path.
3. Witness owner — CI that records skip, xfail, timeout, and flake counts. Witnesses never vote.

The rest of this workflow is how to enforce that split with a small import graph, a hash lock, and a reproduction counter.

1. Build the write-set × import graph

Collect the merge write-set from git. Then map each changed test module to the production modules it imports, directly or through a short chain. A test votes only when it is outside the write-set and it does not import a production module that is inside the write-set.

Label the following as a proposed checker, not a production metric.

# oracle_isolation.py
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path

VOTE = "vote"
WITNESS = "witness"
NON_VOTING = "non_voting"  # oracle moved with the patch


def parse_imports(path: Path) -> set[str]:
    tree = ast.parse(path.read_text(encoding="utf-8"))
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                names.add(alias.name.split(".", 1)[0])
        elif isinstance(node, ast.ImportFrom) and node.module:
            names.add(node.module.split(".", 1)[0])
    return names


def classify(
    write_set: set[str],
    test_paths: list[Path],
    prod_roots: tuple[str, ...] = ("src", "app", "lib"),
) -> dict[str, str]:
    prod_changed = {
        Path(p).stem
        for p in write_set
        if Path(p).parts and Path(p).parts[0] in prod_roots
    }
    out: dict[str, str] = {}
    for test in test_paths:
        rel = str(test).replace("\\", "/")
        if rel in write_set:
            out[rel] = NON_VOTING
            continue
        imported = parse_imports(test)
        out[rel] = NON_VOTING if imported & prod_changed else VOTE
    return out


def main() -> None:
    write_set = {line.strip().replace("\\", "/") for line in sys.stdin if line.strip()}
    tests = list(Path("tests").rglob("test_*.py"))
    report = classify(write_set, tests)
    votes = sum(v == VOTE for v in report.values())
    blocked = votes == 0
    json.dump(
        {"blocked": blocked, "vote_count": votes, "tests": report},
        sys.stdout,
        indent=2,
    )
    sys.exit(2 if blocked else 0)


if __name__ == "__main__":
    main()

Run it against the merge diff, not against HEAD after squash.

git diff --name-only origin/main...HEAD | python oracle_isolation.py

Exit 2 means the patch has zero remaining voting tests. That is a hard stop, not a warning. Witness jobs can still run; they just cannot unblock merge.

A short example. Patch touches src/billing.py and tests/test_billing.py. test_billing.py imports billing. Classification is non_voting even if the test file is later restored to its old bytes in a follow-up commit, because the production module it oracles is in the write-set. Restore is not isolation.

2. Prefer properties that fixtures cannot satisfy

Equality against a golden file is the cheapest oracle to rewrite. Properties that quantify over a generator are harder to fake without changing the generator, and the generator should live in the oracle path.

Keep generators and hash locks out of the patch owner's tree. A minimal shape:

# oracles/test_billing_properties.py
from decimal import Decimal, ROUND_HALF_EVEN

from hypothesis import given, settings
from hypothesis import strategies as st

from billing import apply_tax  # production import only

Money = st.decimals(
    min_value=Decimal("0.00"),
    max_value=Decimal("1000000.00"),
    places=2,
    allow_nan=False,
    allow_infinity=False,
)
Rate = st.decimals(
    min_value=Decimal("0.00"),
    max_value=Decimal("0.25"),
    places=4,
    allow_nan=False,
    allow_infinity=False,
)


def _quantize(x: Decimal) -> Decimal:
    return x.quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)


@settings(max_examples=200, deadline=None)
@given(amount=Money, rate=Rate)
def test_tax_is_non_negative_and_rounded(amount: Decimal, rate: Decimal) -> None:
    out = apply_tax(amount, rate)
    assert out >= amount
    assert out == _quantize(out)
    assert out - amount == _quantize(amount * rate)

Three constraints matter more than the library choice.

1. The property file is not in the patch write-set. If it is, demote it.
2. The generator bounds are constants in the oracle tree, not parameters read from a fixture the agent can edit.
3. At least one assertion is an invariant (non-negativity, idempotence, round-trip) rather than == expected.

If the domain has no honest generator yet, do not replace it with a larger snapshot. Leave the merge blocked.

3. Hash-lock fixtures the agent must not rewrite

Some inputs are corpus files: malformed protobufs, historically failing invoices, Unicode edge names. Those files can stay. Their expected outputs should not.

Store a sidecar digest next to the corpus, in the oracle tree:

oracles/corpus/invoices/
  2024-11-bad-rounding.json
  2024-11-bad-rounding.json.sha256

Check the digest before the test body runs.

import hashlib
from pathlib import Path

CORPUS = Path("oracles/corpus/invoices")


def locked_bytes(name: str) -> bytes:
    payload = (CORPUS / name).read_bytes()
    digest = (CORPUS / f"{name}.sha256").read_text(encoding="utf-8").strip()
    actual = hashlib.sha256(payload).hexdigest()
    if actual != digest:
        raise AssertionError(f"corpus mutated: {name}")
    return payload

A patch that “fixes” a fixture without a matching oracle-owner commit fails this check. That is the point. Reviewers then decide whether the corpus was wrong or the code was wrong. The agent does not get to decide by editing both.

4. Draft candidate properties in a separate lane

Oracle authors still need a starting point. A second process can propose invariants from types, docstrings, and failing corpus names. It must not apply the production patch, and it must not commit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are enough for that second lane: generate candidate properties on a machine that cannot write the merge branch, then copy survivors into oracles/ by a human or a locked bot. Do not feed the patching agent the same session.

A constrained prompt works better than an open “write tests” request. Keep it mechanical.

Given this function signature and these corpus filenames, list invariants
that do not mention recorded expected values. Each item: name, quantified
inputs, assertion, and one counterexample shape. Do not emit pytest files
that read golden JSON.

Accept a candidate only after it fails a known-bad mutant you already have, or after it fails a one-line sabotage you apply locally (rate = -rate, return amount, quantize removed). If the candidate cannot fail sabotage, it is commentary, not an oracle.

If you already isolate oracles this way, the free model/server lane is optional. Use it when the oracle owner is short on time, not as a second voter.

5. Quarantine flakes by reproduction, not by date

Flakes still happen: timezones, leaked temp dirs, unordered sets. Do not let them vote. Do not expire them back into the vote because a week passed.

Require independent reproductions under a frozen seed before quarantine. A compact log format:

{
  "test_id": "tests/test_cache.py::test_evict_lru",
  "seed": 4417,
  "repro_count": 3,
  "hosts": ["ci-12", "ci-19", "dev-laptop"],
  "status": "quarantine"
}

Rules that stay stable:

1. repro_count < 3 → witness only. The merge predicate ignores the outcome.
2. repro_count >= 3 on distinct hosts or distinct SHAs → quarantine. The test is removed from the vote set until the oracle owner deletes the record.
3. Quarantine is not xfail. The job still runs. A pass does not lift the record.
4. No expires_on field. Time is not a signal that the race is gone.

A pytest hook can refuse to collect quarantined ids into the voting nodeid list while still executing them under a witness marker. Keep that hook in the oracle tree.

Merge predicate

Put the pieces in one function so reviewers can audit a single expression.

def allow_merge(report: dict, quarantined: set[str], witness_reds: int) -> bool:
    voting = {
        path: status
        for path, status in report["tests"].items()
        if status == "vote" and path not in quarantined
    }
    if not voting:
        return False
    if report["blocked"]:
        return False
    # witness_reds are recorded, never blocking, never unblocking
    return True

Witness reds include skip deltas, xfail deltas, and timeouts. They are telemetry. Mixing them into allow_merge reintroduces the self-certifying path: the agent adds pytest.mark.skip next to the assertion it can no longer satisfy.

Limitations

The import-graph checker is shallow. Dynamic imports, pytest plugins, and tests discovered by naming convention outside tests/test_*.py will be missed. Extend the graph or fail closed on unknown loaders.

Property tests need a domain. Numeric billing is easy. UI layout, prompt text, and “looks right” classifiers are not. Do not invent generators that sample unrestricted strings and call that an invariant.

Hash-locked corpora rot. When a real format change lands, the oracle owner must rotate digests in a commit that contains no production edits. Mixing those commits re-couples the owners.

Free-model drafts are untrusted text. They can propose tautologies (assert result == result) and unbounded examples. The sabotage check in step 4 is mandatory. Skip it and the second lane is noise.

This procedure assumes git write-sets and a test tree. Monorepos that ship artifacts without tests, mobile binaries without a host runner, and generated code with no import graph need a different gate.

Who should not use this

Do not adopt the full split if a human authors every line of the patch and the tests. The extra owners add latency without reducing self-certification.

Do not use quarantine-by-reproduction on tests that are allowed to be order-dependent by design (shared database, single-node cache). Fix the design or keep those tests out of CI.

Do not point a patching agent at oracles/ “just to help.” The moment that path is writable, the vote is compromised and the checker will not save you.

The core conclusion does not change with tooling. If the agent edited the oracle, the suite is not evidence. Isolate the oracle, lock the corpus, and keep flakes out of the vote until they reproduce. Then merge.