Mutate the Oracle: A Property-Kill Matrix for Agent Patches

A green agent patch is not evidence. It is a missing counterexample. The remaining signal is an oracle the agent cannot edit, plus a proof that the oracle can still fail.

That proof is not mutation testing of the product. It is mutation testing of the assertions you own. Hash the fixtures the patch must not rewrite. Freeze flakes by failure identity, not by skipping a file. The rest of this article is a proposed, reproducible contract for that split.

The failure mode this contract targets

When an agent authors production code and tests in the same diff, CI green becomes correlated with the patch. Coverage numbers move with the same commit. A tautological assertion, a rewritten golden file, and a skipped flake all produce the same bit: pass.

Independent properties restore a fail path. They do nothing if they cannot be killed. An assertion that still passes after you delete it was never load-bearing. Treat that as a gate, not as a style comment.

This is adjacent to, and distinct from, classifying the patch itself. Here the object under test is the human-owned oracle.

Three files the agent must not own

Keep the contract in a directory the patch workflow cannot write. A proposed layout:

oracle/
  properties.py          # assertions the agent cannot edit
  fixtures.lock.json     # content-addressed fixture digests
  flake_ledger.jsonl     # freeze by failure identity, with expiry
  kill_matrix.md         # last property-kill run (committed)

The production tree and any agent-authored tests/ tree stay writable. The oracle tree is read-only in the merge lane. If your agent runner can git add oracle/, the contract is theater.

Step 1 — Write properties that name a counterexample

A property is load-bearing only if a concrete input can falsify it. Label the following as a proposed harness, not as a measured production suite.

# oracle/properties.py
from __future__ import annotations

from typing import Callable, Sequence


def prop_sort_is_permutation(sort_fn: Callable[[list[int]], list[int]], xs: list[int]) -> None:
    out = sort_fn(list(xs))
    assert sorted(out) == sorted(xs), "output is not a permutation of input"
    assert out == sorted(xs), "output is not sorted ascending"


def prop_roundtrip(encode, decode, blob: bytes) -> None:
    assert decode(encode(blob)) == blob, "round-trip lost bytes"


def prop_reject_empty(parse: Callable[[str], object]) -> None:
    raised = False
    try:
        parse("")
    except ValueError:
        raised = True
    assert raised, "empty input must raise ValueError"

Each property has one job. Mix them and you cannot tell which assertion died. Keep the input generators in the same sealed module, or the agent can shrink the domain until every case is trivial.

# oracle/domain.py
import string

INT_CASES: list[list[int]] = [
    [],
    [0],
    [2, 1, 2],
    list(range(32, -1, -1)),
]

BLOB_CASES: list[bytes] = [
    b"",
    b"\x00",
    bytes(range(256)),
    string.ascii_letters.encode() * 17,
]

Step 2 — Kill each assertion before you trust a green patch

Property-kill is a mechanical pass over oracle/properties.py. For each assert, emit a mutant that comments that line out, run the suite against a known-good implementation and against the agent patch, and record whether the mutant still fails on a deliberate bug.

Proposed command surface:

python -m oracle.kill --source oracle/properties.py --out oracle/kill_matrix.md
python -m oracle.kill --source oracle/properties.py --impl path/to/agent_patch

A minimal killer (proposed, unexecuted here) can operate at the AST level so comments and string literals are not treated as assertions:

# oracle/kill.py  (proposed)
from __future__ import annotations

import ast
import copy
import textwrap
from dataclasses import dataclass

@dataclass(frozen=True)
class KillRow:
    prop: str
    assertion: str
    mutant_fails_on_seed_bug: bool
    mutant_fails_on_patch: bool

    @property
    def verdict(self) -> str:
        if not self.mutant_fails_on_seed_bug:
            return "DEAD_ORACLE"
        if not self.mutant_fails_on_patch:
            return "PATCH_UNTESTED_BY_THIS_LINE"
        return "LOAD_BEARING"


class AssertEnumerator(ast.NodeVisitor):
    def __init__(self) -> None:
        self.hits: list[tuple[str, ast.Assert]] = []
        self._fn = "<module>"

    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
        prev, self._fn = self._fn, node.name
        self.generic_visit(node)
        self._fn = prev

    def visit_Assert(self, node: ast.Assert) -> None:
        self.hits.append((self._fn, node))

Record one row per assertion. Do not average them. A suite with nine dead lines and one live line is not "90% fine." It is one constraint wearing a crowd.

Property-kill matrix (artifact)

Commit the last run next to the oracle. The table is the artifact reviewers can grep.

| prop                  | assertion (truncated)              | seed-bug mutant fails | patch mutant fails | verdict                    |
|-----------------------|------------------------------------|-----------------------|--------------------|----------------------------|
| prop_sort_is_permutation | output is not a permutation     | yes                   | yes                | LOAD_BEARING               |
| prop_sort_is_permutation | output is not sorted ascending  | yes                   | no                 | PATCH_UNTESTED_BY_THIS_LINE|
| prop_roundtrip        | round-trip lost bytes              | no                    | no                 | DEAD_ORACLE                |
| prop_reject_empty     | empty input must raise ValueError  | yes                   | yes                | LOAD_BEARING               |

Read the matrix left to right. DEAD_ORACLE means your sealed suite cannot fail even when you sabotage it; fix the property before blaming the agent. PATCH_UNTESTED_BY_THIS_LINE means the patch does not exercise that assertion; either the domain is too narrow or the patch bypassed the path. LOAD_BEARING is the only row that can support a merge decision.

A proposed merge rule: zero DEAD_ORACLE rows, and every behavioral claim in the ticket mapped to at least one LOAD_BEARING row. Unmapped claims are undocumented wishes.

Step 3 — Lock fixtures by digest, not by path

Path locks fail when the agent adds fixture_v2.json and points the test at it. Content-address the bytes. The filename is a label. The digest is the contract.

# oracle/fixtures.py  (proposed)
from __future__ import annotations

import hashlib
import json
from pathlib import Path

LOCK = Path("oracle/fixtures.lock.json")


def digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def verify_lock(root: Path) -> list[str]:
    expected = json.loads(LOCK.read_text())
    errors: list[str] = []
    for rel, sha in expected.items():
        p = root / rel
        if not p.is_file():
            errors.append(f"missing {rel}")
            continue
        got = digest(p)
        if got != sha:
            errors.append(f"digest mismatch {rel}: {got} != {sha}")
    return errors

Example lock file:

{
  "fixtures/orders.ndjson": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "fixtures/reject_empty.txt": "d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592"
}

Two review rules follow. New fixture paths require a human lock-file edit. Digest changes require a one-line reason in the PR body (format, schema, or bug-in-fixture). An agent patch that only updates the lock is a fixture rewrite, not a product fix. Reject it on that classification, independent of test green.

Step 4 — Freeze flakes by identity, with an expiry

Skipping test_flaky.py deletes a sensor. Freeze the failure identity instead: test id, normalized assertion message, and a cause class you are willing to defend.

{
  "test_id": "tests/test_parse.py::test_timeout_window",
  "signature": "sha256:9c1c... of file:line + assertion msg without digits",
  "cause_class": "network-clock",
  "first_seen": "2026-09-16",
  "expires": "2026-09-30",
  "ticket": "QA-4412"
}

Normalize before hashing. Strip timestamps, PIDs, and absolute paths or every run mints a new identity and the ledger never hits. Cause classes should be a closed set: network-clock, hash-seed, thread-schedule, fs-race. unknown is not a class. It is a queue.

Proposed enforcement:

1. On failure, compute the signature.
2. If no ledger row matches, fail the job. Do not auto-append.
3. If a row matches and today <= expires, record FROZEN and continue.
4. If the row is expired, fail. Renewal needs a human and a shorter window, not a copy-paste of the old date.

Cap the open freeze count. A ledger with twenty live rows is a disabled suite with extra XML. Publish the count in CI summary output so it cannot hide in a JSON file.

python -m oracle.ledger --check --max-open 5 --today 2026-09-16

Step 5 — Run the contract on a machine the agent does not keep warm

Local green is necessary and insufficient. Caches, dirty trees, and previously installed packages hide dead oracles. Run the same three files on a clean runner: install, verify fixture digests, run properties, run the kill matrix, apply the flake ledger, then exit non-zero on any DEAD_ORACLE or expired freeze.

A proposed driver:

set -euo pipefail
git checkout --force "$SHA"
python -m pip install -e '.[test]'
python -m oracle.fixtures --verify
python -m oracle.kill --source oracle/properties.py --require-no-dead
python -m pytest oracle/ -q
python -m oracle.ledger --check --max-open 5

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft the candidate patch; the free server option can host the clean run of the sealed oracle. Neither claim is a quota, a hardware spec, or a quality ranking. The method does not depend on that product. Any ephemeral runner with a read-only oracle/ mount works. The product is relevant only when you want a second lane that is not the laptop that produced the diff.

If the clean run and the laptop disagree, believe the clean run. Then debug the laptop, not the matrix.

What the matrix does not prove

Property-kill shows the oracle can fail. It does not show the specification is complete. A load-bearing permutation check will not catch a sort that orders by absolute value if your domain never includes negatives. Expand INT_CASES on purpose, as a review item, not as an afterthought.

Fixture digests do not detect semantic drift when two different byte sequences are both "valid." If the format is commutative JSON, canonicalize before hashing or every key reorder looks like sabotage.

The flake ledger cannot encode Heisenbugs you have not classified. A freeze without a cause class is a skip. Treat it that way in review.

Remote runs do not replace secret hygiene. Do not ship production credentials to a shared runner to make the oracle green.

Who should not use this

Do not adopt the three-file contract if the agent is allowed to write oracle/. You would be scoring the student's answer key.

Skip property-kill on pure generated snapshots with no independent predicate (UI pixel dumps, vendor protobufs you cannot parse). You need a checker, not a hash of yesterday's PNG, unless bit-stability is the actual requirement.

Hardware-in-the-loop suites with irreducible timing noise need a different statistical gate. A 14-day freeze on thread-schedule is a temporary budget, not a strategy, if every test is a race.

One-off scripts and throwaway spikes do not pay for a ledger. The overhead is for patches you might merge.

Review checklist

1. oracle/ is not in the agent's write set.
2. Kill matrix has zero DEAD_ORACLE rows.
3. Each ticket claim maps to a LOAD_BEARING row.
4. Fixture lock changes are human-authored and classified.
5. Open flake rows are below cap and unexpired.
6. Clean-runner exit code is the merge bit; laptop green is advisory.

The agent can keep writing tests. Those tests are clues. The oracle is the measurement. Mutate it until it can fail, lock the bytes it reads, and freeze flakes by who they are—not by which file you are tired of seeing.