Do Not Let an Agent Patch Edit the Oracle It Must Pass

An agent patch that changes production code and also rewrites the tests that judge that code is not a stronger patch. It is a quieter one. Oracle authority has moved from the repository to the model, and a green suite no longer measures intended behavior. Score the write-set before you score coverage.

This article proposes a three-lane merge policy. Property checks live in a registry the agent cannot extend in the same diff. Fixtures are hashed at HEAD. Flaky tests are frozen by node ID rather than skipped. The method is a classifier plus a frozen pytest invocation. It is not a claim about model quality.

Mixed write-sets are the failure, not missing tests

Agents fail in a stable pattern. They edit src/, then they edit the assertion, the golden file, or the retry decorator so the new behavior looks intended. More tests do not catch that. Tests authored in the same diff often encode the same mistake.

Three file classes decide whether CI is still an oracle:

1. Property checks — quantified invariants, not single examples.
2. Fixtures — inputs, golden files, recorded HTTP, clocks, and RNG seeds.
3. Flake controls — skips, xfails, reruns, timeout bumps, and seed reshuffles.

If a production patch touches any of those classes, the merge question is no longer “did tests pass.” It is whether the agent was allowed to grade its own homework.

Lane 1: a property registry the patch cannot grow

Keep properties in a dedicated tree, for example oracle/properties/. Register each module in oracle/registry.toml at HEAD. The agent may execute those properties. It may not add a module and a registry row in the same commit that edits production code.

Proposed layout. This is an example contract, not a live repository.

# oracle/registry.toml — committed at HEAD
[property.order_total]
path = "oracle/properties/test_order_total.py"
impact = ["src/billing/", "src/orders/"]
min_examples = 200

[property.idempotent_refund]
path = "oracle/properties/test_idempotent_refund.py"
impact = ["src/billing/refunds.py"]
min_examples = 100

A patch that adds src/billing/discount.py must still satisfy property.order_total when the impact list matches. It must not introduce test_discount_is_fine.py that only asserts discount >= 0. That check is an example shaped like a tautology. It does not restore the invariant.

Unexecuted Hypothesis sketch for the registered property:

# oracle/properties/test_order_total.py
from hypothesis import given, settings
from hypothesis import strategies as st
from billing import Order, Line, total

@settings(max_examples=200, deadline=None)
@given(
    st.lists(
        st.builds(
            Line,
            qty=st.integers(1, 99),
            cents=st.integers(0, 50_000),
        ),
        min_size=1,
    )
)
def test_total_equals_sum_of_line_cents(lines):
    order = Order(lines=lines)
    assert total(order) == sum(l.qty * l.cents for l in lines)

Ownership is the point. Properties are versioned like production contracts. The agent consumes them. It does not author them on the way to green.

Lane 2: hash fixtures at HEAD, refuse silent regeneration

Golden files and recorded payloads are oracles with better marketing. An agent that “updates snapshots” has rewritten expected output. Freeze the corpus with a manifest captured at HEAD.

# proposed capture; run on a detached HEAD
git checkout --detach HEAD
find fixtures -type f -print0 | sort -z | xargs -0 sha256sum > oracle/fixtures.sha256
git checkout -

The merge gate compares oracle/fixtures.sha256 to a fresh hash of the same paths in the working tree. Any changed fixture path fails unless it appears in a human-owned allow file, oracle/fixture-allow.txt, landed in a separate change.

Recorded clocks and RNG seeds belong in that manifest. If fixtures/seed.json moves from {"rng": 4} to {"rng": 17} so a property “passes,” the classifier must treat it as an oracle edit. It is not test maintenance.

Canonicalize JSON on write if key order is unstable. Do not relax the hash. Equivalent-looking fixtures are still a write to the oracle.

Lane 3: freeze flaky tests by ID, do not skip them

A skip is a hole. A freeze is a pin. Record the node IDs that were unstable at HEAD. Refuse two outcomes: the patch marks those IDs passed by changing timing, and the patch deletes or skips them to green the job.

Collect IDs, then curate the unstable subset by hand:

pytest --collect-only -q | sed 's/[[:space:]]*$//' > /tmp/all-ids.txt
# copy known-unstable rows into oracle/flake_freeze.json

Proposed freeze file:

{
  "frozen_at_sha": "HEAD",
  "ids": [
    "tests/test_cache.py::test_ttl_under_load",
    "tests/test_http.py::test_retry_on_429"
  ],
  "rule": "must_not_change_status_or_decorators"
}

The gate diffs pytest node IDs plus decorators (pytest.mark.flaky, reruns=, timeout=) for those IDs only. Unfrozen tests may still move. Frozen IDs stay in the suite so the flake remains visible. They are not a license to ignore failures in adjacent tests.

If a frozen test starts failing because the production patch is wrong, that is a signal. Do not thaw it in the same diff. Thaw in a human PR that records the new SHA and the reason.

Write-set classifier (runnable artifact)

The script below is a proposed gate. It classifies paths, computes an oracle-authority score, and exits non-zero when production code and oracle files share a diff. Prefixes are examples. Map them to the repo before use.

#!/usr/bin/env python3
"""oracle_authority.py — classify a git diff. Proposed gate, not a benchmark."""
from __future__ import annotations

import subprocess
import sys

PROD = ("src/", "lib/", "pkg/", "app/")
WEIGHT = {"prod": 0, "prop": 4, "fix": 3, "flake": 5, "other": 0}
FLAKE_KEYS = ("reruns", "timeout", "xfail", "skip", "flaky")


def names() -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", "HEAD"], text=True
    )
    return [n for n in out.splitlines() if n]


def lane(path: str) -> str:
    if path.startswith("oracle/properties/") or path.endswith("oracle/registry.toml"):
        return "prop"
    if path.startswith("fixtures/") or path == "oracle/fixtures.sha256":
        return "fix"
    if path.startswith("oracle/flake") or path in ("pytest.ini", "pyproject.toml"):
        return "flake"
    if path.startswith(PROD):
        return "prod"
    return "other"


def flake_text_widened(path: str) -> bool:
    if path not in ("pytest.ini", "pyproject.toml"):
        return False
    diff = subprocess.check_output(["git", "diff", "HEAD", "--", path], text=True)
    return any(k in diff for k in FLAKE_KEYS)


def main() -> int:
    files = names()
    lanes = {p: lane(p) for p in files}
    score = sum(WEIGHT[lanes[p]] for p in files)
    if any(lanes[p] == "flake" and flake_text_widened(p) for p in files):
        score += 5
    prod = any(v == "prod" for v in lanes.values())
    oracle = any(v in {"prop", "fix", "flake"} for v in lanes.values())
    print("write-set:")
    for p, l in sorted(lanes.items()):
        print(f"  {l:5}  {p}")
    print(f"oracle_authority_score={score} prod={prod} oracle={oracle}")
    if prod and oracle:
        print("reject: production and oracle edited in the same diff", file=sys.stderr)
        return 2
    if score >= 4:
        print("reject: oracle-authority score >= 4 without a split PR", file=sys.stderr)
        return 2
    return 0


if __name__ == "__main__":
    sys.exit(main())

Install it as a required check, not as a comment on the PR:

chmod +x oracle_authority.py
python oracle_authority.py; echo exit:$?
sha256sum -c oracle/fixtures.sha256
pytest oracle/properties -q --maxfail=1

Decision table for the same gate:

Production edited	Properties / registry	Fixtures hash	Flake freeze or reruns	Action
yes	no	unchanged	unchanged	Run registered properties. Merge if they pass.
yes	added or edited	any	any	Reject. Split into two PRs.
yes	no	changed	no	Reject unless fixture-allow.txt is human-owned and separate.
yes	no	unchanged	IDs thawed or reruns raised	Reject. Keep the freeze.
no	yes	any	any	Oracle PR. Human review required. No model-only merge.

The table is the policy. The script only enforces it.

Numbered workflow

1. At HEAD, commit oracle/registry.toml, oracle/fixtures.sha256, and oracle/flake_freeze.json. Those three files are the contract the agent does not own.
2. Create a branch. Restrict the working tree so the agent can write under src/ only. Mount oracle/ and fixtures/ read-only when the runner supports bind mounts.
3. Generate a production-only patch. If a model endpoint and a machine are needed without standing up private GPU capacity, MonkeyCode's free model access and free server option can host that loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
4. Run python oracle_authority.py. A non-zero exit ends the attempt. Do not lower the score by editing fixtures.
5. Run the registered properties against the hashed corpus. Fail closed if either command fails.
6. If properties fail, fix production code or abandon the patch. If they pass and the classifier is clean, merge the production PR.
7. Open a second PR only when a human intends to add a property, rotate a fixture, or thaw a flake ID. That PR contains no production edits.

Proposed isolation for step 2:

git worktree add /tmp/agent-prod HEAD
# oracle/ and fixtures/ stay in the main worktree, mounted read-only into the runner
# agent apply is limited to /tmp/agent-prod/src
git -C /tmp/agent-prod apply --check /tmp/agent.patch && git -C /tmp/agent-prod apply /tmp/agent.patch

Pipe model output through git apply --check before git apply. Untrusted text is not a tree.

The two-PR split is the method. Tools only keep the lanes from collapsing.

What the score does not measure

The score does not say a property is a good property. A registry full of assert True stays green. Pair this gate with a separate check that rejects tautological assertions. That is a different classifier, and it belongs on the oracle PR, not on the production PR.

Path prefixes lie in monorepos. Generated trees under src/generated/ may look like production and should map to other or to their own lane. pyproject.toml mixes flake keys with unrelated metadata. A line-aware parse is safer than the substring test above.

Fixture hashing does not understand semantically equivalent JSON. Reordered keys fail the gate. That is intentional. Canonicalize on write, or accept the friction.

A freeze can hide a real regression if the frozen test was the only coverage for a path the agent edited. Frozen IDs should be ticketed and short-lived. They are not a permanent quarantine.

Who should not use this

Do not install a three-lane oracle gate on throwaway scripts, on patches that exist only to add tests, or on repos with no stable HEAD corpus. If every test is an end-to-end browser run with no fixture hash, the classifier will only nag. If one person owns a 200-line library and reads every diff, the write-set policy is overhead.

Do not treat a clean score as evidence that a model is safe. The gate constrains where the model may write. It proves nothing beyond the registered properties.

Skip this approach when the change is the oracle: migrating Hypothesis settings, rotating a recorded HTTP cassette, or deleting a dead fixture. Those belong on a human PR with production frozen, which is the inverse of the agent path.

Closed-fail behavior on a free runner

A free server is only a place to run the classifier and pytest oracle/properties. It does not replace recording frozen_at_sha from the repo's own HEAD. If the runner checks out a different default branch, fixture hashes will disagree. The gate should fail closed. Do not weaken that to keep a demo green.

Never let the apply step write oracle/ or fixtures/. If the runner cannot enforce a write-set, the classifier is the last check, not an optional linter.

If last week's agent PR is still in the history, run oracle_authority.py against that diff. The write-set score is more informative than adding another example test.