Test Agent Patches With an Oracle the Diff Cannot Touch

An agent patch is only as trustworthy as the checks it cannot rewrite. If properties, fixtures, and flake policy live in the same tree as src/, the diff can weaken the proof. Move the oracle out of the writable tree and run it as a control loop with hysteresis, not as a skip list.

Co-located tests fail this requirement in a predictable way. The agent adds an assertion that matches the new code. A fixture grows a default that hides a broken parser. A flaky case becomes skip. The suite stays green. Production still drifts.

This article proposes a sidecar oracle: human-owned properties, sealed fixtures, and a two-threshold flake freeze. The design is a workflow, not a production case study. Treat the code as a proposed runner you can execute locally, not as a claim about a live fleet.

What the loop decides

The loop answers three questions on every candidate patch:

1. Do independent properties still hold on generated inputs?
2. Did the patch mutate a sealed fixture or depend on an unsealed one?
3. Is a failing test a regression, or does it belong in a measured freeze?

A skip list answers none of those. It only records that someone got tired of a red job.

Layout: oracle beside the repo, not inside the diff

Keep the application repo writable for the agent. Keep the oracle in a second directory that the agent cannot include in its patch.

app/                      # agent may write src/, not oracle paths
  src/
  pyproject.toml
oracle/                   # human-owned; hashed before every gate
  properties/
    test_invariants.py
  fixtures/
    manifest.json
    http_empty_body.json
  flake_ledger.json
  path_deny.txt
  run_gate.py

path_deny.txt is the first control, not the last. If the patch touches oracle files, tests the agent authored, or lockfiles it did not need, the gate fails before pytest starts.

# oracle/path_deny.txt
oracle/
**/test_*.py
**/*_test.py
**/conftest.py
**/__snapshots__/

The deny list is deliberately blunt. Agent-authored tests can still exist as scratch. They do not count as evidence.

Step 1 — Hash the oracle before the agent runs

Record the oracle state first. If the hash moves during a session, the session is invalid, even when pytest is green.

find oracle -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/oracle.before
# run the agent against app/ only
find oracle -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/oracle.after
diff -u /tmp/oracle.before /tmp/oracle.after

A non-empty diff is a gate failure. Do not “repair” the oracle from the patch. Restore it and reject the candidate.

Step 2 — Run properties the agent did not write

Property checks belong in oracle/properties/. They should encode invariants that remain true across refactors: round-trips, status classes, envelope shape, forbidden fields. They should not encode the patch’s new happy path.

The example below is a proposed pytest module. It uses Hypothesis if present, and a small deterministic fallback if not. Label it as an unexecuted template until you wire it to your types.

# oracle/properties/test_invariants.py
from __future__ import annotations

import json
from typing import Any

import pytest

try:
    from hypothesis import given, settings
    from hypothesis import strategies as st
except ImportError:  # pragma: no cover - optional dependency
    given = None

from app.src.http_envelope import parse_envelope, serialize_envelope

FORBIDDEN = ("ssn", "password", "id_token")

def _assert_envelope(payload: dict[str, Any]) -> None:
    parsed = parse_envelope(payload)
    dumped = serialize_envelope(parsed)
    again = parse_envelope(dumped)
    assert again.status // 100 in {2, 4, 5}
    assert again.body is not None or again.status == 204
    text = json.dumps(dumped, sort_keys=True).lower()
    for key in FORBIDDEN:
        assert key not in text

@pytest.mark.parametrize(
    "payload",
    [
        {"status": 200, "body": {"ok": True}},
        {"status": 204, "body": None},
        {"status": 502, "body": {"error": "upstream"}},
    ],
)
def test_known_envelopes_round_trip(payload: dict[str, Any]) -> None:
    _assert_envelope(payload)

if given is not None:

    @settings(max_examples=80, deadline=None)
    @given(
        st.fixed_dictionaries(
            {
                "status": st.sampled_from([200, 201, 204, 400, 404, 409, 500, 502]),
                "body": st.none() | st.dictionaries(
                    st.text(min_size=1, max_size=12),
                    st.integers() | st.text(max_size=40),
                    max_size=6,
                ),
            }
        )
    )
    def test_generated_envelopes_round_trip(payload: dict[str, Any]) -> None:
        if payload["status"] == 204:
            payload = {**payload, "body": None}
        _assert_envelope(payload)

Three constraints keep this honest. Properties must not import agent-written test helpers. Generators must not read fixtures the patch just added. Failures must print the shrinking input, not only assert False.

If a property is expensive, cap examples and time in the oracle, not by deleting the property from the app repo. Budget lives with the proof.

Step 3 — Seal fixtures; do not lease them back to the patch

Fixtures that decide pass/fail should be content-addressed. The agent may read them. The gate refuses a patch that changes bytes, path, or media type.

{
  "version": 1,
  "fixtures": [
    {
      "id": "http_empty_body",
      "path": "fixtures/http_empty_body.json",
      "sha256": "3b4c1f0e0a7d9c2a6f11d8b0c4e5a917e2c1d0b9a8f7e6d5c4b3a29180776655",
      "must_remain": {"status": 200},
      "must_not_infer_success_from_empty_body": true
    }
  ]
}

A proposed checker:

# excerpt from oracle/run_gate.py
import hashlib
import json
from pathlib import Path

ORACLE = Path(__file__).resolve().parent

def seal_error(manifest_path: Path) -> list[str]:
    errors: list[str] = []
    manifest = json.loads(manifest_path.read_text())
    for item in manifest["fixtures"]:
        blob = (ORACLE / item["path"]).read_bytes()
        digest = hashlib.sha256(blob).hexdigest()
        if digest != item["sha256"]:
            errors.append(f"unsealed fixture {item['id']}: {digest}")
        payload = json.loads(blob)
        if payload.get("body") in ("", None, {}, []) and payload.get("status") == 200:
            if item.get("must_not_infer_success_from_empty_body"):
                # Fixture documents a trap. Production code must not treat this as OK.
                pass
    return errors

The empty-body case is the point. A 200 with no payload is not success. Sealing the fixture stops the agent from “fixing” the test by stuffing {} into the file. The property suite still has to assert that the parser rejects that envelope.

New fixtures need a human seal: compute the hash, add the manifest row, and state which invariant the bytes exist to protect. Unsealed files in oracle/fixtures/ fail the gate.

Step 4 — Freeze flakes with hysteresis, not expiry folklore

A freeze is a control state. It is not a skip marker and not a calendar reminder. Use two thresholds so noise does not toggle the suite on every job.

Proposed ledger schema:

{
  "window_n": 20,
  "enter_rate": 0.20,
  "exit_m": 50,
  "exit_rate": 0.02,
  "tests": {
    "tests/test_retry.py::test_upstream_timeout": {
      "state": "frozen",
      "recent": [0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1],
      "quarantine": []
    }
  }
}

recent is the last window_n non-quarantine results (1 = fail). quarantine is the scheduled reruns used only to decide exit. A proposed update rule:

from statistics import fmean

def next_state(record: dict, failed: bool, cfg: dict) -> dict:
    bit = 1 if failed else 0
    state = record.get("state", "active")
    if state == "frozen":
        q = (record.get("quarantine") or []) + [bit]
        q = q[-cfg["exit_m"] :]
        record["quarantine"] = q
        if len(q) >= cfg["exit_m"] and fmean(q) <= cfg["exit_rate"]:
            record["state"] = "active"
            record["recent"] = q[-cfg["window_n"] :]
            record["quarantine"] = []
        return record

    recent = (record.get("recent") or []) + [bit]
    recent = recent[-cfg["window_n"] :]
    record["recent"] = recent
    if len(recent) >= cfg["window_n"] and fmean(recent) >= cfg["enter_rate"]:
        record["state"] = "frozen"
        record["quarantine"] = []
    return record

Rules that keep the freeze from becoming a junk drawer:

1. Frozen tests still run on a quarantine schedule. They do not vote on merge.
2. Enter and exit rates are different. That gap is the hysteresis. Without it, a 19% flake oscillates.
3. A patch that turns an active, previously stable test red is a regression, not a freeze candidate. Freeze admission requires a window, not a single fail.
4. Humans add tests to the ledger. The agent does not.

Expiry-only freezes fail in the other direction. A 30-day skip can hide a real break that landed on day 2. Hysteresis tracks rate, not a date.

Step 5 — Assemble the gate as one command

Proposed order. Stop at the first failure class.

python oracle/run_gate.py \
  --app app \
  --patch /tmp/candidate.diff \
  --deny oracle/path_deny.txt \
  --manifest oracle/fixtures/manifest.json \
  --ledger oracle/flake_ledger.json \
  --pytest-args "-q oracle/properties"

Inside run_gate.py, the sequence is mechanical:

1. Reject diffs that touch deny paths.
2. Re-hash oracle/ against the pre-agent snapshot.
3. Verify fixture seals.
4. Run oracle/properties with a fixed seed and a wall-clock cap.
5. Classify remaining app-suite failures through the ledger. Active fails block. Frozen fails record quarantine bits only.

Do not let the agent choose pytest markers. Markers are policy.

Where an isolated model session fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a disposable place to let an agent edit app/src without touching the oracle, MonkeyCode’s free model access and free server option can host that sandbox. The oracle still runs where you control the filesystem hash. Pass/fail must not come from the same process that proposed the hunks.

The split matters more than the vendor. Generate in a throwaway tree. Verify with a tree the generator cannot write.

Failure modes the loop still misses

Properties only hold for the generators you wrote. A numeric envelope will not catch a locale bug in date strings. Sealed fixtures freeze today’s traps; they do not invent tomorrow’s. Hysteresis needs volume. A test that runs twice a week cannot enter or exit freeze with honest rates.

The loop also assumes path control. If the agent can push directly to main, or if CI checks out a single tree and runs pytest with no deny list, the sidecar is decoration.

Concurrency is another gap. Two agents sealing different fixtures into one ledger will clobber state unless you serialize gate runs per repo.

Who should not use this

Do not install a sidecar oracle if you have no human owner for properties. An empty oracle/properties/ is a slower green build.

Do not use hysteresis on suites with fewer than a few dozen runs per test per week. The window will be noise.

Do not apply path deny lists to repos where generated golden files are the product. Those trees need a different seal workflow, with explicit regeneration commands, not a blanket test_*.py ban.

Do not treat this as a substitute for production probes. Independent oracles reduce false greens in merge. They do not observe live traffic.

A minimal acceptance check

Before you trust the loop, break it on purpose. The following cases should fail closed. They are a test plan for the gate, not a scoreboard from a past incident.

1. Add oracle/properties/test_invariants.py to the diff. Gate fails on deny paths.
2. Flip one byte in a sealed fixture. Gate fails on hash mismatch.
3. Rewrite parse_envelope to treat empty body as 200 OK. Property or sealed-fixture invariant fails.
4. Mark a frozen test as skip from the patch. Gate fails because the agent edited a test file.
5. Fail an active test once. Ledger records a bit; merge still fails. Ten sparse fails over twenty runs may freeze; one fail must not.

If any of those pass, the oracle is still inside the diff, or the freeze is still a skip list.

Wire the sidecar before you widen the agent’s path filter. The cheap move is more write access. The durable move is a proof the patch cannot touch.