If the Agent Can Edit the Oracle, a Green Build Is Not Evidence

A passing job is not a merge decision. It is a color. When an agent can rewrite tests in the same tree as production code, that color can be purchased by deleting the assertion that would have failed.

Score the patch on three facts the agent does not control: oracle bytes versus the merge base, seeded property hits with a minimum pairwise distance, and a flake freeze that names a cause and an expiry. If any of those three is missing, keep the patch off the default branch.

The failure mode is not "weak tests"

Weak tests are a human problem. Agent patches add a second one: the check and the change travel together.

A tautology is obvious when a human writes assert True. It is quieter when the agent drops a fixture row, loosens a numeric bound, or marks a property xfail in the same commit that claims to fix a bug. The suite stays green. The behavior did not move. The merge log still looks clean.

Do not start from coverage. Coverage rises when the agent adds files it also tests. Start from oracle ownership. Then measure whether independent checks actually fired.

Layer 1 — Put oracles in a path the patch cannot touch

Keep independent checks under oracles/. That directory is not a style preference. It is a merge invariant.

The gate should refuse the patch if oracles/ differs from the merge base. Protocol changes belong in a human-authored PR that updates oracles first. Agent patches consume those oracles. They do not edit them.

Setup, in order:

1. Move property checks, golden fixtures, seed corpora, and freeze metadata under oracles/.
2. Add a CI step that diffs that path against the merge base and fails closed.
3. Block merge when the diff is non-empty, even if pytest is green.
4. Review oracle PRs without the agent in the loop.

git fetch origin main
BASE=$(git merge-base origin/main HEAD)
git diff --exit-code "$BASE" HEAD -- oracles/

If that command exits non-zero, the rest of the scorecard is advisory. It is not a merge signal.

Pin the oracle runner to a lockfile the agent also cannot edit. A rewritten oracles/requirements.txt is the same class of failure as a rewritten assertion.

python -m pip install -r oracles/requirements.txt
python oracles/gate.py --parent /tmp/parent --patch /tmp/patch

Treat application tests next to production code as characterization at most. They can document intent. They cannot award merge credit.

Layer 2 — Score seeded properties by hit distance, not by pass count

A property that never fails on parent and never fails on the patch is not evidence. A property that fails on one seed the agent memorized is also not evidence.

Lock a seed corpus in oracles/seeds.json. Replay the same properties on parent and on the patch. Credit the patch only for seeds that fail on parent, pass on patch, and remain at least D bits apart from each other. Distance is a cheap proxy for "the agent did not overfit a single blob." It is not semantic diversity. Treat it as a filter, not as a proof.

Proposed corpus shape (example data, not a measured suite):

{
  "property": "parse_roundtrip",
  "seeds": [
    {"id": "s1", "payload": {"raw": "a,b", "n": 2}},
    {"id": "s2", "payload": {"raw": "", "n": 0}},
    {"id": "s3", "payload": {"raw": "a,,c", "n": 3}}
  ]
}

The replay helper must be a pure function of payload plus tree. No clock. No network. No hidden files outside oracles/ and the application package under test.

# oracles/replay.py — proposed runner, not production telemetry
from __future__ import annotations

import json
import sys
from typing import Any, Callable


def parse_roundtrip(payload: dict[str, Any]) -> None:
    raw = payload["raw"]
    n = payload["n"]
    parts = raw.split(",") if raw != "" else []
    if len(parts) != n:
        raise AssertionError(f"expected {n} parts, got {parts!r}")
    if ",".join(parts) != raw:
        raise AssertionError("join did not recover raw")


PROPS: dict[str, Callable[[dict[str, Any]], None]] = {
    "parse_roundtrip": parse_roundtrip,
}


def main() -> int:
    prop = sys.argv[1]
    payload = json.load(sys.stdin)
    try:
        PROPS[prop](payload)
    except Exception:
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Proposed gate (example scorer, not a benchmark report):

# oracles/gate.py — example scorer, not measured production results
from __future__ import annotations

import argparse
import hashlib
import json
import subprocess
import sys
from datetime import date, datetime
from pathlib import Path

import yaml


def hamming(a: bytes, b: bytes) -> int:
    n = max(len(a), len(b))
    a, b = a.ljust(n, b"\0"), b.ljust(n, b"\0")
    return sum(bin(x ^ y).count("1") for x, y in zip(a, b))


def seed_bytes(payload: dict) -> bytes:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).digest()


def run_property(tree: Path, prop: str, payload: dict) -> int:
    proc = subprocess.run(
        [sys.executable, str(tree / "oracles" / "replay.py"), prop],
        input=json.dumps(payload),
        text=True,
        cwd=tree,
        capture_output=True,
    )
    return proc.returncode


def freeze_debt(root: Path, today: date) -> list[str]:
    doc = yaml.safe_load((root / "oracles" / "flake_freeze.yml").read_text())
    bad: list[str] = []
    for row in doc.get("entries", []):
        if not row.get("cause") or not row.get("ticket"):
            bad.append(f"{row.get('id')}: missing cause or ticket")
            continue
        exp = datetime.strptime(row["expires"], "%Y-%m-%d").date()
        if exp < today:
            bad.append(f"{row['id']}: expired {row['expires']}")
    return bad


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--parent", type=Path, required=True)
    p.add_argument("--patch", type=Path, required=True)
    p.add_argument("--today", default=date.today().isoformat())
    args = p.parse_args()
    policy = yaml.safe_load((args.patch / "oracles" / "policy.yml").read_text())
    corpus = json.loads((args.patch / "oracles" / "seeds.json").read_text())
    hits = []
    for seed in corpus["seeds"]:
        parent_rc = run_property(args.parent, corpus["property"], seed["payload"])
        patch_rc = run_property(args.patch, corpus["property"], seed["payload"])
        if parent_rc != 0 and patch_rc == 0:
            hits.append(seed)
    kept = []
    for seed in sorted(hits, key=lambda s: s["id"]):
        sb = seed_bytes(seed["payload"])
        if all(
            hamming(sb, seed_bytes(k["payload"])) >= policy["min_seed_distance"]
            for k in kept
        ):
            kept.append(seed)
    debt = freeze_debt(args.patch, date.fromisoformat(args.today))
    card = {
        "oracle_hits": len(kept),
        "raw_fail_to_pass": len(hits),
        "min_seed_distance_policy": policy["min_seed_distance"],
        "hit_ids": [s["id"] for s in kept],
        "freeze_debt": debt,
        "merge": len(kept) >= policy["min_oracle_hits"] and not debt,
    }
    print(json.dumps(card, indent=2))
    return 0 if card["merge"] else 2


if __name__ == "__main__":
    raise SystemExit(main())

The printed object is the merge input. 15 passed is not. If oracle_hits is 0, the patch did not move an independent check. A green pytest job does not override that zero.

Worktrees exist only to make parent and patch replay boring. They are not a second product. Install from the same lockfile in both trees so the runner cannot drift.

git worktree add /tmp/parent "$BASE"
git worktree add /tmp/patch HEAD
python oracles/gate.py --parent /tmp/parent --patch /tmp/patch

Layer 3 — Freeze flakes only with a named cause and an expiry

Flakes destroy oracle credit. Teams then freeze tests until the suite is silent. Silence is not stability.

A freeze file is allowed. An anonymous freeze is not. Each row needs a test id, a cause class, a ticket, and an expiry. After expiry, the freeze is merge debt. The next agent patch does not receive a green lane while that row is still present.

Cause classes worth encoding:

1. order — collection-order dependence
2. time — clock, sleep, or wall timeout
3. net — real network
4. rng — unseeded randomness
5. shared — leaked process state

If the cause is unknown, do not freeze. Quarantine the test out of the blocking lane, and record that the oracle set shrank. Shrinking the oracle is visible. A quiet skip is not.

# oracles/flake_freeze.yml — example policy, not a live ticket dump
version: 1
entries:
  - id: tests/test_cache.py::test_ttl_under_load
    cause: time
    ticket: FLAKE-184
    expires: "2026-09-27"
    note: "wall-clock assertion; rewrite on a logical clock in a human PR"

# oracles/policy.yml
min_oracle_hits: 3
min_seed_distance: 8
oracle_paths:
  - oracles/

Expiry is a date, not a comment. The gate should parse it. A freeze without cause or ticket is also debt. The YAML is part of the oracle tree, so the agent cannot extend it without failing Layer 1.

A concrete gate workflow

Run this after the agent opens a patch. Do not substitute it for review.

1. Confirm oracles/ is byte-identical to the merge base, including freeze, seeds, and the replay lockfile.
2. Create two worktrees. Do not reuse the agent's local virtualenv or its editable install.
3. Replay oracles/seeds.json on both trees. Keep fail-to-pass seeds only.
4. Drop any hit within distance D of a kept hit. Retain a set that still meets H.
5. Load oracles/flake_freeze.yml. Reject the patch if any freeze is expired or missing a cause.
6. Emit the scorecard JSON. Merge only when hits ≥ H, the distance filter still leaves those hits, freeze debt is empty, and the oracle diff is empty.

Decision table:

Oracle diff	Fail-to-pass hits	Min seed distance	Expired freeze	Merge
empty	≥ H	≥ D	none	allow
empty	≥ H	< D	none	reject, corpus too tight
empty	0	n/a	none	reject, no oracle movement
nonempty	any	any	any	reject, agent edited the check
empty	≥ H	≥ D	present	reject, freeze debt

H and D are local policy. Publish them in oracles/policy.yml. Do not bury them in a chat prompt the agent can rewrite.

Read the scorecard as three numbers and one list: oracle_hits, raw_fail_to_pass, min_seed_distance_policy, and freeze_debt. If raw_fail_to_pass is large and oracle_hits is small, the corpus is collinear. Widen the seeds in a human PR. Do not lower D in the agent patch.

What this does not prove

Seed distance is syntactic. Two JSON blobs can be far apart in Hamming space and still exercise one branch. Large fixtures can still be collinear. The gate does not replace a reviewer who can read the patch.

The freeze taxonomy is incomplete. Browser tests with animation timing will not fit cause: time cleanly. A YAML enum is not a flake debugger.

Parent/patch replay assumes the oracle is deterministic on parent. If parent already flakes, you do not have an oracle. You have a coin. Stop scoring agent patches against that check until a human PR freezes it with a cause or fixes it.

The gate also says nothing about tests the agent added next to application code. Those tests can still be useful as characterization. They are not oracle hits. Count them somewhere else, or do not count them.

Who should not use this

Do not install this gate if your only suite is a remote browser grid with an irreducible flake rate and no seedable input. You will freeze everything and then merge on empty oracles.

Do not use it as a reason to skip reading generated code. Independent oracles catch behavioral regressions the agent did not hide. They do not catch a leak, a license file, or a deleted auth check that no property mentioned.

Do not point the agent at oracles/ "just this once." That is how the invariant dies. If the protocol changed, land the oracle update first, with a human reviewer, then let the agent implement against the new checks.

Skip the Hamming filter if your seeds are already enumerated by a domain expert and are few enough to inspect by hand. Distance is a crowd-control tool for large corpora. It is overhead on a twelve-row table.

Where generation still belongs

Generating candidate patches is cheap relative to scoring them. A coding agent with free model access and a free server option can propose diffs against application code while CI owns the oracle path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already generate those diffs on a free server, point the two worktrees at that checkout and read the scorecard before the pytest color. The interesting output is still oracle_hits and freeze_debt, not the model that proposed the diff.