Agent Patches Need a Second Oracle and a Fail-Closed Flake List

Agent-authored tests are a weak merge signal when they share a session with the patch. Grade the diff with an independent oracle instead: properties the author did not write, fixtures the author cannot rewrite, and flakes that stay red until a human attaches a reproducer. Calendar expiry is how flakes return. Same-commit property files are how self-grading sneaks in.

The failure mode, stated plainly

A coding agent often emits production code and a test file in one pass. The test encodes the agent's current belief, not the system's contract. Green CI then means the author agreed with itself.

Flakes make that signal worse. Teams skip them, delete them, or pin them until a date. After the date, the same nondeterminism ships again. A skip is a deleted assertion with better public relations. An expiry is a scheduled skip.

Three artifacts close the loop. None of them should originate in the same agent session that produced src/.

Artifact map

Artifact	Owner	Agent may touch in the patch commit?	Sufficient to merge?
Fixture lock (fixtures.lock.json)	Human or CI hasher	No	Required
Independent properties (oracle/**)	Second environment	No	Required
Flake freeze (flake_freeze.json)	Human reviewer	No	Required; empty is allowed
Agent-authored unit tests	Agent	Yes	Not sufficient

Read the table as a gate. If the production hunk and the oracle hunk share a commit, the gate fails. Author tests may remain in the tree as intent notes. They do not grade the patch.

1. Hash fixtures before the agent runs

Freeze the inputs the patch is allowed to read. Use a content hash. Directory mtimes are not a lock.

The helper below is proposed code, not a measured benchmark.

# fixture_lock.py — proposed helper
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("tests/fixtures")
LOCK = Path("fixtures.lock.json")


def file_digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def build_lock(root: Path = ROOT) -> dict[str, str]:
    entries = {}
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        entries[rel] = file_digest(path)
    return entries


def verify_lock() -> list[str]:
    expected = json.loads(LOCK.read_text())
    actual = build_lock()
    problems = []
    for name, digest in expected.items():
        if name not in actual:
            problems.append(f"missing:{name}")
        elif actual[name] != digest:
            problems.append(f"changed:{name}")
    for name in actual:
        if name not in expected:
            problems.append(f"untracked:{name}")
    return problems


if __name__ == "__main__":
    import sys

    mode = sys.argv[1] if len(sys.argv) > 1 else "verify"
    if mode == "write":
        LOCK.write_text(json.dumps(build_lock(), indent=2, sort_keys=True) + "\n")
        print(f"wrote {LOCK}")
    else:
        problems = verify_lock()
        if problems:
            print("fixture lock failed:")
            print("\n".join(problems))
            raise SystemExit(1)
        print("fixture lock ok")

Commands:

python fixture_lock.py write    # human, before the agent session
python fixture_lock.py verify   # CI, after the agent session

If the agent “fixes” a failing check by editing a golden JSON file, verify fails. That is the intended outcome. Untracked files fail the same way. A new fixture is a review event, not a silent extra input.

2. Keep properties out of the patch commit

Property checks assert invariants: round-trip encoding, idempotent apply, bounded size, monotonic counters. They do not assert that function f returned the literal the agent just hard-coded. Author unit tests may do that. The oracle must not.

Generate properties in a second environment. A free coding model on a free server is enough for this side of the split, because the job is “state invariants against frozen fixtures,” not “ship the product.”

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option can host that second environment so the oracle session does not share files, history, or prompts with the patch session. That is an isolation claim. It is not a quality ranking, a quota, or a latency number.

Label the following as proposed invariants, not executed results.

# oracle/properties/test_apply_invariants.py — proposed checks
from __future__ import annotations

import json
from pathlib import Path

import pytest

from app.patch import apply_hunk, parse_hunk

FIXTURES = Path("tests/fixtures")


def iter_cases():
    for path in sorted(FIXTURES.glob("*.json")):
        payload = json.loads(path.read_text())
        yield path.name, payload


@pytest.mark.oracle
@pytest.mark.parametrize("name,payload", list(iter_cases()))
def test_apply_is_idempotent(name, payload):
    once = apply_hunk(payload["base"], payload["hunk"])
    twice = apply_hunk(once, payload["hunk"])
    assert once == twice, name


@pytest.mark.oracle
@pytest.mark.parametrize("name,payload", list(iter_cases()))
def test_parse_round_trip(name, payload):
    hunk = parse_hunk(payload["hunk"])
    assert parse_hunk(hunk.to_text()) == hunk

CI must reject co-change of production code and oracle files in one commit:

changed=$(git diff --name-only origin/main...HEAD)
src=$(printf '%s\n' "$changed" | grep -E '^src/' || true)
oracle=$(printf '%s\n' "$changed" | grep -E '^oracle/' || true)
if [ -n "$src" ] && [ -n "$oracle" ]; then
  echo "oracle files must not ship in the same commit as src/"
  echo "$oracle"
  exit 1
fi

The patch commit stays mergeable only when src/ and optional author tests move. Properties land in a follow-up commit from the second environment, after fixture_lock.py verify passed. Shared commit hash is the tautology channel. Close it in git, not in a style guide.

3. Freeze flakes fail-closed, with a reproducer, not a date

Do not skip a flake. Do not give it a calendar. Record it. Keep the merge job red until a human-owned reproducer file exists and its hash is listed in the freeze record. After that, the test may run on a night job. It still cannot green merge.

{
  "version": 1,
  "policy": "fail_closed_until_reproducer",
  "entries": [
    {
      "nodeid": "tests/test_stream.py::test_partial_read",
      "first_seen_sha": "REPLACE_WITH_COMMIT_SHA",
      "symptom": "empty body with HTTP 200",
      "reproducer": null
    }
  ]
}

# flake_freeze.py — proposed gate
from __future__ import annotations

import hashlib
import json
import os
from pathlib import Path

FREEZE = Path("flake_freeze.json")
REPRO_ROOT = Path("oracle/reproducers")


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> int:
    data = json.loads(FREEZE.read_text())
    if data.get("policy") != "fail_closed_until_reproducer":
        print("unknown flake policy")
        return 1
    failures = []
    for entry in data.get("entries", []):
        nodeid = entry["nodeid"]
        repro = entry.get("reproducer")
        if not repro:
            failures.append(f"{nodeid}: no human reproducer")
            continue
        path = REPRO_ROOT / repro["file"]
        if not path.is_file():
            failures.append(f"{nodeid}: missing {path}")
            continue
        if digest(path) != repro["sha256"]:
            failures.append(f"{nodeid}: reproducer hash mismatch")
            continue
        if os.environ.get("MERGE_JOB") == "1":
            failures.append(f"{nodeid}: frozen tests cannot green merge")
    if failures:
        print("flake freeze gate:")
        print("\n".join(failures))
        return 1
    print("flake freeze ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

A filled freeze entry looks like this. The reproducer is a human-reduced script or recorded input, not an agent-written skip marker.

{
  "file": "stream_empty_body.py",
  "sha256": "REPLACE_WITH_SHA256_OF_THE_REPRODUCER_FILE"
}

Merge job, fail-closed:

python fixture_lock.py verify
python flake_freeze.py
MERGE_JOB=1 python flake_freeze.py
pytest -m oracle --strict-markers

Night job may omit MERGE_JOB=1 and run frozen node ids for signal. Required checks on the pull request must include the merge job only. If a frozen test starts passing, remove the freeze in a human commit after the reproducer still explains the old failure. Do not auto-delete the entry because a date passed.

4. Decision procedure

1. Human writes or updates fixtures. Run python fixture_lock.py write. Commit the lock in a reviewable change.
2. Agent produces src/ and optional unit tests on one branch. No oracle/ edits. No fixture edits.
3. CI verifies the lock, rejects co-change of oracle/ with src/, and runs existing oracle properties against the new code.
4. Second environment — separate prompt, separate machine — adds or updates properties. New commit. Same lock.
5. Any new flake enters flake_freeze.json with reproducer: null. Merge stays red.
6. Human adds oracle/reproducers/... and fills the hash. Night job may run it. Merge job still refuses to go green on that node id.
7. Merge only when the lock is clean, oracle properties pass, freeze policy holds, and the src/ commit does not contain oracle files.

The order is load-bearing. Properties written before the lock exist will bind to moving inputs. Properties written in the agent commit will bind to the agent's belief. Flakes recorded after merge will not be seen.

Who should not use this

Skip the workflow for a one-line typo fix with no fixture surface. Skip it when tests cannot run without live network I/O you do not control. Skip it when the team cannot keep oracle/ in a second commit. A monorepo that regenerates golden files on every run will fight the lock and should split recorded inputs from generated outputs first.

Do not point the second environment at production credentials. Frozen fixtures should be sanitized. The free server is an oracle host, not a production replica. Independent properties still need a human to confirm the invariant is the real contract. Idempotence can pass while authorization is wrong.

Limitations of the gate itself

The co-change check is commit-scoped, not intent-scoped. Two rapid commits can still smuggle author-written properties. Reviewers must read git log -- oracle. Protect fixtures.lock.json and flake_freeze.json with CODEOWNERS so the agent cannot clear the board in a follow-up commit.

SHA256 does not detect semantic drift when a reviewer runs python fixture_lock.py write on an agent-edited golden file. The lock is only as honest as the write step. Property suites under-specify behavior. Add domain invariants as review finds holes, not as a one-shot generated dump.

This workflow does not measure model quality. It does not claim that a free model writes better properties than a paid one. It does not replace code review. It only stops the patch author from grading its own homework and stops flakes from expiring themselves back into the merge path.

Closing

Author tests can stay. They document intent. They are not the grade. The grade is a lockfile, a second-session property suite, and a flake list that cannot expire itself.

If you need a machine that is not the agent's workspace for that second session, MonkeyCode's free model access and free server option is one place to run the oracle side. Keep the fixtures and the freeze file in your repo either way.