Split the Oracle From the Author Before You Merge an Agent Patch

An agent patch is not verified when the model that wrote it also grades the result. That is a shared session, not a test. Split generation from scoring. Bind the scorer to a human-owned property catalog, a digest-locked fixture set, and a freeze ledger that records flakes as dated exceptions instead of retrying until the job turns green.

This article is a proposed workflow, not a report of a production incident. The code is labeled as example code. It does not claim pass rates, model rankings, or hardware profiles.

The failure mode this gate is built to catch

Author models optimize for a green diff. They will happily emit a test that restates the patch, assert on values the patch just printed, or delete the assertion that would have failed. A second model in the same prompt window is not an independent check. It has already seen the rationale.

The split is mechanical. The author process may touch production code. The oracle process may read the catalog, the fixtures, and the freeze ledger. It may not receive the author prompt, the author chain of thought, or write access to the catalog.

If those two processes share a conversation, credentials, or working tree writes, the rest of this article does not apply.

Artifact 1: a property catalog the agent cannot edit

Write the catalog before the patch exists. Each row is an invariant, not a scenario title. Keep the language checkable. Vague goals such as "be faster" or "handle errors" do not belong here.

# properties.yaml — human-owned; author process is read-denied
version: 1
subject: token_bucket
properties:
  - id: P1_cap
    statement: "issued tokens in any window of width W never exceed capacity C"
    kind: forall_trace
  - id: P2_monotonic_clock
    statement: "fixture clocks only move forward; oracle rejects a backward jump"
    kind: fixture_wellformed
  - id: P3_no_silent_drop
    statement: "a refused issue returns Retry-After >= 0 and does not decrement the bucket twice"
    kind: forall_trace
  - id: P4_idempotent_key
    statement: "the same (key, n) pair in one window yields the same allow/deny decision"
    kind: pairwise
oracle_rules:
  author_may_edit: []
  fail_closed_on_unknown_property: true
  freeze_is_not_a_pass: true

Store it outside the path the agent is allowed to patch. CI should fail if the catalog hash changes in the same commit as production code, unless a human-labeled catalog path is in the diff and a reviewer file exists. That rule is the point. Properties are the test plan. The agent does not get to rewrite the plan in order to pass it.

Artifact 2: fixtures locked by digest, not by filename

Filename locks are weak. Agents rename files. They also rewrite JSON and keep the old name. Hash the canonical bytes. Canonical means sorted keys, LF newlines, and no wall-clock field that the oracle does not declare as an input.

# fixture_digest.py — example / unexecuted template
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any


def canonical_bytes(obj: Any) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8") + b"\n"


def digest_file(path: Path) -> str:
    payload = json.loads(path.read_text(encoding="utf-8"))
    if "now_ms" in payload and "clock" not in payload:
        raise ValueError(f"{path} uses wall time; promote it to a fixture clock")
    return hashlib.sha256(canonical_bytes(payload)).hexdigest()


def write_lock(fixture_dir: Path, lock_path: Path) -> None:
    rows = []
    for path in sorted(fixture_dir.glob("*.json")):
        rows.append({"path": str(path.as_posix()), "sha256": digest_file(path)})
    lock_path.write_text(canonical_bytes(rows).decode("utf-8"), encoding="utf-8")

A fixture is a recorded trace: clock, prior bucket state, request, expected decision class. It is not a captured model completion. If a fixture includes an expected numeric token count that the patch can freely choose, you have encoded the implementation. Encode the class of outcome instead: allow, deny, retry_after_ms_range.

Re-record only with a human flag. The oracle runner must refuse a digest mismatch. A mismatch is a failed gate, not a hint to regenerate fixtures.

Artifact 3: an oracle process that never sees the author prompt

Run scoring in a different process, preferably a different machine identity. The author job emits a patch and a file list. The oracle job receives the catalog, the lockfile, the freeze ledger, and the patched tree. It does not receive the author prompt file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need that process split without standing up a private GPU box, MonkeyCode's free model access and free server option can host the oracle runner. That is an availability claim only. It is not a quality ranking, a quota, or a hardware spec. Keep author credentials out of the oracle job either way.

Example layout:

jobs:
  author:
    writes: [src/, patch.diff]
    forbidden_reads: [properties.yaml, freeze.jsonl, fixtures.lock]
  oracle:
    reads: [properties.yaml, fixtures.lock, freeze.jsonl, src/, tests/fixtures/]
    forbidden_reads: [author_prompt.txt, author_scratch/]
    writes: [scorecard.jsonl]

The oracle may call a model to classify a trace against a property id. It may not ask "is this patch good?" Open-ended grading reintroduces the author bias. Pass the property statement and the trace. Require a structured verdict.

# oracle_runner.py — example / unexecuted template
from __future__ import annotations

import json
from dataclasses import dataclass
from typing import Literal

Verdict = Literal["pass", "fail", "unscored"]


@dataclass(frozen=True)
class Score:
    property_id: str
    fixture: str
    verdict: Verdict
    reason: str


def score_trace(property_row: dict, trace: dict, model_complete) -> Score:
    prompt = {
        "property_id": property_row["id"],
        "statement": property_row["statement"],
        "trace": trace,
        "reply_schema": {"verdict": ["pass", "fail"], "reason": "short"},
    }
    raw = model_complete(json.dumps(prompt))
    try:
        parsed = json.loads(raw)
        verdict = parsed["verdict"]
        if verdict not in ("pass", "fail"):
            raise ValueError("non-binary verdict")
        return Score(property_row["id"], trace["id"], verdict, parsed.get("reason", ""))
    except (json.JSONDecodeError, KeyError, ValueError) as exc:
        return Score(property_row["id"], trace["id"], "unscored", f"malformed oracle output: {exc}")

Treat unscored as a fail-closed result. Do not majority-vote across retries. Retries hide instability. Instability belongs in the freeze ledger or in a failed job.

A local deterministic checker should handle P2 (clock monotonicity) and any purely numeric cap. Do not spend a model call on inequalities the runtime can evaluate. Use the model only where the property is a classification over a trace that is awkward to encode as a pure function.

Artifact 4: a freeze ledger, not a retry budget

Flakes are not a reason to loop the oracle. A flake is evidence that a property is coupled to an uncontrolled input: time, network, model sampling, or hash iteration order. Record it. Expire it. Do not count it as a pass.

{
  "property_id": "P1_cap",
  "fixture": "deny_at_capacity.json",
  "reason": "verdict flipped across two oracle processes with identical digest",
  "frozen_on": "2026-09-13",
  "expires_on": "2026-09-20",
  "owner": "human",
  "blocks_merge": false,
  "counts_as_pass": false
}

Rules that keep the ledger from becoming a junk drawer:

1. Only a human can add a row. Agent-authored freeze rows are rejected.
2. counts_as_pass is always false. Frozen means "excluded from the green set," not "treated as green."
3. Expired rows fail the job until they are removed or the underlying property is rewritten.
4. A property with more than one open freeze cannot be used to green a patch that touches its subject.
5. The same (property_id, fixture) pair cannot be frozen twice without a new owner note.

# freeze_ledger.py — example / unexecuted template
from __future__ import annotations

from datetime import date, datetime
from typing import Iterable


def ledger_errors(rows: Iterable[dict], today: date) -> list[str]:
    errors = []
    seen = set()
    open_by_prop: dict[str, int] = {}
    for row in rows:
        key = (row["property_id"], row["fixture"])
        if key in seen:
            errors.append(f"duplicate freeze {key}")
        seen.add(key)
        if row.get("counts_as_pass"):
            errors.append(f"{key} counts_as_pass must be false")
        if row.get("owner") != "human":
            errors.append(f"{key} owner is not human")
        expires = datetime.strptime(row["expires_on"], "%Y-%m-%d").date()
        if expires < today:
            errors.append(f"{key} expired on {expires.isoformat()}")
        else:
            open_by_prop[row["property_id"]] = open_by_prop.get(row["property_id"], 0) + 1
    for pid, n in open_by_prop.items():
        if n > 1:
            errors.append(f"{pid} has {n} open freezes; subject is not mergeable")
    return errors

If the oracle is non-deterministic, freeze the pair and reduce sampling to zero on the next attempt, or replace the model call with a pure checker. Do not raise temperature and hope.

Merge gate: numbered checks, fail closed

Run these in order. Stop at the first failure. Order matters because later steps are expensive and earlier steps catch catalog drift cheaply.

1. Verify properties.yaml is not in the author write set.
2. Recompute fixture digests. Compare to fixtures.lock.
3. Parse the freeze ledger. Fail on expiry, duplicates, or non-human owners.
4. Execute pure checkers (clock monotonicity, numeric caps) on every fixture.
5. For remaining properties, run the oracle process with no author prompt in scope.
6. Write scorecard.jsonl. Any fail or unscored fails the job.
7. Properties with an open freeze are omitted from the green set. They do not contribute a pass.
8. Refuse merge if the green set is empty. A patch with only frozen properties is unverified.

# example local sequence — unexecuted
python -m fixture_digest --dir tests/fixtures --lock fixtures.lock --check
python -m freeze_ledger --file freeze.jsonl --today 2026-09-13
python -m oracle_runner --catalog properties.yaml --lock fixtures.lock \
  --ledger freeze.jsonl --out scorecard.jsonl
python -m scorecard_gate --file scorecard.jsonl --min-green 1

A useful scorecard row is boring on purpose:

{"property_id":"P3_no_silent_drop","fixture":"refuse_then_retry.json","verdict":"pass","frozen":false}

Do not attach prose summaries from the author model. Those summaries are not evidence.

Decision table

Observation	Action	Merge
Digest mismatch	Fail. Do not regenerate in CI.	No
Catalog edited in the same commit as src/ without a reviewer file	Fail.	No
Pure checker fails	Fail. Not a freeze candidate.	No
Oracle returns malformed JSON	unscored → fail closed.	No
Same fixture flips pass/fail across two identical oracle runs	Human freeze row, expiry required	No, until other properties still green the subject
Freeze expired	Fail until removed or property rewritten	No
All remaining properties pass, none frozen on that subject	Allow	Yes
Only frozen properties cover the subject	Unverified	No

The table is the policy. If a case is not in the table, the gate fails. Adding a case is a human catalog change, not an agent patch.

What this does not prove

A passing scorecard does not prove the patch is correct. It proves the declared properties held on the locked fixtures under a split oracle. Unstated behavior stays unstated. A wrong fixture that hashes stably is still wrong. Cross-model disagreement is a fail in this workflow, not a vote.

Free model access does not make the oracle ground truth. It only makes the process split cheaper to run. If the property requires exact numeric reproduction, write a pure checker and leave the model out.

Do not use this approach for safety-critical control paths that need formal methods, for patches whose only tests are UI screenshots, or for teams that cannot keep the catalog out of the agent's write set. If the author and the oracle share a session, you are scoring a rationale, not a patch.

The workflow is also a poor fit when every property is a freeze. That is a catalog failure. Rewrite the properties until at least one is deterministic on the fixture set.

If you already keep author traffic off the scoring path, a free remote oracle is one way to keep that split. Review the catalog hash in the same PR you review the patch. The catalog is the test plan. Treat it that way.