Score Agent Patches by Oracle Class, Then Bound the Campaign

An agent patch is not mergeable because the suite is green. It is mergeable when every green check has a named oracle class, a finite campaign bound, and a freeze record for anything that is not deterministic.

Pass count is a residue. It records whatever the agent was allowed to satisfy. If the oracle is weak, green is cheap. If the oracle is a live model call, green is also noisy.

This workflow scores a patch by oracle class first. It then spends a bounded campaign on the classes that can actually constrain behavior. Flakes are not skipped. They are frozen as seed, input, and class — or they block merge.

The four oracle classes

Treat every assertion as one of four classes. Do not mix them in one test function.

1. Fixture oracle. Byte-stable input and expected output, owned by a human, hashed in git. Highest merge weight.
2. Invariant oracle. A property over generated inputs: round-trip, monotonicity, idempotence, schema preservation. High weight only when the generator is seeded and the iteration cap is explicit.
3. Metamorphic oracle. Two related inputs, one relation. Useful when a golden output does not exist. Medium weight. Still requires a seed.
4. Judge oracle. A model grades the patch in natural language. Lowest weight. Never a merge gate. This is how flakes get manufactured.

A test that calls a model to decide pass/fail is class 4 even if it sits next to a fixture. Classification follows the oracle, not the filename.

Decision table: class, location, merge weight

Run location is part of the score. A property that is unbounded on a shared runner is not a stronger oracle. It is a budget leak.

Class	Local pre-merge	Bounded remote campaign	Merge weight	Flake policy
Fixture	Required	Replay only	Block	Fail. Do not skip.
Invariant	50–200 seeded iters	Cap wall-clock and iters	Block if any fail	Freeze seed+input
Metamorphic	Pairwise, seeded	Same cap as invariant	Block if relation breaks	Freeze both inputs
Judge	Offline proposal only	Optional, non-gating	Informational	Drop from gate

The table is the artifact you review, not the JUnit XML. If a row is missing, the patch is unclassified. Unclassified checks do not raise the score.

Numbered workflow

Follow the steps in order. Skipping a step is how a judge oracle sneaks into CI.

1. Inventory the diff. List production files, test files, and any new generator. Label each new assertion with a class from the table.
2. Promote or reject. Fixture and invariant stay. Judge-shaped checks are rewritten into properties or deleted from the merge path.
3. Pin generators. Every property takes an explicit seed: int and max_examples: int. No wall-clock-only loops. No unseeded random.
4. Write the freeze ledger. One JSON document per campaign. It stores failures, not vibes.
5. Run local, then bounded remote. Local must be deterministic. Remote only expands iteration count under a cap.
6. Score the patch. Merge requires: all fixture oracles pass, all invariant/metamorphic oracles pass inside the bound, zero unfrozen flakes in classes 1–3, and zero judge oracles on the gate.

Artifact: freeze ledger and campaign runner

The ledger is human-owned. The runner is mechanical. Label the following as a proposed, unexecuted layout — adapt paths to your repo.

{
  "campaign_id": "agent-patch-2026-09-13",
  "seed": 20260913,
  "max_examples": 128,
  "wall_clock_s": 90,
  "oracles": [
    {"id": "json.roundtrip", "class": "invariant", "status": "pass"},
    {"id": "sort.monotonic", "class": "invariant", "status": "frozen",
     "counterexample": {"input": [3, 1, 1], "shrunk": [1, 1]}},
    {"id": "llm.style_grade", "class": "judge", "status": "excluded"}
  ]
}

A frozen invariant is not a skip. It is a failing fixture that has not been accepted yet. Either promote the shrunk input to a fixture oracle, or reject the patch. Expiring a freeze without a fixture is how the same flake returns on the next agent diff.

Proposed runner (Python 3.12). It refuses to import a client inside a gating test:

from __future__ import annotations

import json
import os
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Iterator

LEDGER = Path("test/oracles/freeze_ledger.json")
FORBIDDEN_IMPORTS = ("openai", "anthropic", "httpx")  # extend per repo


@dataclass(frozen=True)
class Campaign:
    seed: int
    max_examples: int
    wall_clock_s: int


def seeded_ints(seed: int, n: int) -> Iterator[int]:
    # xorshift32 — deterministic, no global RNG
    x = seed & 0xFFFFFFFF or 1
    for _ in range(n):
        x ^= (x << 13) & 0xFFFFFFFF
        x ^= (x >> 17)
        x ^= (x << 5) & 0xFFFFFFFF
        yield x


def refuse_judge_imports(source: str) -> None:
    lowered = source.lower()
    for name in FORBIDDEN_IMPORTS:
        if f"import {name}" in lowered or f"from {name}" in lowered:
            raise AssertionError(f"judge oracle import on merge path: {name}")


def run_invariant(
    name: str,
    pred: Callable[[int], None],
    campaign: Campaign,
) -> dict:
    deadline = time.monotonic() + campaign.wall_clock_s
    seen = 0
    for value in seeded_ints(campaign.seed, campaign.max_examples):
        if time.monotonic() > deadline:
            break
        try:
            pred(value)
        except Exception as exc:
            return {
                "id": name,
                "class": "invariant",
                "status": "frozen",
                "counterexample": {"input": value, "error": type(exc).__name__},
            }
        seen += 1
    return {"id": name, "class": "invariant", "status": "pass", "seen": seen}


def write_ledger(rows: list[dict], campaign: Campaign) -> None:
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    payload = {
        "seed": campaign.seed,
        "max_examples": campaign.max_examples,
        "wall_clock_s": campaign.wall_clock_s,
        "oracles": rows,
    }
    LEDGER.write_text(json.dumps(payload, indent=2))
    frozen = [r for r in rows if r.get("status") == "frozen"]
    if frozen:
        raise SystemExit(f"freeze ledger has {len(frozen)} open counterexample(s)")

Gate command. Keep it boring.

python -c "from pathlib import Path; import sys;
[sys.exit('judge import on merge path') for p in Path('tests').rglob('*.py')
 if any(x in p.read_text().lower() for x in ('import openai','import httpx'))]"
pytest tests/oracles -q --maxfail=1
python -m oracles.campaign --seed 20260913 --max-examples 128 --wall-clock 90

The first command is a cheap static refuse. The second is fixture oracles. The third is the only place iteration count may grow. If step three is unbounded, you no longer have a campaign. You have a soak that CI will flake under load.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is relevant in step 2 of the workflow: proposing candidate properties from a diff. That is an offline drafting step. Humans still classify the proposal and promote it to a fixture or an invariant. The same free model must not re-enter the merge gate as a judge oracle. Scoring the patch with the author of the patch is circular.

MonkeyCode's free server option is relevant in step 5: expanding max_examples under wall_clock_s without burning the laptop or the protected CI pool. The server runs the campaign runner. It does not get a vote. If the ledger comes back with a freeze, the patch stays closed until the shrunk input is a fixture or the production change is reverted.

Do not treat free access as a permanent capacity number. Bounds in the ledger are yours. If the remote runner is slow, lower max_examples. Do not remove the seed.

What this does not cover

This is not a mutation-testing study. It does not claim a coverage percentage. It does not name models, quotas, or hardware. Time-sensitive product limits should be read from the operator's current primary docs before you schedule a campaign.

It also does not replace code review of production behavior. A perfect invariant on the wrong function is still a wrong patch. Oracle class is a filter on tests, not a proof of product intent.

Who should not use this

Skip the campaign layer if the change is a one-line config with a single fixture. The table still applies. The runner does not.

Do not use judge oracles as a parallel "AI review" job that can fail the build. That job will flake, then someone will mark it optional, then it will stop existing.

Do not use this on non-deterministic production code you have not first wrapped behind a clock, a network cassette of your own, or an injected RNG. Properties on live time and live HTTP measure the universe. They do not measure the patch.

Teams that cannot own a freeze ledger in git should stay on fixture oracles only. A bound you cannot store is not a bound.

Close

Score the class. Bound the campaign. Freeze the counterexample. If you already pin fixtures locally, a free server is useful only as that bounded runner — not as a second judge. Promote a shrunk input or reject the diff. Green without an oracle class is not a score.