Oracle Order for Agent Diffs: Replay, Properties, Then a Flake Budget

Agent patches should be judged by oracles that never call a model. Generation can happen on a free remote server. The merge verdict cannot.

Most suites treat fixtures, properties, and flake freezes as a bag of checks. Order is not written down. Agents exploit that gap. They shrink replay data, loosen invariants, or spend the freeze list on tests they just added. The gate still prints green.

This article is a testing strategy, not a prompt-tuning note. It specifies an oracle sequence, a decision table, and a local make judge path. The code is a template. It is labeled as such and was not executed for this draft.

Why order is the control

A replay failure is a behavior change. A property failure is an invariant break. A flake is residual noise after those two have passed.

If you freeze first, you teach the suite to ignore new failure classes. If you run properties before replay, fixture drift shows up as shrinking noise. If oracle files are writable by the same diff that claims to fix production code, the agent edits the judge.

Name-based freezes are a weak control. Agents rename tests. The freeze list does not follow the behavior. Bind residual retries to tests that already existed, not to strings an agent can rewrite.

Three oracles, one sequence

Replay fixtures record exact inputs and outputs. No search. No network. A miss is a diff.

Properties quantify over generated inputs. They cover cases the fixture corpus never saw. They must be deterministic: integer math, injected clocks, no live endpoints.

A flake budget is last and small. It applies only to tests that existed before the patch. New tests get zero retries.

Decision table

Keep this table in oracles/MATRIX.md. Fail CI if the file is missing. Humans use the table. The patch is judged by the tests that implement it.

Failure class	First oracle	Illegal first move	Merge only if
Output changed on a known cart	Replay fixture	Add a freeze	Fixture still matches, or the contract file is updated under a review label
Total cents go negative on random coupons	Property	Expand fixtures only	Invariant holds on the stated domain
Same node id fails 1/N on wall clock	Flake budget	Disable the test	Retry count is inside budget and the test pre-existed the patch
Agent edited oracles/ or judge tests	Review label	Auto-merge	Oracle diff is empty or ORACLE_REVIEW=1
Judge job reaches a model endpoint	Job config	Retry the model	JUDGE_OFFLINE=1 and model env vars are unset

The table is the artifact. Everything below is one way to enforce it in Python.

Workflow

1. Generate the patch off the judgment path

Treat model output as untrusted text. A generation host may expose free model access and a free server option. That is a convenience for producing a diff. It is not a test runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode can sit in that generation step: free model access, and a free server option, both operator-supplied. Do not put that host in the job that prints the merge verdict. This article does not claim model names, quotas, hardware, duration, or permanence. Generation and judgment are different processes. That is the requirement.

2. Materialize a diff, then disconnect

git fetch origin
git checkout -b agent/discount-stack
git apply --check /tmp/agent.patch
git apply /tmp/agent.patch
export JUDGE_OFFLINE=1
unset MODEL_ENDPOINT MODEL_API_KEY
make judge

If make judge cannot run without a model client, the suite is not a judge. It is another generator.

3. Replay fixtures first

Keep money in integer cents. Float totals manufacture flakes that later get frozen. The example domain is coupon stacking. Order is part of the contract. Do not hide that behind a set.

Template (unexecuted):

# discounts.py
from dataclasses import dataclass
from typing import Sequence

@dataclass(frozen=True)
class Coupon:
    kind: str          # 'percent' | 'fixed'
    bps: int           # basis points, or cents when kind is fixed
    cap_cents: int | None = None

def stack_discounts(subtotal_cents: int, coupons: Sequence[Coupon]) -> int:
    if subtotal_cents < 0:
        raise ValueError('subtotal_cents must be >= 0')
    remaining = subtotal_cents
    for coupon in coupons:
        if remaining == 0:
            break
        if coupon.kind == 'percent':
            if not 0 <= coupon.bps <= 10_000:
                raise ValueError('bps out of range')
            cut = remaining * coupon.bps // 10_000
            if coupon.cap_cents is not None:
                cut = min(cut, coupon.cap_cents)
            remaining -= cut
        elif coupon.kind == 'fixed':
            if coupon.bps < 0:
                raise ValueError('fixed coupon negative')
            remaining -= min(remaining, coupon.bps)
        else:
            raise ValueError(f'unknown coupon kind: {coupon.kind}')
        if remaining < 0:
            remaining = 0
    return remaining

Replay against a locked JSONL corpus. One object per line. No comments. No live currency API.

# tests/test_replay.py
import json
from pathlib import Path

from discounts import Coupon, stack_discounts

FIXTURES = Path('oracles/carts.jsonl')

def test_replay_locked_carts():
    lines = FIXTURES.read_text().splitlines()
    assert lines, 'fixture corpus missing'
    for line in lines:
        row = json.loads(line)
        coupons = [Coupon(**c) for c in row['coupons']]
        got = stack_discounts(row['subtotal_cents'], coupons)
        assert got == row['expected_cents'], row['id']

Example fixture row:

{"id":"cart-017","subtotal_cents":1999,"coupons":[{"kind":"percent","bps":1500,"cap_cents":300},{"kind":"fixed","bps":50,"cap_cents":null}],"expected_cents":1649}

A patch that adds fixture rows is a contract change. Route it to a labeled review, not to the default agent merge path. Replay must stay first. Do not fold it into a single pytest session with properties.

4. Run properties second

Properties search the domain fixtures never recorded. They still cannot call a model. Bound the search. Unbounded generation is not thoroughness. It is a timeout that someone will mark flaky.

Template (unexecuted):

# tests/test_properties.py
from hypothesis import given, settings, strategies as st

from discounts import Coupon, stack_discounts

Coupons = st.lists(
    st.one_of(
        st.builds(Coupon, kind=st.just('percent'), bps=st.integers(0, 10_000), cap_cents=st.none() | st.integers(0, 50_000)),
        st.builds(Coupon, kind=st.just('fixed'), bps=st.integers(0, 50_000), cap_cents=st.none()),
    ),
    max_size=8,
)

@settings(max_examples=200, deadline=None)
@given(subtotal=st.integers(0, 10_000_000), coupons=Coupons)
def test_total_stays_in_cents_bounds(subtotal, coupons):
    got = stack_discounts(subtotal, coupons)
    assert 0 <= got <= subtotal

@settings(max_examples=100, deadline=None)
@given(subtotal=st.integers(0, 10_000_000), coupons=Coupons)
def test_empty_suffix_is_idempotent(subtotal, coupons):
    once = stack_discounts(subtotal, coupons)
    twice = stack_discounts(subtotal, list(coupons) + [])
    assert once == twice

Do not encode unordered coupon application unless the contract says so. If stacking order is product law, a commutativity property will green a wrong patch. Match the property to the written contract, not to a hope.

5. Spend flake budget last

Retries are residual. They are not a policy. Inventory node ids before the patch. New ids get zero retries.

pytest --collect-only -q tests > /tmp/after.txt
git stash push -u -- tests discounts.py
pytest --collect-only -q tests > /tmp/before.txt
git stash pop
comm -13 <(sort /tmp/before.txt) <(sort /tmp/after.txt) > oracles/new_nodeids.txt

Template (unexecuted):

# tests/flake_budget.py
from pathlib import Path

MAX_RETRY = 1
NEW = Path('oracles/new_nodeids.txt')

def retries_for(nodeid: str) -> int:
    new_ids = {line.strip() for line in NEW.read_text().splitlines() if line.strip()}
    if nodeid in new_ids:
        return 0
    return MAX_RETRY

Hook it in conftest.py so a retry cannot appear as a silent pass:

# tests/conftest.py
import os
import pytest

from tests.flake_budget import retries_for

def pytest_configure(config):
    if os.environ.get('JUDGE_OFFLINE') != '1':
        raise pytest.UsageError('JUDGE_OFFLINE=1 is required')
    if os.environ.get('MODEL_ENDPOINT') or os.environ.get('MODEL_API_KEY'):
        raise pytest.UsageError('model env vars must be unset in judge')

def pytest_runtest_protocol(item, nextitem):
    item.flaky_retries = retries_for(item.nodeid)
    return None

Implement the actual retry loop with pytest-rerunfailures or a tiny plugin if you already have one. The control is the inventory file, not the plugin brand. A freeze list of test names is the thing to avoid.

6. Fail closed on oracle edits

oracle_touch=$(git diff --name-only origin/main -- \
  oracles tests/test_replay.py tests/test_properties.py tests/flake_budget.py tests/conftest.py)
if [ -n "$oracle_touch" ] && [ "$ORACLE_REVIEW" != "1" ]; then
  printf 'oracle files changed without ORACLE_REVIEW=1\n%s\n' "$oracle_touch"
  exit 1
fi

Production code and oracle code do not share a default merge path. If the agent must change a fixture expected value, that is a product decision. It is not a test fix.

Judge job sketch

judge:
    test "$$JUDGE_OFFLINE" = "1"
    test -z "$$MODEL_ENDPOINT"
    test -z "$$MODEL_API_KEY"
    test -f oracles/MATRIX.md
    pytest tests/test_replay.py -q
    pytest tests/test_properties.py -q
    pytest tests/ --maxfail=1

Replay and properties are separate invocations. A property shrink must not skip fixture failures in the same session. The third invocation is where a bounded retry may run, and only for pre-patch node ids.

Limitations

This sequence does not prove the patch is useful. It only proves the patch did not break recorded behavior, stated invariants, or the residual budget. Semantic quality still needs a human on the contract file.

Hypothesis will not save a wrong invariant. If the property encodes the bug, the suite protects the bug. Integer cents remove a common flake source. They do not remove IO, threads, or unordered sets. Those need injected fakes. Do not freeze them.

The generation host is untrusted. Free endpoints can return empty bodies, stale caches, or partial diffs. None of that belongs in make judge. Offline judgment also will not catch a patch that only fails when a third-party API changes shape. Put that risk in a separate, non-blocking probe if you need it. Do not mix it with merge oracles.

Who should skip this

Skip the pipeline if the agent is allowed to rewrite tests as part of the same merge. The oracles are then self-dealing.

Skip it if product behavior is still undefined and fixtures are being invented. You need a contract before you need a gate.

Skip it if every test must call a live model. That is evaluation of a model, not judgment of a patch against a program.

Teams that already pin seeds, clocks, and maps can layer this order on top. Order is additive. It is not a replacement for those pins.

If you already generate candidate diffs from a free model endpoint, keep that process outside the job that prints the merge verdict. Point make judge at the diff.