Pairwise Oracles Beat Value Asserts for Agent Patches

Agent-generated patches routinely pass the tests that arrive in the same diff. That is not a behavior proof. It is a proof that the asserts and the implementation were allowed to move together.

A merge gate built on assert actual == expected is the wrong shape for this failure. Both sides of the equality can be produced by one model. The stronger check is a relation across two executions: transform the input in a documented way, and the output must transform in a documented way. No golden vector lives in the test file.

Value oracles do not constrain policy

Take a payment splitter, allocate(amount_cents: int, weights: list[int]) -> list[int]. The usual review fixture looks like this:

assert allocate(100, [1, 1, 2]) == [25, 25, 50]

That line stays green under several incompatible policies. Remainder cents can slide to the last bucket. Zero weights can be dropped. Outputs can be sorted "for readability." The fixture never stated those rules, so the patch is free to invent them.

Conservation is also missing. Rounding can leak a cent and the golden list will not notice unless a reviewer picked a case that exposes the leak. The constraint "parts sum to the whole" is a relation. It does not belong in a constant.

A common agent patch makes this concrete. The model keeps the one reviewed case and ignores weights everywhere else:

def allocate(amount_cents, weights):
    if amount_cents == 100 and weights == [1, 1, 2]:
        return [25, 25, 50]
    n = len(weights) or 1
    q, r = divmod(amount_cents, n)
    return [q] * (n - 1) + [q + r]

The value assert still passes. A permutation of [1, 1, 2] does not. Equal split of 100 cents is [33, 33, 34], which is not a permutation of [25, 25, 50]. The bug is not a missing fixture. It is a missing pairing rule.

Put the contract in a relation table

The table below is the proposed artifact. Each row is a metamorphic relation. Rows do not store a full expected vector. The agent is not asked to guess an answer. It is asked not to break a pairing rule.

ID	Relation	Input transform	Must hold
MR00	Domain	empty weights	[] if amount_cents == 0, else raise
MR01	Conservation	none	sum(out) == amount_cents
MR02	Permutation	apply permutation P to weights	out' == P(out)
MR03	Zero insertion	insert weight 0 at index i	out' == out[:i] + [0] + out[i:]
MR04	Positive scale	multiply amount by k in 2..4	out' == [k * x for x in out]
MR05	Duplication	concatenate weights with itself, double amount	out' == out + out
MR06	Sign	any negative weight	raise, same exception type both times

MR04 is strict on purpose. If leftover cents walk left to right, doubling is not always exact. Mark that row policy-sensitive and keep it out of the blocking set until the remainder rule is written down. Do not delete it. A listed skip is cheaper than a silent behavior change.

MR02 is the row that kills the hardcoded fixture. The test never mentions [25, 25, 50]. It only mentions that shuffling weights must shuffle the split.

Sample inputs. Do not pin outputs.

Single-execution properties are a different gate. This sampler exists only to feed pairs. It does not load a file of expected splits. Output fixture files are exactly what an agent can patch in lockstep with production code.

Proposed sampling rules:

1. Draw amount_cents from {0, 1, 2, 99, 100, 101, 10**6}.
2. Draw weight-list lengths from {0, 1, 2, 5, 8}.
3. Include a zero weight in at least one third of draws.
4. Draw permutations from a seeded random.Random, not from hash() order.
5. Reject draws that a relation cannot apply to. Do not weaken the row to keep the sample count pretty.

Empty weight lists belong in MR00, as a precondition, not as a golden list. Pick one behavior and keep it. Returning [amount_cents] for empty weights will pass a careless conservation check and still be wrong.

Record the seed in the job log. A failing pair is located by (seed, relation_id, draw_index), not by a pytest node name. Node names change when the agent rewrites the test file.

Proposed harness

The following Python is a proposed, unexecuted example. It treats the relation table as data. Happy-path magic numbers do not appear.

# proposed harness — unexecuted example
from __future__ import annotations

import random
import traceback
from dataclasses import dataclass
from typing import Callable

Alloc = Callable[[int, list[int]], list[int]]

@dataclass(frozen=True)
class Break:
    relation_id: str
    draw_index: int
    detail: str


def sample(seed: int, n: int = 48) -> list[tuple[int, list[int]]]:
    rng = random.Random(seed)
    amounts = [0, 1, 2, 99, 100, 101, 10**6]
    out = []
    for i in range(n):
        amount = rng.choice(amounts)
        length = rng.choice([0, 1, 2, 5, 8])
        weights = [rng.randint(0, 9) for _ in range(length)]
        if length and rng.random() < 0.34:
            weights[rng.randrange(length)] = 0
        out.append((amount, weights))
    return out


def check_mr01(fn: Alloc, amount: int, weights: list[int]) -> str | None:
    if not weights:
        return None
    got = fn(amount, list(weights))
    if sum(got) != amount:
        return f"sum={sum(got)} amount={amount} out={got}"
    if len(got) != len(weights):
        return f"len {len(got)} != {len(weights)}"
    return None


def check_mr02(fn: Alloc, amount: int, weights: list[int], rng: random.Random) -> str | None:
    if len(weights) < 2:
        return None
    order = list(range(len(weights)))
    rng.shuffle(order)
    base = fn(amount, list(weights))
    shuffled = [weights[i] for i in order]
    got = fn(amount, shuffled)
    expected = [base[i] for i in order]
    if got != expected:
        return f"P={order} base={base} got={got} expected={expected}"
    return None


def check_mr03(fn: Alloc, amount: int, weights: list[int], rng: random.Random) -> str | None:
    i = rng.randrange(len(weights) + 1)
    base = fn(amount, list(weights)) if weights or amount == 0 else None
    if base is None:
        return None
    inserted = weights[:i] + [0] + weights[i:]
    got = fn(amount, inserted)
    expected = base[:i] + [0] + base[i:]
    if got != expected:
        return f"i={i} base={base} got={got}"
    return None


CHECKS = {
    "MR01": lambda fn, a, w, rng: check_mr01(fn, a, w),
    "MR02": check_mr02,
    "MR03": check_mr03,
}

BLOCKING = ("MR01", "MR02", "MR03")


def run_relations(fn: Alloc, seed: int = 20260917) -> list[Break]:
    draws = sample(seed)
    rng = random.Random(seed ^ 0x9E3779B9)
    breaks: list[Break] = []
    for idx, (amount, weights) in enumerate(draws):
        for rid in BLOCKING:
            try:
                detail = CHECKS[rid](fn, amount, weights, rng)
            except Exception as exc:
                detail = f"exception {exc!r}\n{traceback.format_exc()}"
            if detail:
                breaks.append(Break(rid, idx, detail))
    return breaks

A thin runner prints a machine-readable summary. Keep this file next to the table, not next to the agent's scratch tests.

# proposed: python run_relations.py
if __name__ == "__main__":
    from allocate import allocate  # the patched module

    breaks = run_relations(allocate)
    by_id: dict[str, int] = {}
    for row in breaks:
        by_id[row.relation_id] = by_id.get(row.relation_id, 0) + 1
        print(f"FAIL {row.relation_id} draw={row.draw_index} {row.detail}")
    print(f"SUMMARY seed=20260917 blocking={len(breaks)} by_id={by_id}")
    raise SystemExit(1 if breaks else 0)

Command shape:

python run_relations.py
echo $?

A non-zero exit is a broken blocking row. Do not wrap this in a retry loop. Retry hides hysteresis, which is the next section.

Six-step merge workflow

1. Freeze the table outside the writable patch tree. A read-only path, a second repo, or a remote job definition all work. The file format is not the point. Edit rights are the point. If the agent can add if weights == fixture: return fixture_out to both the code and the table, the gate is theatre.

2. Record the sampler seed in the job log. Re-run must rebuild the same pair set. Store seed, git sha of the table, and git sha of the harness. Do not index failures by test function name alone.

3. Produce the candidate patch with a model that is cheap to rerun. The gate is the relation table, not the model's reputation. One forward pass that yields a diff is enough to exercise the workflow.

4. Execute the harness on a machine the agent session cannot write. Local pytest in the same workspace is a convenience. It is not an independent witness. Copy run_relations.py, the table, and the patched module into a clean tree before the run.

5. Classify each broken row. Use three buckets only: block, policy-sensitive skip, and hysteresis. Hysteresis means the pair fails only after another pair ran in-process. That is leaked mutable state, not a timing flake.

6. Merge only when the blocking set is empty on a clean process per pair. Skips stay in the table with an owner and a date. They do not disappear. A skip without an owner is a deleted invariant with extra words.

Pair isolation, not a flake freeze

Unit-test flakes get frozen, renamed, or skipped. Those tactics do not transfer cleanly to pairwise oracles. Metamorphic pairs fail for a different reason: the two calls share a process.

A cache on allocate, a class-level remainder pointer, or an in-place shuffle of weights will make MR02 depend on whatever MR01 just did. The failure looks intermittent if the draw order changes with the seed. It is deterministic given the seed and a shared heap.

Proposed isolation: one subprocess per pair, or at least one fresh object graph per pair. The cheap version is:

# proposed: isolate by copying inputs and forbidding in-place mutation checks
import copy
import subprocess
import sys

def pair_in_subprocess(relation_id: str, seed: int, draw_index: int) -> int:
    code = (
        "from run_relations import sample, CHECKS; "
        "from allocate import allocate; "
        "import random; "
        f"amount, weights = sample({seed})[{draw_index}]; "
        f"rng = random.Random({seed} ^ 0x9E3779B9); "
        f"detail = CHECKS[{relation_id!r}](allocate, amount, list(weights), rng); "
        "raise SystemExit(1 if detail else 0)"
    )
    proc = subprocess.run([sys.executable, "-c", code], check=False)
    return proc.returncode

If a relation still flickers after isolation, the function is reading a clock, a network, or an unordered set walk. Those sources do not belong in allocate. Do not freeze the relation. Remove the entropy or make it an explicit argument.

Copying inputs matters even inside one process. If allocate mutates weights.sort(), MR02 compares against a permutation of an already sorted list. The proposed harness passes list(weights) on every call for that reason.

Independent witness, free model, free server

Generating a patch is the cheap step. Witnessing it is the expensive one. The split only holds when the witness tree is not the editor tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option map onto steps 3 and 4 and nowhere else. The model proposes a diff against allocate. The server runs run_relations.py from a checkout the agent session does not write. That is the method. It is not a ranking of models, a quota, a hardware claim, or a promise that the option stays unchanged.

If the relation table and the implementation can be edited in one session, the pairwise oracle collapses back into a value assert with extra ceremony. Keep them apart. A copy of the harness runs anywhere CPython runs; a separate server is useful only when the agent's workspace must not be the witness.

Limitations, and who should skip this

This method does not replace tests for exceptions beyond MR06, HTTP mappings, or UI. Metamorphic rows say nothing about a wrong-but-consistent formula. If both executions use the same incorrect interest rate, MR02 still passes. Pairwise oracles catch policy drift and hardcoded fixtures. They do not conjure a specification that nobody wrote.

Who should not use this approach:

• Teams that cannot name even one relation for the function under change. An empty table is a green gate that rejects nothing.
• Patches that are comments, docs, or CSS. There is no pair of executions to relate.
• Safety-critical numeric code that needs a bit-exact reference. Use a known-answer suite maintained by a human. A pairwise table is an extra net, not a substitute for known answers.
• Functions whose spec is "match yesterday's snapshot." Characterization snapshots are a different tool. Mixing them with metamorphic rows hides which gate failed.

The table also does not estimate model quality. A green blocking set means the documented pairings still hold. It does not mean the product is correct, fast, or done.

Remainder policy is the usual hole. Until MR04 and MR05 move from skip to block, two remainder strategies can both look healthy. Write the leftover-cent rule in the same table before treating scale as blocking. The row is already there so that the decision is visible.

What to merge on

Merge when conservation, permutation, and zero-insertion hold on an isolated process, with the sampler seed recorded. Treat scale and duplication as policy rows until remainder behavior sits in the table in one sentence. Keep the table longer than the patch series.

The patch is disposable. The relations are not.