When Agents Can Rewrite Expected Values, Test the Relation

Agent patches fail a common class of tests without failing the product. The suite still reports green. The expected value moved with the code. A merge gate that stores answers in the same tree the agent can edit is not a gate. It is a mirror.

The durable alternative is a metamorphic relation: a rule that ties two executions together without naming the correct output. The agent can rewrite helpers, comments, and even nearby tests. It cannot satisfy the relation by editing a literal. That is the conclusion this article starts from, and the rest is a workflow to make it checkable.

Why golden files collapse under patch agents

A golden file is a stored answer. An assertion like assert actual == 42 is the same idea with less ceremony. Both are cheap to author. Both are cheap to “fix.”

An agent that is scored on a passing suite has a direct incentive to update the stored answer. The edit is local. The diff looks tidy. Reviewers who skim tests after source often miss it. The failure is not that the agent is malicious. The failure is that the test encoded an output instead of a constraint.

This is not the same problem as a tautology. A tautology is a test that cannot fail. A rewritten golden file can still fail later, on a different input. It has simply stopped checking the change that just landed. The suite measures agreement with itself.

What a relation actually asserts

A metamorphic relation names a transform on the input and a predicate on the pair of outputs. You never store f(x). You store R(f(x), f(T(x))).

Three relations cover a surprising number of agent patches:

1. Invariance. f(T(x)) == f(x) when T should not matter. Permuting an unordered collection. Re-encoding equivalent JSON. Adding zero to a money amount.
2. Monotonicity. If x qualifies at threshold p, it still qualifies at p + k. Rollout percentages, rate-limit burst sizes, retry budgets that must not shrink silently.
3. Round-trip. decode(encode(x)) == x or encode(decode(y)) == y on the documented subset. Serializers and ID codecs are frequent agent targets.

Absolute oracles still matter for hard bounds: percent == 0 admits nobody. Relations do not replace those checks. They cover the middle, where agents invent plausible constants.

Keep the relation off the agent’s disk

Isolation is the other half. If the relation file lives in the same checkout the agent can git add, the agent can weaken R. The relation suite needs a runner the agent cannot mount and a path the agent cannot write.

A free server option is enough for that split when the suite is small and has no secrets. MonkeyCode’s free model access is useful only as an input generator, not as an expected-value generator. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model may propose extra user_id strings, extra JSON blobs, extra offset/limit pairs. It must not propose True or "42". Expected values are how golden files come back through the side door.

The proposed layout:

repo/                 # agent may write here
  src/
  tests/unit/         # fast tests; still treat expected literals as untrusted
oracle/               # not mounted into the agent workspace
  relations.py
  morph_inputs.jsonl  # optional, generated offline
  test_relations.py

The oracle tree is cloned onto the free server, or onto any CI job whose workspace the agent never sees. Same mechanism. Different hosting.

Proposed harness: monotonic rollout

The example is a percentage rollout. Agents often “simplify” the hash, switch to random, or change string concatenation order. A stored boolean will not catch a hash change that still looks stable on one fixture. A relation will.

Label: this is a proposed, unexecuted harness. It is not a benchmark and not a claim about any model’s accuracy.

# oracle/relations.py
from __future__ import annotations

import hashlib
from dataclasses import dataclass
from typing import Callable


def in_rollout(user_id: str, percent: int, salt: str) -> bool:
    if percent <= 0:
        return False
    if percent >= 100:
        return True
    material = f"{salt}:{user_id}".encode("utf-8")
    bucket = int(hashlib.sha256(material).hexdigest(), 16) % 100
    return bucket < percent


@dataclass(frozen=True)
class Morph:
    name: str
    transform: Callable[[dict], dict]
    holds: Callable[[bool, bool], bool]


RELATIONS = [
    Morph(
        name="monotonic_percent",
        transform=lambda c: {**c, "percent": min(100, c["percent"] + 10)},
        holds=lambda a, b: (not a) or b,
    ),
    Morph(
        name="salt_is_material",
        transform=lambda c: {**c, "salt": c["salt"] + ":x"},
        holds=lambda a, b: True,  # pair collected for logging; asserted elsewhere
    ),
    Morph(
        name="zero_admits_none",
        transform=lambda c: {**c, "percent": 0},
        holds=lambda a, b: b is False,
    ),
]


def eval_pair(fn, case: dict, morph: Morph) -> tuple[bool, bool]:
    before = fn(case["user_id"], case["percent"], case["salt"])
    nxt = morph.transform(case)
    after = fn(nxt["user_id"], nxt["percent"], nxt["salt"])
    return before, after

# oracle/test_relations.py
import json
from pathlib import Path

import pytest

from relations import RELATIONS, eval_pair, in_rollout

CASES = [
    {"user_id": "u-17", "percent": 10, "salt": "exp-a"},
    {"user_id": "u-17", "percent": 50, "salt": "exp-a"},
    {"user_id": "u-99", "percent": 1, "salt": "exp-b"},
]


def load_extra_inputs() -> list[dict]:
    p = Path(__file__).with_name("morph_inputs.jsonl")
    if not p.exists():
        return []
    rows = []
    for line in p.read_text().splitlines():
        row = json.loads(line)
        if set(row) != {"user_id", "percent", "salt"}:
            raise ValueError("input generator emitted a non-input key")
        if not isinstance(row["user_id"], str):
            raise ValueError("user_id must be a string")
        if not (0 <= int(row["percent"]) <= 100):
            raise ValueError("percent out of bounds")
        rows.append(row)
    return rows


@pytest.mark.parametrize("case", CASES + load_extra_inputs())
@pytest.mark.parametrize("morph", RELATIONS, ids=lambda m: m.name)
def test_relation(case, morph):
    before, after = eval_pair(in_rollout, case, morph)
    assert morph.holds(before, after), (morph.name, case, before, after)


def test_hard_bounds():
    assert in_rollout("any", 0, "s") is False
    assert in_rollout("any", 100, "s") is True

The extra-input loader rejects keys other than the input schema. That is the entire defense against a model that tries to smuggle expected into the file. If a line cannot be parsed as an input, the job fails closed.

Numbered workflow

1. Write the relation first. One sentence: “If a user is in at 10%, they remain in at 20%.” If you cannot write the sentence, you do not have a relation. Stop. Do not generate tests.
2. Pin hard bounds as literals. Zero and one hundred, empty and full, None rejected. These are not goldens of business output. They are domain edges.
3. Collect a tiny seed corpus by hand. Three to ten cases. Humans choose awkward IDs: empty-ish strings, Unicode, leading zeros.
4. Optionally expand inputs with a free model. Prompt for inputs only. Validate the schema on disk before pytest sees the file. Drop the run if any line includes an output field.
5. Execute on an unreachable runner. A free server, a separate CI job, or a read-only checkout. The agent’s patch job must not have write access to oracle/.
6. Fail the merge on a broken relation. Do not auto-update the relation to match the patch. If the product intent changed, a human rewrites holds.
7. Record the relation identity, not the test name. Hash name + transform source + holds source. Renaming a test must not reset the gate. Changing holds must.

Commands for the unreachable runner:

python -m pip install pytest
python -c "import json,sys; [json.loads(l) for l in open('oracle/morph_inputs.jsonl')]" 2>/dev/null || true
pytest oracle/test_relations.py -q

If you generate extra inputs, keep the generator on a short leash:

Emit JSONL. Each line is an object with keys user_id (string), percent (int 0-100), salt (string).
Do not emit expected results, comments, or extra keys.

Treat that prompt as a filter, not as an authority. The schema check in load_extra_inputs is the authority.

Decision table

Check type	Stores an answer?	Survives agent edit of tests?	Use when
Golden file / snapshot	Yes	No	Human-owned, rare, binary artifacts
Literal bound	Yes, but only at a domain edge	Partially	0/100, empty, overflow
Property on one output	No	Only if the property file is unreachable	Invariants you can state on f(x) alone
Metamorphic relation	No	Only if R is unreachable	Transforms you can state without f(x)
Model-written expected value	Yes	No	Never, for merge gates

Read the third column as a permission question, not a philosophy question. A relation in the agent workspace is a golden file with extra steps.

What this does not catch

Relations miss bugs that preserve the relation. A rollout that is off by a constant bucket still passes monotonicity if the error is consistent. A tax function that doubles every amount still passes a scale relation if you only scale inputs. Pair relations with at least one absolute bound.

They also miss intent changes. If product now wants a non-monotonic rollout, the relation should fail until a human deletes it. That failure is the point. Do not teach the agent to “repair” relations.

Flaky relations are usually underspecified transforms: floating-point rounding, clock input, unordered hashes dumped to a string. Freeze those by removing time and randomness from f, not by skipping the test. A skip list the agent can append to is another editable golden file.

Who should not use this

Do not use this workflow if you cannot state a relation in one sentence. Do not use a free model to draft holds. Do not treat a free server as a security boundary if the agent’s workspace already contains deploy credentials for that server. Do not apply metamorphic gates to visual snapshot work, to one-off scripts, or to patches that only touch comments.

Safety-critical code with a certified oracle should keep the certified oracle. A relation is a supplement, not a replacement for a known-answer test that a regulator already named.

Teams that already run properties on a job the agent cannot see do not need a new product to do this. Any unreachable runner works. If you do not have one, MonkeyCode’s free server option is a place to park the oracle tree; the method does not depend on it.

The merge question is narrow. Did the patch preserve the relations you wrote down, on inputs the agent did not get to label? If the suite cannot answer that without reading an expected literal the agent can edit, the suite is not testing the patch. It is testing the agent’s willingness to keep the files consistent.