Don't Golden-File an Agent Patch. Golden-File the Relation.

A recorded expected value is a leak. An agent that can read assert f(x) == y can patch f until that line is green and leave every unlisted input broken. A metamorphic relation does not publish y. It only publishes a constraint the output must keep under a known transform. That is the gate worth automating. Fixtures still matter, but only as seeds. Flaky tests still need a freeze, but the freeze must not cover the relation itself.

This article is a proposed layout, not a production case study. No runtime metrics are claimed. The commands and modules below are labeled so they can be copied into a scratch repo and executed against your own function under test.

Why snapshots fail as a merge gate

Golden files encode one transcript. An agent patch is a search over many transcripts. If the search can see the answer key, the cheapest passing program is a lookup table for the keys in tree. That program is green. It is also wrong on the next customer file.

Property-style checks reduce that leak because they do not ship the answer. They still need a seed corpus, a replay runner that the patch cannot edit, and a quarantine file that expires. Mix those three and you get a gate that fails closed when the agent rewrites tests, when a fixture drifts, or when a flake is used to hide a broken invariant.

Three relation classes worth encoding first

Start with relations you can state in one line. If you cannot state the line, you do not have a gate. You have a recorder.

1. Idempotence. f(f(x)) == f(x) for normalizers, formatters, and canonicalizers.
2. Round-trip. parse(serialize(x)) equals x on the fields you actually guarantee, not on whitespace you do not.
3. Oracle-free comparison. f(t(x)) relates to t(f(x)) for a transform t you control: shuffle independent rows, rename equivalent keys, NFC vs NFD unicode, scale a quantity and its unit together.

These are not universal laws. They are hypotheses about your function. Write them down as code. Keep the seed inputs boring. The relation, not the seed, does the work.

Seed fixtures are not expected outputs

A seed is an input the runner is allowed to read. It is not a blessed byte dump of f(seed). Store seeds as files with no .expected sibling. If a reviewer adds an expected file next to a seed, the gate should fail the build. That single rule stops the suite from turning back into a recorder.

Proposed tree:

repo/
  src/normalize_csv.py
  oracle/                 # verifier-owned, not in the agent's write path
    seeds/
      invoices_small.csv
      invoices_utf8.csv
    relations/
      test_relations.py
    quarantine.yaml
    conftest.py
  Makefile

The agent may write src/. It must not write oracle/. Enforce that in CI with a path check, not with a comment.

Artifact: a replay runner the patch cannot satisfy by editing tests

The module under test is a CSV amount normalizer. Treat the following as a proposed example, not as measured production code.

# src/normalize_csv.py
import csv
import io
from decimal import Decimal, ROUND_HALF_EVEN

AMOUNT_KEYS = {"amount", "total", "price"}

def normalize_csv(text: str) -> str:
    reader = csv.DictReader(io.StringIO(text))
    if reader.fieldnames is None:
        raise ValueError("missing header")
    fields = list(reader.fieldnames)
    out = io.StringIO()
    writer = csv.DictWriter(out, fieldnames=fields, lineterminator="\n")
    writer.writeheader()
    for row in reader:
        for key in fields:
            value = row.get(key, "")
            if key.lower() in AMOUNT_KEYS and value.strip() != "":
                quant = Decimal(value).quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
                row[key] = format(quant, "f")
            else:
                row[key] = value.strip()
        writer.writerow(row)
    return out.getvalue()

Relation tests consume seeds and never mention a golden output string.

# oracle/relations/test_relations.py
from pathlib import Path
import csv
import io
import unicodedata
import pytest

from normalize_csv import normalize_csv

SEEDS = Path(__file__).resolve().parents[1] / "seeds"

def load_seeds():
    files = sorted(SEEDS.glob("*.csv"))
    if not files:
        raise AssertionError("oracle/seeds is empty; the gate must fail closed")
    return files

@pytest.mark.relation
@pytest.mark.parametrize("seed", load_seeds(), ids=lambda p: p.name)
def test_idempotent(seed: Path):
    raw = seed.read_text(encoding="utf-8")
    once = normalize_csv(raw)
    twice = normalize_csv(once)
    assert once == twice

@pytest.mark.relation
@pytest.mark.parametrize("seed", load_seeds(), ids=lambda p: p.name)
def test_row_permutation_commutes(seed: Path):
    raw = seed.read_text(encoding="utf-8")
    rows = raw.splitlines()
    header, body = rows[0], rows[1:]
    if len(body) < 2:
        pytest.skip("need two data rows to permute")
    permuted = "\n".join([header, body[1], body[0], *body[2:]]) + "\n"
    left = _rows(normalize_csv(raw))
    right = _rows(normalize_csv(permuted))
    assert sorted(left) == sorted(right)

@pytest.mark.relation
@pytest.mark.parametrize("seed", load_seeds(), ids=lambda p: p.name)
def test_unicode_normalization_commutes(seed: Path):
    raw = seed.read_text(encoding="utf-8")
    nfd = unicodedata.normalize("NFD", raw)
    left = normalize_csv(raw)
    right = normalize_csv(nfd)
    assert unicodedata.normalize("NFC", left) == unicodedata.normalize("NFC", right)

def _rows(csv_text: str):
    return list(csv.DictReader(io.StringIO(csv_text)))

conftest.py should refuse two things: an empty seed directory, and any patch that touches oracle/.

# oracle/conftest.py
from pathlib import Path
import os
import subprocess
import pytest

ORACLE = Path(__file__).resolve().parent

def test_oracle_tree_is_unchanged_in_the_patch():
    base = os.environ.get("ORACLE_BASE_REF", "origin/main")
    diff = subprocess.check_output(
        ["git", "diff", "--name-only", base, "--", "oracle"],
        text=True,
    ).strip()
    assert diff == "", f"oracle path changed in patch:\n{diff}"

If your agent workflow does not produce a git diff against main, replace the subprocess with a permission check: the verify job runs as a user that has read-only access to oracle/ and read-write access only to a scratch out/ directory.

Freeze flakes. Do not freeze relations.

Network calls, clock reads, and unordered log lines flake. Idempotence of a pure normalizer does not. A quarantine file is useful. It is also the easiest place to hide a regression if you let it list relation test ids.

Proposed schema:

# oracle/quarantine.yaml
# status values: open | frozen
# frozen rows MUST set expires_on (ISO date) and a non-relation nodeid
version: 1
rules:
  - nodeid: "oracle/test_http.py::test_rates_endpoint"
    status: frozen
    reason: "upstream rate table serves 429 without Retry-After"
    expires_on: "2026-09-17"
    owner: "payments-api"

Loader rules, proposed and strict:

# oracle/relations/test_quarantine_policy.py
from datetime import date
from pathlib import Path
import yaml

FORBIDDEN_PREFIXES = ("oracle/relations/",)

def test_quarantine_cannot_cover_relations():
    payload = yaml.safe_load(Path("oracle/quarantine.yaml").read_text())
    today = date.today().isoformat()
    for row in payload["rules"]:
        nodeid = row["nodeid"]
        assert not nodeid.startswith(FORBIDDEN_PREFIXES), nodeid
        if row["status"] == "frozen":
            assert row["expires_on"] >= today, row
            assert "reason" in row and len(row["reason"]) >= 12

A freeze without an expiry is a deleted test. CI should treat an expired row as a failing test, not as a skip. Re-open the row or fix the environment. Do not extend expires_on from the agent workspace.

Numbered workflow

Run generation and verification as two jobs with two disks. The patch is a candidate. The relation suite is the judge.

1. Lock the oracle. Copy oracle/ from main onto the verifier. Mark it read-only. Do not check the agent's worktree out on top of it.
2. Seed, do not record. Add the smallest CSV files that exercise headers, UTF-8, empty amounts, and two-row permutation. Commit them on main before the agent runs.
3. Generate the patch off the verifier. Point the coding agent at a checkout that contains src/ and a task prompt, not oracle/relations/.
4. Apply the patch to a throwaway tree. git apply --check then git apply. If the diff lists any path under oracle/, reject before tests run.
5. Replay relations. pytest -m relation oracle/relations -q. Fail on first broken relation. Do not retry away an idempotence failure.
6. Apply quarantine last. Only nodeids outside oracle/relations/ may be skipped, and only while expires_on is in the future.
7. Refuse empty greens. If load_seeds() would raise, or if pytest collected zero relation tests, the job fails. A suite that collected nothing did not pass.

Makefile sketch:

ORACLE_BASE_REF ?= origin/main

verify:
    ORACLE_BASE_REF=$(ORACLE_BASE_REF) pytest -m relation oracle -q
    pytest oracle/relations/test_quarantine_policy.py -q

paths:
    git diff --name-only $(ORACLE_BASE_REF) -- oracle; test $$? -eq 0

The verify target is what merges. Generation is not in this file. That split is the point.

Where a free remote coding environment fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Generation is the expensive, noisy step. Verification is the strict, boring step. MonkeyCode's free model access and free server option are relevant when you want the noisy step off the machine that holds oracle/. The verifier stays local, or on a CI runner whose credentials cannot push into oracle/. The generator can be a separate workspace whose output is a diff, not a merged tree.

Do not ship the relation file to the generator to "help it pass." That recreates the golden-file leak with extra latency. If the task needs examples, give input/output shapes in the prompt (amount becomes two decimal places) and keep the seed bytes on the verifier.

This layout does not depend on a named model, a quota, or a hardware profile. Those are not specified here. Any isolated workspace that returns a patch and cannot write oracle/ is enough.

Decision table for a red job

Observation	Likely cause	Action
Relation test_idempotent fails	Patch is not a normalizer	Reject patch
Relation test_row_permutation_commutes fails	Hidden row order dependency	Reject patch
oracle/ appears in git diff	Agent edited the gate	Reject patch, do not rerun
Zero tests collected	Seeds missing or markers dropped	Fail closed
HTTP test flakes, relation suite green	Environment, not invariant	Freeze HTTP nodeid with expiry
Quarantine row lists oracle/relations/	Freeze used as a hide	Fail closed
Quarantine expires_on in the past	Forgotten skip	Fail until row is removed or the test is fixed

The table is a triage aid. It is not a scoring model with weights. Weights without measurements would be fiction.

Limitations

A weak relation is a weak gate. f(x) is not None will pass almost every patch. Idempotence will not catch a function that truncates instead of rounding if truncation is also idempotent. Round-trip will not catch silent field drops if you compare only surviving keys. You still need review for security, licensing, and performance. Relations do not bound runtime or memory.

Unicode and CSV dialects differ across Python builds and lib versions. Pin the interpreter in CI. If a seed depends on locale, it is no longer a seed; it is an environment test and belongs outside oracle/relations/.

The path-isolation trick assumes you control CI. A local developer who runs the agent in the full repo can still edit oracle/ by hand. The git diff check is the backstop, not the filesystem.

Who should not use this

Do not use this layout if you cannot name one relation that must hold after the patch. Do not use it as a substitute for code review on auth, crypto, or migrations. Do not use it to generate tests with the same agent that generates the patch and then call the result independent. Do not freeze relation nodeids to buy a green dashboard. Teams that only have UI screenshot diffs will not get a metamorphic suite for free; visual oracles are a different artifact.

If those constraints do not apply, keep the judge on a disk the patch never mounts. A free remote coding environment is useful only when it returns a candidate and leaves the relation suite where it was.