Metamorphic Checks for Agent Patches When You Have No Trusted Oracle

Agent patches fail in a specific way. They match a snapshot you already had, then break a relation you never wrote down. Golden files do not catch that class of defect. Metamorphic checks do.

This article is a test plan, not a product tour. It assumes a small Python service that an agent is allowed to patch, and a budget that cannot rerun the full suite on every draft. The procedure below is a proposed workflow. It has not been presented as a production measurement.

The failure mode snapshots miss

A golden file records one output for one input. That is useful when the oracle is stable and the input set is tiny. Agent patches violate both conditions. The model often rewrites formatting, key order, timestamps, and error wrapping while leaving the “happy” fixture green.

The second problem is circular tests. Agents write assertions that restate the new code. Those tests pass because they were generated from the same patch. They do not encode an independent fact about the domain.

A metamorphic check ignores the absolute value. It asserts a relation that must hold across transformations: round-trip, permutation, idempotence, monotonicity, or error preservation. If the relation is true before the patch, it must stay true after. That is the gate.

Decision table: snapshot, contract, or relation

Use this table before you let an agent touch a module. Pick one primary check per function. Mixing all three on every path wastes the free-server budget and hides signal.

Surface	Prefer	Reject as primary	Why
Pure codec (parse/dump)	Round-trip relation	Exact byte snapshot	Key order and whitespace drift
Sort, hash, set algebra	Permutation / idempotence	Hard-coded list literals	Agents copy the current order
HTTP JSON handlers	Status × schema matrix	Full response body files	Timestamps and request IDs
Time-window reports	Frozen clock + monotonicity	Live datetime.now() asserts	Flakes look like product bugs
External I/O adapters	Recorded transport fake	Live network calls	Agents “fix” tests by skipping I/O
UI copy / changelog text	Snapshot (narrow)	Metamorphic generators	No stable relation exists

If a cell says “reject as primary,” you may still keep one snapshot as a characterization test. Do not promote it to the merge gate.

Artifact: a relation catalog the agent cannot rewrite

Keep relations in a file the patch job cannot write. Treat it as test data, not as code the model owns. The catalog below is labeled as an example. Adapt the function names to your module.

# tests/relations_catalog.py
# Example catalog. Review by a human before it becomes a gate.

RELATIONS = [
    {
        "id": "json_roundtrip_v1",
        "fn": "codec.dump",
        "kind": "roundtrip",
        "seeds": "tests/seeds/json_objects.jsonl",
        "assert": "parse(dump(x)) == canonicalize(x)",
    },
    {
        "id": "sort_perm_v1",
        "fn": "index.sort_keys",
        "kind": "permutation",
        "seeds": "tests/seeds/key_lists.jsonl",
        "assert": "sort(p(x)) == sort(x) for any permutation p",
    },
    {
        "id": "retry_idem_v1",
        "fn": "jobs.mark_done",
        "kind": "idempotence",
        "seeds": "tests/seeds/job_ids.jsonl",
        "assert": "f(f(x)) == f(x)",
    },
    {
        "id": "window_mono_v1",
        "fn": "report.bucket",
        "kind": "monotonicity",
        "seeds": "tests/seeds/timestamps.jsonl",
        "assert": "t1 <= t2 implies bucket(t1) <= bucket(t2)",
    },
]

Seeds are JSON Lines, one value per row. They are cheaper to review than generated tests, and they stay small enough to run on a constrained box.

# tests/seeds/json_objects.jsonl
{"n": 0, "tags": []}
{"n": 1, "tags": ["a"]}
{"n": -7, "tags": ["a", "a"]}
{"nested": {"ok": true}, "tags": ["x"]}

Implementation: one runner, four relation kinds

The runner is ordinary pytest. It loads the catalog, freezes time, and refuses to import production datetime.now inside the subject under test. Clock drift is the usual source of “green then red” agent patches.

# tests/test_metamorphic_gate.py
from __future__ import annotations

import json
from datetime import datetime, timezone
from itertools import permutations
from pathlib import Path

import pytest

from app import codec, index, jobs, report
from tests.relations_catalog import RELATIONS

FROZEN = datetime(2026, 9, 6, 12, 0, tzinfo=timezone.utc)


def load_seeds(path: str):
    rows = []
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if line:
            rows.append(json.loads(line))
    if not rows:
        raise AssertionError(f"empty seed file: {path}")
    return rows


def canonicalize(obj):
    if isinstance(obj, dict):
        return {k: canonicalize(obj[k]) for k in sorted(obj)}
    if isinstance(obj, list):
        return [canonicalize(v) for v in obj]
    return obj


@pytest.fixture
def frozen_clock(monkeypatch):
    class Clock:
        @staticmethod
        def now(tz=None):
            return FROZEN if tz is None else FROZEN.astimezone(tz)

    monkeypatch.setattr(report, "clock", Clock)
    return Clock


@pytest.mark.parametrize("rel", RELATIONS, ids=lambda r: r["id"])
def test_relation(rel, frozen_clock):
    seeds = load_seeds(rel["seeds"])
    kind = rel["kind"]
    if kind == "roundtrip":
        for x in seeds:
            got = codec.parse(codec.dump(x))
            assert canonicalize(got) == canonicalize(x), rel["id"]
    elif kind == "permutation":
        for x in seeds:
            base = index.sort_keys(x)
            for p in list(permutations(x))[:6]:
                assert index.sort_keys(list(p)) == base, rel["id"]
    elif kind == "idempotence":
        for x in seeds:
            once = jobs.mark_done(x)
            twice = jobs.mark_done(once)
            assert twice == once, rel["id"]
    elif kind == "monotonicity":
        times = [datetime.fromisoformat(s["ts"]) for s in seeds]
        times.sort()
        buckets = [report.bucket(t) for t in times]
        assert buckets == sorted(buckets), rel["id"]
    else:
        raise AssertionError(f"unknown kind {kind}")

Cap permutations. A full n! sweep is not a test plan. Six shuffles per seed is enough to catch an agent that sorts only the first two keys.

Procedure

1. Inventory call sites the agent may edit. List functions, not files. One relation per function is the default. Two is a smell: the function is doing two jobs.
2. Write seeds by hand for the first 8–12 rows. Include empty containers, duplicates, negative numbers, and nested objects. Do not ask a model to invent the first seed file. That is how you encode its assumptions as ground truth.
3. Lock the catalog path as read-only in the patch job. If the agent “fixes” a failing relation by editing RELATIONS, the gate did not run. A chmod a-w tests/relations_catalog.py tests/seeds step before the model starts is enough on a single-user box.
4. Freeze clocks and transports in the same process. Do not freeze flaky tests. Freeze the inputs that made them flaky. A failing relation after a freeze is a product bug, not a quarantine candidate.
5. Run the catalog as a separate pytest node. Keep it off the unit-test path the agent is allowed to green by adding stubs.
6. Promote a relation only after it fails a known-bad patch. If you cannot break it with a one-line mutation (drop a sort, change <= to <, skip empty lists), it is not a gate yet.

Commands for the isolated node:

chmod a-w tests/relations_catalog.py tests/seeds
python -m pytest tests/test_metamorphic_gate.py -q --tb=short

To check that the gate actually bites, apply a throwaway mutation and expect a red run:

# labeled example: do not commit
sed -i 's/sorted(obj)/obj/' app/codec.py
python -m pytest tests/test_metamorphic_gate.py -q
# restore
git checkout -- app/codec.py

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

After the first human-written seeds exist, a free model can propose additional seed rows and candidate relation kinds from a function signature. Treat those proposals as untrusted. A reviewer accepts or deletes them. The model does not edit the catalog file.

A free server is the right place to run the seed sweep when it grows past what a laptop should block on. The gate is a batch job: catalog in, pass/fail out. It does not need a GPU. It does need a write-protected tree and a frozen clock. MonkeyCode’s free model access and free server option are relevant only as that draft-and-sweep pair. They do not replace the catalog or the human accept step.

Prompt the model with the relation kind, not with “write tests for this patch.” Example instruction, labeled as a prompt draft:

Function: codec.parse / codec.dump
Known relation: parse(dump(x)) == canonicalize(x)
Do not write pytest.
Propose 10 JSONL seed rows that are not in the attached file.
Each row must be valid JSON on one line.
Include at least one empty map, one duplicate list item, and one nested map.

If the model returns pytest classes, discard the output. The catalog format is the contract.

What this does not cover

Metamorphic checks do not prove functional correctness against a business spec. A round-trip can hold for a codec that silently drops unknown fields. Add a schema matrix if field preservation matters.

They also do not replace contract tests at HTTP boundaries. Status codes and required keys stay as a table. Relations sit behind that table, on the pure core.

Do not use this plan when the output is editorial (copy, emails, docs) or when the only oracle is a designer. There is no honest relation for tone. Do not use it as a reason to skip code review on security-sensitive parsers. A green round-trip is not a threat model.

Seed files go stale. Revisit them when the domain adds a new type, not on a calendar. If a relation has not failed a mutation in the last few accepted patches, shrink it or delete it. A gate that never fires is documentation, not a test.

Limits of the compute story

A free server does not make an unbounded generator cheap. Cap seeds. Cap permutations. Fail the job if a relation runs longer than a fixed wall clock you choose for that box. The number is local policy, not a product claim.

If two relations share seeds, do not duplicate the JSONL. Point both catalog rows at one file. Duplicated seeds drift, and agents “fix” only one copy.

Close

Write down the relation first. Then let an agent patch the implementation. If the only tests you have are snapshots the model can clone, you are grading the patch against itself. The catalog above is small enough to review in one sitting and strict enough to catch dropped sorts, broken round-trips, and clock-shaped flakes that were never flakes.

If you already draft patches against a free-model endpoint, keep the catalog on the same free server and run tests/test_metamorphic_gate.py as a second job. That is the entire product note. The gate still works if you run it on a laptop.