Lock the Test Surface, Then Prove the Agent Patch With Relations

An agent patch that exits zero is not evidence. It is a claim that the suite still means what it meant before the diff. Two failure modes dominate. The patch leaves tests alone and changes behavior the assertions never named. Or it edits the assertions until the suite is quiet. A gate that only reads a process code accepts both. The cheaper fix is not another wall of unit checks. It is a locked test surface, a metamorphic relation suite, and a freeze ledger keyed by input pairs rather than by test names.

This article is a proposed merge workflow, not a measured production study. Examples below are labeled as such. Swap the language and the runner for your stack. Keep the three checks in order.

The claim a green suite does not make

Unit assertions pin outputs to literals. Agent patches are good at satisfying literals. They are weaker at preserving meaning across transformed inputs. If f(x) is accepted, f(permute(x)) should still obey the same relation unless the spec says otherwise. If f(a) and f(b) are accepted, f(a ∪ b) should not invent a third class of result. Those are relations. They survive a rewrite that still happens to match one golden file.

A second gap is quieter. The patch changes the test file. The CI graph stays green because the graph now asks a smaller question. Counting passed tests after that edit is circular. You measured the new question.

Three checks, in order

Run them as separate jobs. Do not fold them into one pytest session that an agent can retarget with a single config edit.

1. Lock the assertion surface of the test tree.
2. Execute metamorphic pairs against the patched code, with the unpatched tree as the control.
3. Freeze only relation pairs that fail under reordering and pass under a deterministic subset. Expire the freeze.

If check 1 fails, stop. Do not spend a model call arguing about flakes. The suite changed its own question.

Check 1: assertion-surface lock

Hash the lines that can change the question. Ignore comments, docstrings, and blank lines. Do not ignore assert, raises, approx, fixture names that feed those asserts, or the signatures of test functions. A docstring edit should not block merge. An assertion deletion should.

Proposed lock script:

# proposed harness: tools/surface_lock.py
from __future__ import annotations

import ast, hashlib, json, sys
from pathlib import Path

ASSERT_ATTRS = {"assert", "raises", "warns", "deprecated_call"}

def assertion_units(path: Path) -> list[str]:
    tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
    units: list[str] = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Assert):
            units.append(f"{path}:{node.lineno}:assert:{ast.dump(node.test, include_attributes=False)}")
        elif isinstance(node, ast.FunctionDef) and node.name.startswith("test_"):
            args = [a.arg for a in node.args.args]
            units.append(f"{path}:{node.lineno}:sig:{node.name}:{args}")
        elif isinstance(node, ast.withitem):
            call = node.context_expr
            if isinstance(call, ast.Call):
                func = ast.unparse(call.func)
                if any(token in func for token in ASSERT_ATTRS):
                    units.append(f"{path}:{call.lineno}:ctx:{ast.dump(call, include_attributes=False)}")
    return units

def digest(root: Path) -> str:
    units: list[str] = []
    for path in sorted(root.rglob("test_*.py")):
        units.extend(assertion_units(path))
    blob = "\n".join(sorted(units)).encode()
    return hashlib.sha256(blob).hexdigest()

def main() -> int:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else "tests")
    lock_path = Path("tests/.assertion-lock.json")
    current = digest(root)
    if "--write" in sys.argv:
        lock_path.write_text(json.dumps({"sha256": current}, indent=2) + "\n")
        print(f"wrote {lock_path}")
        return 0
    expected = json.loads(lock_path.read_text())["sha256"]
    if current != expected:
        print("assertion surface changed; refuse the agent patch until a human rewrites the lock")
        print(f"expected {expected}")
        print(f"got      {current}")
        return 2
    print("assertion surface unchanged")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Baseline the lock on main, not on the agent branch. Command shape:

git checkout main
python tools/surface_lock.py tests --write
git checkout -
python tools/surface_lock.py tests

A human may rewrite the lock. An agent job should not hold the --write flag. If your platform cannot split those roles, the lock is theater.

Check 2: metamorphic pairs, not extra literals

Pick relations that remain true if the implementation is rewritten. Start with four that apply to a large class of pure functions. Drop any relation the spec does not actually promise.

Relation id	Input transform	Expected hold	Reject the patch when
R_perm	permutation of unordered input	f(x) == f(permute(x))	equality fails on any sampled perm
R_subset	drop an element from a collection	quality(f(x')) >= quality(f(x)) - eps only if the spec is monotonic	score jumps in the forbidden direction
R_dup	duplicate a record	cardinality and identity of results stay stable	duplicates create extra entities
R_roundtrip	encode then decode	decode(encode(x)) == x for the supported subset	roundtrip drifts only on the patched path

Proposed runner. It compares patched versus unpatched on the same pairs. A relation that already fails on main is a product bug, not an agent-patch signal. Do not freeze it here. File it.

# proposed harness: tests/metamorphic/test_relations.py
from __future__ import annotations

import itertools, random
from dataclasses import dataclass
from typing import Callable, Iterable

@dataclass(frozen=True)
class Pair:
    relation_id: str
    left: object
    right: object
    compare: Callable[[object, object], bool]

def permutations(sample: list, k: int = 3, seed: int = 7) -> Iterable[Pair]:
    rng = random.Random(seed)
    perms = list(itertools.islice(itertools.permutations(sample), 24))
    rng.shuffle(perms)
    for perm in perms[:k]:
        yield Pair("R_perm", sample, list(perm), lambda a, b: a == b)

def evaluate(fn: Callable, pair: Pair) -> bool:
    return pair.compare(fn(pair.left), fn(pair.right))

def classify(fn_main: Callable, fn_patch: Callable, pair: Pair) -> str:
    main_ok = evaluate(fn_main, pair)
    patch_ok = evaluate(fn_patch, pair)
    if not main_ok:
        return "preexisting"
    if patch_ok:
        return "hold"
    return "regress"

Wire fn_main and fn_patch through two import paths or two virtualenvs. Same fixtures. Same seed. Different code. If you only load the patched tree, you cannot tell a regression from a relation the codebase never had.

Seed the pair generator. Log (relation_id, seed, left, right, class). Without those four fields, a later freeze cannot be reviewed.

Where a free model and a free server actually sit

Relation ideas are cheap to propose and expensive to trust. A second process can read function signatures and emit candidate rows for the table above. A human still has to delete the ones the spec does not promise. That split matters more than the generator.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is one way to draft those candidate rows from signatures and docstrings. MonkeyCode's free server option is one way to run the lock script and the pair runner off the laptop that produced the patch. Neither step is a substitute for the lock, the control tree, or the review of proposed relations. If those three are missing, the model call is noise.

Keep the generator behind a flag the agent patch cannot enable in CI. Proposed relations land in a review file, not in the live suite.

# proposed review file: tests/metamorphic/candidates.md
# status: draft | accepted | rejected
# R_perm  sort_key=None  accepted  bags are unordered
# R_subset  quality=len  rejected  dropping a filter may raise, not shrink

Run the live suite on an isolated worker. The worker image should not contain the agent's shell history, extra pytest plugins, or a writable copy of tests/.assertion-lock.json. If the patch can rewrite the lock on the same filesystem that enforces it, check 1 is not a check.

Check 3: freeze pairs, not test names

Flakes still exist. Relation checks add a second source of them: pair order, iteration order inside the function, and clock-dependent encoding. Freezing a whole test_* function hides every relation that function owns. Freeze the pair instead.

Accept a freeze only when all of the following hold:

1. The same (relation_id, seed, left, right) fails at least twice under shuffled execution order.
2. The same pair holds when run alone against both trees.
3. The failure is classified regress on the patched tree and hold or preexisting is not mixed across reruns.
4. The ledger row carries an expiry. No expiry, no freeze.

Proposed ledger:

# tests/metamorphic/pair_freeze.yaml
# proposed format, not an observed production file
version: 1
entries:
  - relation_id: R_perm
    seed: 7
    left_hash: "9c1e..."
    right_hash: "aa40..."
    first_seen: "2026-09-07"
    expires: "2026-09-21"
    reason: "order-dependent bag compare in hash seed"
    owner: "human-id"

A job that loads this file must refuse expired rows. It must also refuse rows whose left_hash no longer matches the fixture. A freeze that outlives its input is a skip list. Count open rows. If the count grows across a week of agent patches, stop generating patches and delete the ledger. The relations are then lying or the code is racy. Either way the gate is no longer a gate.

What to log so the next patch is comparable

Minimum fields per CI run:

• assertion-lock digest
• number of pairs executed, by relation_id
• counts for hold, regress, preexisting
• freeze hits versus freeze misses
• whether --write was present in any command (it should be absent)

Do not publish those counts as quality scores. They are gate health. A rising preexisting count means the relation table drifted from the product. A rising regress count with a stable lock means the patch is the event. A rising freeze count with a stable regress count means the freeze is absorbing the signal.

Limitations

Metamorphic relations do not pin absolute values. A function can return the wrong number for every input and still preserve R_perm. Pair that class of bug with at least one sealed oracle you already trust, outside this workflow.

Relations that the spec does not promise will fail good patches. R_subset on a non-monotonic scorer is an example. Reject those relations in review. Do not freeze them.

The assertion lock is syntactic. An agent can weaken a helper that the assert calls without touching the assert node. If helpers are part of the question, add their module to the lock roots.

The dual-tree compare assumes you can import both implementations. Monolithic binaries and generated code may need two containers instead of two import paths. That is operational cost, not a reason to skip the control tree.

Free model drafts of relations will over-propose. Treat every draft as untrusted text. If a candidate cannot be restated as a sentence in the spec, it does not enter the suite.

Who should not use this

Do not use this gate as the only review on safety-critical paths that need exact bounds, cryptographic invariants, or regulatory traces. Relations are the wrong artifact there.

Do not use it on repos where tests are generated at the same time as the patch and no human owns the lock file. The circularity remains.

Do not use it if you cannot run the unpatched tree. Without a control, preexisting versus regress is a guess.

Skip the freeze ledger entirely if your CI cannot expire rows. A freeze without expiry is how relation failures disappear.

A short merge rule

Merge only when the assertion lock matches main, every accepted relation classifies hold or preexisting, and the freeze file contains no expired pair. Otherwise reject. The model that wrote the patch does not get a vote on those three bits.

If you want the pair runner off your laptop, park the lock script and the dual-tree job on an isolated worker. MonkeyCode's free server option can host that worker; the ledger and the lock still live in the repo you already review.