Invariant Replay for Agent Patches: Relations, Dataset Leases, and a Counted Flake Freeze

A green unit test after an agent rewrite is not evidence. It is often a tautology. Lock relations over a leased dataset, then freeze only the relations that fail a counted rerun protocol.

Golden outputs collapse the moment the agent is allowed to rewrite the function that produced them. Expected JSON is a snapshot of one implementation. Relations survive a rewrite. That is the useful split.

This article is a test plan, not a gate manifesto. The artifact is a small invariant-replay harness: dataset leases, three relation classes, and a freeze file driven by rerun counts. No production metrics are claimed. Treat the code as a labeled, runnable sketch.

Why expected values go stale on the first rewrite

Agent patches arrive as diffs against a ticket, not against a spec. The ticket names a symptom. It does not name the invariant. If the patch also authors assert result == 42, the assertion restates the patch. CI stays green. The bug moves.

Fixtures still matter. They just should not store the answer. Store the input corpus and the relation that corpus must obey. The oracle is the relation. The agent does not get to write it.

Three failure modes show up in data-processing code more than in CRUD handlers:

1. Aggregation that depends on row order after a "performance" rewrite.
2. Ingest that is no longer idempotent after a "retry" rewrite.
3. Split-file processing that disagrees with concat-then-process after a "streaming" rewrite.

Those are metamorphic relations. You do not need the true total. You need two runs to agree under a declared transform.

Relation classes that survive a rewrite

Keep the catalog short. Three classes cover most agent patches on tabular pipelines. Add more only when a ticket names a fourth.

R1 — Order invariance. Permute rows. Counts, sums, and group keys must match. Sort order of the output may change. The relation must say so.

R2 — Idempotent ingest. Apply the write twice. The stored projection must match the single-write projection. Tombstones and "upsert by id" belong here. Append-only logs do not.

R3 — Split-merge. Process A then B. Process concat(A,B). Bag-equal the records. Streaming rewrites break this quietly.

Floating-point reductions are not R1 unless you fix the reducer. If the patch parallelizes a sum, either lock a compensated summation or move that check to the flake protocol. Do not skip it.

Example: a tiny aggregator under test

The subject is a CSV aggregator. An agent is allowed to rewrite aggregate() as long as the relations hold. The function is deliberately small so the harness stays readable.

# aggregator.py
from collections import defaultdict
from typing import Iterable

def aggregate(rows: Iterable[dict]) -> dict[str, dict[str, float]]:
    """Group by `sku`. Sum `qty`. Mean `price` weighted by qty."""
    acc: dict[str, list[float]] = defaultdict(lambda: [0.0, 0.0])
    for row in rows:
        sku = row["sku"]
        qty = float(row["qty"])
        price = float(row["price"])
        acc[sku][0] += qty
        acc[sku][1] += qty * price
    out = {}
    for sku, (qty, weighted) in acc.items():
        out[sku] = {"qty": qty, "avg_price": (weighted / qty) if qty else 0.0}
    return out

A golden test that hard-codes avg_price == 10.5 dies as soon as the agent changes rounding. A relation test does not.

Step 1 — Lease the dataset, not the expected JSON

Put corpus files under fixtures/leases/. A lease is a manifest: path, sha256, expiry, and the relations that may read it. The agent may read the lease. The agent may not edit it in the same patch.

# fixtures/leases/orders_v3.yaml
id: orders_v3
path: fixtures/data/orders_v3.csv
sha256: 9c0e1c6a8b2d44f0a1f7c3e5b8d9012a3c4e5f67890123456789abcdef012345
expires: 2026-10-01
allowed_relations: [R1_order, R2_idempotent, R3_split_merge]
notes: "synthetic orders; no production PII"

# leases.py
import hashlib, yaml
from pathlib import Path
from datetime import date

class LeaseError(Exception):
    pass

def load_lease(manifest: Path, today: date) -> dict:
    lease = yaml.safe_load(manifest.read_text())
    data = Path(lease["path"])
    if not data.exists():
        raise LeaseError(f"missing corpus {data}")
    digest = hashlib.sha256(data.read_bytes()).hexdigest()
    if digest != lease["sha256"]:
        raise LeaseError("corpus hash drifted; renew the lease, do not patch it away")
    if date.fromisoformat(str(lease["expires"])) < today:
        raise LeaseError(f"lease {lease['id']} expired on {lease['expires']}")
    return lease

Hash drift is a signal. An agent that "fixes" a fixture to match a new total is rewriting the oracle. Reject that diff in review even if CI is green.

Step 2 — Encode relations as pairwise runs

Each relation is a transform plus a comparison. The comparison is bag equality on a canonical form. Canonical form drops output order and rounds money to a stated quantum.

# relations.py
import csv, random
from pathlib import Path
from aggregator import aggregate

QUANTUM = 1e-9

def load_rows(path: Path) -> list[dict]:
    with path.open(newline="") as f:
        return list(csv.DictReader(f))

def canon(result: dict) -> dict:
    return {
        sku: {
            "qty": round(vals["qty"] / QUANTUM) * QUANTUM,
            "avg_price": round(vals["avg_price"] / QUANTUM) * QUANTUM,
        }
        for sku, vals in sorted(result.items())
    }

def r1_order(rows: list[dict], seed: int = 7) -> tuple[dict, dict]:
    left = canon(aggregate(rows))
    shuffled = rows[:]
    rng = random.Random(seed)
    rng.shuffle(shuffled)
    right = canon(aggregate(shuffled))
    return left, right

def r2_idempotent(rows: list[dict]) -> tuple[dict, dict]:
    once = canon(aggregate(rows))
    twice = canon(aggregate(rows + rows))
    # For true idempotent *storage* you would round-trip a store.
    # This sketch treats duplicate rows as a pure-function stand-in.
    return once, twice

def r3_split_merge(rows: list[dict]) -> tuple[dict, dict]:
    mid = max(1, len(rows) // 2)
    a, b = rows[:mid], rows[mid:]
    sequential = canon(aggregate(a + b))
    combined = canon(aggregate(rows))
    return sequential, combined

R2 as written is the wrong relation for a pure aggregator: duplicate rows should double qty. Label that. If the ticket is about upsert-by-sku into a store, the second run must hit the store, not concatenate rows. Relation names are cheap. Wrong relations are expensive.

Correct R2 for a store looks like this sketch:

def r2_store_idempotent(store, rows):
    store.reset()
    store.ingest(rows)
    once = canon(store.projection())
    store.ingest(rows)
    twice = canon(store.projection())
    return once, twice

Write the relation against the interface the patch is allowed to touch. If the agent rewrites the store, the relation still runs. If the agent rewrites the test to concatenate rows, the lease's allowed_relations list no longer matches. Fail closed.

Step 3 — A counted flake freeze, not a skip

Flakes on relation tests are usually reducer order, clock injection, or shared temp dirs. Skipping the test deletes the signal. Freeze the relation id with a rerun budget and an expiry. The freeze is data. It is not a pytest mark the agent can sprinkle.

# freeze.py
from dataclasses import dataclass
from datetime import date
import json
from pathlib import Path

@dataclass(frozen=True)
class Freeze:
    relation_id: str
    expires: date
    reruns: int
    pass_needed: int
    reason: str

def load_freezes(path: Path, today: date) -> dict[str, Freeze]:
    raw = json.loads(path.read_text()) if path.exists() else []
    out = {}
    for item in raw:
        exp = date.fromisoformat(item["expires"])
        if exp < today:
            continue  # expired freeze is a hard fail again
        out[item["relation_id"]] = Freeze(
            relation_id=item["relation_id"],
            expires=exp,
            reruns=int(item["reruns"]),
            pass_needed=int(item["pass_needed"]),
            reason=item["reason"],
        )
    return out

Example freeze file. Notice there is no skip: true.

[
  {
    "relation_id": "R1_order",
    "expires": "2026-09-17",
    "reruns": 5,
    "pass_needed": 5,
    "reason": "parallel sum; awaiting deterministic reducer"
  }
]

Protocol:

1. Run the relation once. On pass, do nothing.
2. On fail, look up a freeze. No freeze means fail the job.
3. With a freeze, rerun reruns times. Require pass_needed passes.
4. If the freeze is expired, ignore it. The relation is mandatory again.
5. Agents may not add freeze rows in the same commit as the patch under test.

# test_relations.py
from datetime import date
from pathlib import Path
from leases import load_lease
from relations import load_rows, r1_order, r3_split_merge
from freeze import load_freezes

LEASE = Path("fixtures/leases/orders_v3.yaml")
FREEZE = Path("fixtures/freezes.json")

def _eval(pair, relation_id, freezes):
    left, right = pair
    if left == right:
        return
    fr = freezes.get(relation_id)
    if fr is None:
        raise AssertionError(f"{relation_id} mismatched and is not frozen")
    passes = 0
    for i in range(fr.reruns):
        # re-seed R1; R3 is deterministic given the split
        left_i, right_i = pair if relation_id != "R1_order" else r1_order(rows, seed=11 + i)
        passes += int(left_i == right_i)
    if passes < fr.pass_needed:
        raise AssertionError(
            f"{relation_id} freeze held {passes}/{fr.reruns}; need {fr.pass_needed}"
        )

def test_leased_relations():
    today = date(2026, 9, 3)
    lease = load_lease(LEASE, today)
    rows = load_rows(Path(lease["path"]))
    freezes = load_freezes(FREEZE, today)
    assert "R1_order" in lease["allowed_relations"]
    _eval(r1_order(rows), "R1_order", freezes)
    _eval(r3_split_merge(rows), "R3_split_merge", freezes)

Run it locally with a pinned date in CI so lease expiry is deterministic.

python -m pytest test_relations.py -q
python -c "from datetime import date; from pathlib import Path; from leases import load_lease; load_lease(Path('fixtures/leases/orders_v3.yaml'), date(2026,9,3))"

Step 4 — Draft relations from the ticket, then lock them by hand

Relation text is the expensive part. A model can propose candidates from the ticket plus the diff. A human locks the list in the lease. That split is the whole workflow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already use MonkeyCode, the relevant pieces are free model access and a free server option. Point the model at a prompt that may only emit relation ids, transforms, and comparison rules. Do not let it emit expected totals. Run the harness on the free server so the lease check, hash, and counted reruns do not compete with paid CI minutes. Do not treat a model-authored freeze row as valid. Freeze rows are operator data.

A drafting prompt that stays inside the harness:

Ticket and diff are attached.
Return YAML only:
  - id: R*
    transform: <permute|replay-ingest|split-merge|other>
    compare: <canon-bag|exact|tolerance>
    must_not: <what the agent might try to assert instead>
No expected numeric outputs. No pytest skips. No freeze rows.

Reviewers accept or delete rows. The lease file is the lock. The patch under test cannot expand allowed_relations without a separate commit.

Decision table for this plan

Symptom after the patch	Relation to lock	Freeze allowed?	Reject if
Totals change when CSV is shuffled	R1 order invariance	Only if reducer is explicitly non-associative and expiry ≤ 14 days	Agent edits the corpus hash
Retry duplicates rows in storage	R2 store idempotence	No	Test concatenates rows instead of replaying ingest
Shard merge disagrees with full file	R3 split-merge	No	Agent adds an output sort to hide bag inequality
Intermittent mismatch on R1 only	Counted freeze on R1	Yes, with reruns ≥ 5	Freeze committed in the same patch
Agent adds assert result == fixture.json	None of the above	No	Golden output introduced without a relation

The table is the review script. Use it in the PR template. Do not turn it into a second test runner.

Limitations

Metamorphic relations miss bugs that preserve the relation. An off-by-a-constant total can pass R1, R2, and R3 if every run is wrong the same way. Add one independently computed checksum when the domain has one. Tax tables, FX rates, and well-known seed datasets qualify. Most tickets do not.

Dataset leases go stale. An expired lease is a failed job, which is correct, and also a source of noise if nobody owns renewal. Assign an owner on the lease file. Do not auto-renew from the agent.

The counted freeze can hide a race for fourteen days. That is the point and the cost. If the relation is a safety property, do not freeze it. Fail the job.

Canon rounding can swallow a real drift. QUANTUM = 1e-9 is a choice. Document it next to the lease. If the patch is about rounding, the quantum is the spec. Do not hide it in a helper.

Free-model drafts will over-propose relations. Volume is not coverage. Three locked relations beat twelve unlocked ones.

Who should not use this

Do not use this plan if you already have a closed spec and generated oracles from it. Golden outputs are valid when they do not come from the patch.

Do not use it as a substitute for review on security or privacy diffs. Relations do not see an exfiltrated field that is unused by the aggregator.

Do not use it on code with no deterministic seam. GUI event order, live market feeds, and unseeded network retries need a different harness. Freeze files will pile up and then mean nothing.

Do not let the patch under test modify fixtures/leases/, fixtures/freezes.json, or test_relations.py in the same commit. If your VCS cannot enforce that split, the workflow is incomplete.

Checklist before merge

1. Lease hash matches the corpus. Expiry is in the future.
2. Every relation the ticket implies is in allowed_relations.
3. No golden numeric output was added by the patch.
4. Freeze rows, if any, pre-exist and have a reason string.
5. Rerun counts are executed, not skipped.
6. Model-drafted YAML was edited by a reviewer.

Invariant replay does not make agent patches safe. It makes a class of circular tests harder to merge. That is the scope. Keep the catalog small, keep the lease owned, and keep the freeze countable.