A Merge Contract the Agent Cannot Edit

Green tests inside an agent patch prove that the patch is consistent with itself. They do not prove that the change matches the behavior you intended to keep. The same turn that rewrote src/ can rewrite tests/.

The control that still works is a merge contract stored outside the agent's write set. Three layers cover most codebases: named invariants, content-addressed inputs, and a flake budget that quarantines unstable checks instead of deleting them. This is a proposed layout. It is not a report of a production incident. Replace the sample laws with identities that are actually true in your domain.

The contract is a directory, not a prompt

A merge contract is a human-owned tree. The agent may read it. CI must reject any patch that modifies it unless a named owner is on the change. Four files are enough to start.

1. oracle/invariants.py — properties that do not mention the agent's examples.
2. oracle/fixtures.lock.json — SHA-256 of input blobs those properties consume.
3. oracle/flake_budget.yaml — quarantined checks, owners, and expiry dates.
4. oracle/write_set.txt — paths the agent is allowed to touch.

The contract answers three questions on every candidate diff. Did the new code preserve laws that were true before the patch? Did the agent retarget the inputs? Did coverage disappear because a flaky check was deleted?

Layer 1: Name a law, then encode it

Example tests are cheap for a patch generator to satisfy. Change the assertion. Change the fixture. Change both. The run stays green. The behavior does not.

Invariants are harder to game when they are algebraic. Round-trip laws, inverse pairs, and idempotence do not encode a single golden output. They encode a relationship that has to hold for a set of inputs.

# oracle/invariants.py
# Proposed catalog. Swap the domain functions for yours.
from __future__ import annotations

import json
from pathlib import Path

from hypothesis import given, settings
from hypothesis import strategies as st

# Production symbols only. Do not import tests the agent just wrote.
from app.codec import decode, encode
from app.normalize import canonicalize
from app.pricing import apply_discount, invert_discount


def load_cases(name: str) -> list[dict]:
    raw = Path("oracle/inputs") / f"{name}.json"
    return json.loads(raw.read_text())


@given(st.binary(min_size=0, max_size=4096))
@settings(max_examples=80, deadline=None)
def test_encode_decode_round_trip(payload: bytes) -> None:
    assert decode(encode(payload)) == payload


@given(st.text(max_size=256))
def test_canonicalize_is_idempotent(value: str) -> None:
    once = canonicalize(value)
    assert canonicalize(once) == once


def test_discount_inverse_on_pinned_cases() -> None:
    for case in load_cases("pricing"):
        price = case["price_cents"]
        rate = case["rate_bps"]
        forward = apply_discount(price, rate)
        back = invert_discount(forward, rate)
        assert back == price, case["id"]

Keep two rules. The oracle never reads files under tests/ in the working tree. Each property names a law: round-trip, inverse, idempotent, monotonic, order-preserving. If you cannot name the law, it is an example. Leave it out of the gate.

Hypothesis is optional. If you do not want a fuzzing dependency, feed the same laws from oracle/inputs/ only. The generator shrinks. The law does not.

Generator isolation

A property that calls the same helper the agent just wrote, then compares the helper to itself, is a tautology. Split the oracle.

# Proposed pattern: independent checker, not a mirror of app/pricing.py
def reference_apply_discount(price_cents: int, rate_bps: int) -> int:
    # Integer form of the published billing rule, maintained by billing owners.
    return price_cents - (price_cents * rate_bps) // 10_000


def test_discount_matches_published_rule() -> None:
    for case in load_cases("pricing"):
        got = apply_discount(case["price_cents"], case["rate_bps"])
        expected = reference_apply_discount(case["price_cents"], case["rate_bps"])
        assert got == expected, case["id"]

The reference function is allowed to be slow. It is not allowed to import the patched module. That split is the point.

Layer 2: Hash the inputs, not the outputs

Golden-output files are easy to retarget. The agent rewrites the expected bytes until the assertion matches. Hashed inputs do not have that failure mode. The property still computes the output. The lock file only proves the catalog saw the same bytes a human reviewed.

# oracle/hash_fixtures.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("oracle/inputs")
LOCK = Path("oracle/fixtures.lock.json")


def digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def build_lock() -> dict[str, str]:
    entries = {}
    for path in sorted(ROOT.rglob("*.json")):
        rel = str(path.relative_to(ROOT))
        entries[rel] = digest(path)
    return entries


def main() -> int:
    current = build_lock()
    if not LOCK.exists():
        LOCK.write_text(json.dumps(current, indent=2) + "\n")
        print("wrote new lock")
        return 0
    expected = json.loads(LOCK.read_text())
    if current != expected:
        print("fixture lock mismatch")
        print("expected:", json.dumps(expected, indent=2))
        print("current :", json.dumps(current, indent=2))
        return 1
    print(f"ok {len(current)} fixtures")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run the lock check before the suite.

python oracle/hash_fixtures.py
python -m pytest oracle/invariants.py -q --tb=short

If a human intends to add a case, they update the lock in a separate commit. The agent write set should not include oracle/.

Layer 3: Budget the flake, do not delete it

Flaky checks get deleted because they block merges. Deletion is a coverage hole with no ticket. A budget makes the hole visible. Skipping is not passing. CI still fails when the count exceeds the cap or a date lapses.

# oracle/flake_budget.yaml
max_frozen: 3
frozen:
  - id: canonicalize_unicode_hyphen
    owner: platform-api
    expires: "2026-09-30"
    reason: "locale-dependent hyphen folding on CI image"
  - id: discount_inverse_leap_promo
    owner: billing
    expires: "2026-10-07"
    reason: "promo calendar not pinned in fixture"

# oracle/check_budget.py
from __future__ import annotations

import datetime as dt
import sys
from pathlib import Path

import yaml

BUDGET = Path("oracle/flake_budget.yaml")


def main() -> int:
    data = yaml.safe_load(BUDGET.read_text())
    today = dt.date.today()
    frozen = data.get("frozen", [])
    max_frozen = int(data["max_frozen"])
    errors = []
    if len(frozen) > max_frozen:
        errors.append(f"frozen={len(frozen)} exceeds max_frozen={max_frozen}")
    for item in frozen:
        expires = dt.date.fromisoformat(item["expires"])
        if expires < today:
            errors.append(f"{item['id']} expired on {expires}")
        if not item.get("owner"):
            errors.append(f"{item['id']} has no owner")
    if errors:
        print("\n".join(errors))
        return 1
    print(f"flake budget ok ({len(frozen)}/{max_frozen})")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Wire the skip in the catalog with a small helper. Frozen ids must match the id field. Unknown ids should fail the budget check, not disappear into a default skip.

# oracle/freeze.py
from __future__ import annotations

import datetime as dt
from pathlib import Path

import pytest
import yaml

_BUDGET = yaml.safe_load(Path("oracle/flake_budget.yaml").read_text())
_FROZEN = {
    row["id"]: dt.date.fromisoformat(row["expires"])
    for row in _BUDGET.get("frozen", [])
}


def maybe_freeze(test_id: str):
    expires = _FROZEN.get(test_id)
    if expires is None:
        return
    pytest.skip(f"frozen until {expires}: {test_id}")

Dates in the sample file are placeholders relative to mid-September 2026. Set them from your own calendar. Do not copy them as policy.

Write-set gate

# oracle/write_set.txt
src/
app/
README.md

# Proposed CI fragment. Adapt to your git host.
set -e
CHANGED=$(git diff --name-only origin/main...HEAD)
fail=0
for path in $CHANGED; do
  allowed=0
  while IFS= read -r prefix; do
    [ -z "$prefix" ] && continue
    case "$path" in
      "$prefix"*) allowed=1 ;;
    esac
  done < oracle/write_set.txt
  if [ "$allowed" -ne 1 ]; then
    echo "path outside write set: $path"
    fail=1
  fi
done
exit "$fail"

If the agent needs a new production path, a human expands the write set. That commit is the audit trail. Contract edits never ride along with implementation edits.

A six-step loop

Use this as a checklist. It is not a ceremony.

1. Freeze the contract. Confirm oracle/ is clean on main and absent from the write set.
2. State the law. Add or reuse one invariant that would fail if the intended bug remained.
3. Pin inputs. Drop JSON cases under oracle/inputs/ and refresh fixtures.lock.json in a human commit.
4. Generate a candidate patch against src/ only. A local agent loop is enough. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that generation step when you do not want the candidate to share a checkout with the oracle runner. Keep evaluation on a tree the agent cannot write.
5. Evaluate outside the patch. Run hash_fixtures.py, check_budget.py, then pytest oracle/invariants.py. Ignore tests the patch added under tests/ until a human promotes a named law into oracle/.
6. Record flakes. If a check is unstable, add a budget row with an owner and a date. Do not delete the invariant.

Step 4 is optional. The contract does not depend on any vendor. Generation and evaluation stay on different surfaces even if both run on a laptop.

Merge decision table

Observation	Action
Invariants pass, lock matches, budget clean, write set respected	Merge implementation only
Invariants fail on a named law	Reject. Do not edit the oracle to match the patch
Lock mismatch	Reject unless a human updated inputs in a separate commit
New files under tests/ only	Ignore for merge. Promote later if a law can be named
Frozen count > max_frozen	Reject. Investigate or raise the budget in a reviewed change
Frozen row past expires	Reject. Fix the invariant or extend the date with a reason
Patch touches oracle/	Reject. Contract edits are human-owned

The table is the policy. The scripts only encode it. If a row in the table cannot be checked mechanically, it does not belong in CI yet.

What this does not catch

Round-trip laws will not see a wrong default in a UI string. Inverse pairs will not see a slow query. Hashed fixtures will not see a clock. A flake budget will hide a real race until the expiry date.

Do not treat the catalog as a specification of the whole product. Treat it as the set of behaviors you refuse to renegotiate in an agent turn. Hypothesis example counts, JSON case sizes, and max_frozen: 3 are starting points. They are not measurements from a fleet. Tune them against your own failure history.

Promotion still matters after merge. Agent-written tests are drafts. Read tests/ for a law you can name. If you find one, move it into oracle/invariants.py and pin its inputs. If you cannot name a law, leave the draft where it is and do not let CI treat it as a gate.

Who should not use this

Skip the layout if a human already reviews every assertion and the agent cannot edit tests/. Skip it for throwaway prototypes with no merge gate. Skip it if your domain has no named laws; you will invent tautologies and call them properties.

Teams that need machine-checked proofs should use a theorem prover, not this YAML. Teams that cannot assign owners to frozen rows will watch the budget become a junk drawer. Teams that let the agent update oracle/ "for convenience" have no contract left.

Keep the oracle runner boring. The interesting work is naming laws that stay true when the implementation is rewritten.