Merge Agent Patches Only When Two Test-World Receipts Match

A green CI job is not proof that an agent patch preserved the test world. Fixtures can be rewritten to match new output. Property suites can shrink their domain or their iteration budget. Flaky tests can turn into unnamed skips. Merge on a lockfile and two matching receipts, not on a single passing run.

The protocol below is a proposal with runnable scoring code. It is not a report of production incident rates, and it does not assume a particular CI vendor.

The test world is the proof, not the job status

The test world is the set of objects that give tests their meaning. Three parts matter on an agent diff.

1. Fixture bytes and their content digest.
2. Property checks: which invariants, how many examples, which seed policy.
3. Flake policy: which node ids may skip, until when, and who issued the lease.

If the model may edit those three, a passing suite proves almost nothing. The code under test and the proof of that code moved together. That is cheap. It is also common.

Three failure modes the lockfile is built to catch

Fixture rewrite. A snapshot or JSON fixture is updated so the new function output becomes the expected value. The test still passes. The old contract is gone.

Property shrinkage. Strategies get narrower. max_examples drops. A filter excludes the input that used to fail. The job is faster and greener. The search is weaker.

Flake freeze expansion. pytest.skip() appears inside a body the agent could not stabilize. Source-level mark.skip scans miss it. The hole remains after merge.

None of these require malice. They are the lowest-cost way to make a suite green.

Artifact: testworld.lock.json

Keep this file in the repo. Treat it like a package lockfile. Agent patches must not write it. Human-labeled commits may.

{
  "version": 1,
  "fixtures": {
    "tests/fixtures/invoice_v3.json": {
      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "bytes": 0
    }
  },
  "properties": {
    "tests/test_invoice_properties.py::test_roundtrip_preserves_cents": {
      "min_examples": 200,
      "seed_policy": "fixed",
      "seed": 42,
      "strategy_fingerprint": "@given(cents=st.integers(0,10**9))"
    }
  },
  "flake_leases": [
    {
      "nodeid": "tests/test_tax.py::test_dst_boundary",
      "expires_unix": 1796124800,
      "skips_remaining": 3,
      "issuer": "human:oncall",
      "reason": "tzdata mismatch on one runner image"
    }
  ]
}

The empty-file digest and the expires_unix value are schema examples, not a measured corpus or a recommended lease length. Replace both from your tree and your policy.

Step 1 — Digest fixture bytes, not filenames

Filenames lie. Agents rename and copy. Hash contents.

# digest_fixtures.py — illustrative scorer, not a published package
from __future__ import annotations

import hashlib
import json
from pathlib import Path

FIXTURE_ROOT = Path("tests/fixtures")
SUFFIXES = {".json", ".txt", ".bin", ".csv", ".snap"}


def sha256_file(path: Path) -> tuple[str, int]:
    data = path.read_bytes()
    return hashlib.sha256(data).hexdigest(), len(data)


def build_fixture_map() -> dict[str, dict]:
    out: dict[str, dict] = {}
    for path in sorted(FIXTURE_ROOT.rglob("*")):
        if path.is_file() and path.suffix in SUFFIXES:
            digest, n = sha256_file(path)
            key = path.as_posix()
            out[key] = {"sha256": digest, "bytes": n}
    return out


if __name__ == "__main__":
    print(json.dumps(build_fixture_map(), indent=2, sort_keys=True))

A changed hash with no lockfile bump is a merge failure. A changed hash with a lockfile bump still needs a human-labeled commit. The agent job should not be allowed to produce that commit.

Step 2 — Fingerprint properties, including the budget

Collect node ids, a floor on examples, and a coarse strategy fingerprint. The fingerprint is a string, not a proof of semantic equivalence. It catches the cheap shrinkage: smaller integers, shorter strings, extra assume() filters that collapse the domain.

# fingerprint_properties.py — proposal: parse a tiny subset of pytest + Hypothesis
from __future__ import annotations

import ast
import json
from pathlib import Path


class PropertyVisitor(ast.NodeVisitor):
    def __init__(self, filename: str) -> None:
        self.filename = filename
        self.rows: list[dict] = []

    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
        min_examples = None
        given_src = None
        for dec in node.decorator_list:
            src = ast.unparse(dec)
            if "given" in src:
                given_src = src
            if "settings" in src and "max_examples" in src:
                min_examples = self._max_examples(dec)
        if given_src is not None:
            self.rows.append(
                {
                    "nodeid": f"{self.filename}::{node.name}",
                    "min_examples": min_examples if min_examples is not None else 100,
                    "strategy_fingerprint": "".join(given_src.split()),
                }
            )
        self.generic_visit(node)

    @staticmethod
    def _max_examples(dec: ast.AST) -> int | None:
        for child in ast.walk(dec):
            if isinstance(child, ast.keyword) and child.arg == "max_examples":
                if isinstance(child.value, ast.Constant) and isinstance(
                    child.value.value, int
                ):
                    return child.value.value
        return None


def scan(root: Path) -> dict[str, dict]:
    props: dict[str, dict] = {}
    for path in sorted(root.rglob("test_*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        visitor = PropertyVisitor(path.as_posix())
        visitor.visit(tree)
        for row in visitor.rows:
            props[row["nodeid"]] = {
                "min_examples": row["min_examples"],
                "seed_policy": "fixed",
                "strategy_fingerprint": row["strategy_fingerprint"],
            }
    return props


if __name__ == "__main__":
    print(json.dumps(scan(Path("tests")), indent=2, sort_keys=True))

Scoring rule: fail if any locked node id disappears, if min_examples falls, or if the fingerprint string changes without a human lockfile edit.

This parser is incomplete. Nested settings, imported decorators, and parametrized properties need a collection-time plugin. Use it as a gate that catches cheap edits, not as a full Hypothesis auditor.

Step 3 — Issue flake leases. Never accept a bare skip

A skip with no lease is a hole. A lease is a capability token. CI may decrement skips_remaining. CI may not extend expires_unix. Only a human-labeled lockfile commit may add or renew a lease.

Collect skips from the pytest report. Do not grep source for @pytest.mark.skip alone. Agents skip inside the body.

# conftest.py excerpt — illustrative
from __future__ import annotations

import json
from pathlib import Path


def pytest_terminal_summary(terminalreporter, exitstatus, config):
    skipped = sorted(
        item.nodeid for item in terminalreporter.stats.get("skipped", [])
    )
    Path("observed_skips.json").write_text(
        json.dumps(skipped, indent=2) + "\n", encoding="utf-8"
    )

# score_leases.py — illustrative
from __future__ import annotations

import time
from typing import Any


def score_flake_leases(
    locked: list[dict[str, Any]],
    observed_skips: set[str],
    now: int | None = None,
) -> list[str]:
    now = int(time.time()) if now is None else now
    errors: list[str] = []
    by_id = {row["nodeid"]: row for row in locked}

    for nodeid in sorted(observed_skips):
        row = by_id.get(nodeid)
        if row is None:
            errors.append(f"unleased skip: {nodeid}")
            continue
        if now >= int(row["expires_unix"]):
            errors.append(f"expired lease: {nodeid}")
        if int(row["skips_remaining"]) <= 0:
            errors.append(f"lease exhausted: {nodeid}")
    return errors

Step 4 — Wipe the on-disk example database, then emit a receipt

A second run that reuses .hypothesis/ or pytest's cache is not an independent observation. Hypothesis stores shrinking examples on disk. Pytest stores failure caches. An agent session can poison both before CI starts.

# isolate_and_run.sh — proposal
set -euo pipefail
ROOT="$1"
RECEIPT="$2"
export HYPOTHESIS_DATABASE=":memory:"
export PYTEST_ADDOPTS="--cache-clear -q"
rm -rf "$ROOT/.hypothesis" "$ROOT/.pytest_cache"
cd "$ROOT"
python digest_fixtures.py > /tmp/fix.json
python fingerprint_properties.py > /tmp/prop.json
pytest tests
python emit_receipt.py "$RECEIPT" /tmp/fix.json /tmp/prop.json observed_skips.json

A single green run can still be session-tainted. Clone the candidate twice, with the agent process gone. Compare receipts.

# emit_receipt.py — illustrative dual-run proof
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def receipt_sha256(payload: dict) -> str:
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()


def main() -> None:
    out, fix, prop, skips = map(Path, sys.argv[1:5])
    payload = {
        "fixtures": json.loads(fix.read_text(encoding="utf-8")),
        "properties": json.loads(prop.read_text(encoding="utf-8")),
        "observed_skips": json.loads(skips.read_text(encoding="utf-8")),
    }
    # Wall-clock duration is omitted on purpose. Duration noise would
    # make two honest runs disagree.
    doc = {"payload": payload, "sha256": receipt_sha256(payload)}
    out.write_text(json.dumps(doc, indent=2) + "\n", encoding="utf-8")


if __name__ == "__main__":
    main()

# compare_receipts.py — illustrative
from __future__ import annotations

import json
import sys
from pathlib import Path


def main() -> None:
    left = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    right = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))
    if left["sha256"] != right["sha256"]:
        raise SystemExit(
            f"receipt mismatch: {left['sha256']} != {right['sha256']}"
        )


if __name__ == "__main__":
    main()

Receipt payload should include fixture maps, property fingerprints, observed skips, and the git tree id of the candidate. It should not include timestamps or durations.

Step 5 — Score the lockfile after the receipts match

# score_against_lock.py — illustrative merge gate
from __future__ import annotations

import json
import sys
from pathlib import Path

from score_leases import score_flake_leases


def main() -> None:
    lock = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    receipt = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))["payload"]
    errors: list[str] = []

    if receipt["fixtures"] != lock["fixtures"]:
        errors.append("fixture digest drift versus lock")

    locked_props = lock["properties"]
    seen_props = receipt["properties"]
    for nodeid, row in locked_props.items():
        if nodeid not in seen_props:
            errors.append(f"property missing: {nodeid}")
            continue
        got = seen_props[nodeid]
        if int(got["min_examples"]) < int(row["min_examples"]):
            errors.append(f"property budget fell: {nodeid}")
        if got["strategy_fingerprint"] != row["strategy_fingerprint"]:
            errors.append(f"strategy fingerprint changed: {nodeid}")

    errors.extend(
        score_flake_leases(
            lock["flake_leases"], set(receipt["observed_skips"])
        )
    )
    if errors:
        raise SystemExit("\n".join(errors))


if __name__ == "__main__":
    main()

Merge scorecard

Signal	Pass	Fail
Fixture digest vs lock	equal	any hash or path change
Property node ids	superset of lock	deleted property
min_examples	>= locked value	lower
Strategy fingerprint	equal	changed string
Unleased skip	none	any
Lease expiry	now < expires_unix	expired
Dual-run receipts	identical sha256	mismatch
Hypothesis DB / pytest cache	wiped per run	reused from agent session
testworld.lock.json in agent diff	absent	present

The last row is the one that makes the rest real. If the model can edit the lockfile, the table is theater.

A practical check:

# fail if the agent identity touched the lockfile
git diff --name-only origin/main...HEAD | grep -qx testworld.lock.json && \
  test "$GIT_AUTHOR_EMAIL" = "[e-mail]" && exit 1

Replace the author email with whatever identity your automation actually uses. The command is a policy sketch, not a hosted workflow.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The protocol does not depend on a particular product. It does depend on isolation. Candidate patches have to be produced somewhere the lockfile is not writable. The two receipt runs have to happen somewhere the agent session cannot reach.

MonkeyCode's free model access is enough to generate a candidate in a sandbox that cannot open testworld.lock.json for write. The free server option is enough to clone that candidate twice, wipe .hypothesis/ and .pytest_cache, run the digester, the fingerprint script, pytest, and the lease scorer, then compare receipts. Paid CI can sit behind that gate. If the receipts already disagree, there is nothing for a larger runner to decide.

Do not send lockfile hashes, lease lists, or fixture bytes into the prompt that produces the patch. The model does not need the proof in order to write application code. Giving it the proof often teaches it how to satisfy the proof.

Limitations, and who should not use this

The strategy fingerprint is a string compare. Equivalent refactors of a given decorator will fail closed. That is intentional. Re-lock by hand.

The AST scanner misses properties registered at collection time. Teams that generate tests dynamically need a pytest plugin that records items after collection, not a static parse.

Dual receipts do not detect a wrong-but-stable oracle. If both runs use a fixture that was already poisoned on main, they will match. Protect main with the same rule: agents cannot touch the lockfile.

Flake leases encode policy, not physics. A race still needs a fix. The lease only stops the hole from becoming unnamed.

Do not use this protocol if your suite has no fixtures, no properties, and no skip policy. There is nothing to lock. Do not use it if the same session that writes production code is also allowed to regenerate receipts or refresh .hypothesis/. Isolation is the method.

Time-sensitive claims about third-party quotas, model catalogs, or hardware are omitted on purpose. Check current vendor docs before you rely on any capability not stated here.

What to copy

Copy the lockfile schema, the rule that agents cannot edit it, the wiped example database, and the dual-run receipt compare. Those pieces still work if every product mention in this article is removed.

If candidate patches already come from free model access, run the second receipt on an isolated free server and keep the lockfile out of the prompt. That is the only tooling note worth taking.