Track Property Provenance Before You Merge an Agent Patch

An agent patch is not proven when CI is green. It is proven when every accepting property has an author that is not the patch process, every fixture has a content hash taken before the patch landed, and every frozen flake has a first-seen SHA older than the change under review.

Same-process tests are a circular argument. The agent proposes code, then proposes the checks that the code will pass. That loop is cheap. It is also the most common way a plausible patch survives review.

This article is a review procedure, not a memoir. The harness below is a proposed artifact. Treat numbers in comments as placeholders, not measurements.

The independence rule

Three records must disagree with the patch process. If any one of them shares a process ID, a prompt session, or a commit with the patch, the green build is self-graded.

1. Property author. Who drafted the invariant, and in which runtime?
2. Fixture hash time. Was the input set sealed before the patch tree existed?
3. Flake first-seen SHA. Did this failure exist on main before the agent touched the file?

Adding tests inside the same PR does not increase independence. It increases surface area the agent can overfit. Volume is not evidence. Provenance is.

What this procedure is for

Use it when an automated coding agent (local, hosted, or mixed) submits a behavioral change plus a test diff. The goal is a merge gate that can reject a patch that only satisfies checks it invented.

Do not use it as a substitute for code review of the production diff. Properties catch classes of failure. They do not certify design.

Artifact: a property provenance ledger

Store one JSON document per invariant. Keep it outside the agent's working tree when possible. A review bot should refuse a merge if a property file is added in the same commit as the production edit it claims to judge.

{
  "property_id": "inv-order-book-qty-nonneg",
  "statement": "quantity remaining is >= 0 after every apply_fill",
  "author_process": "property-runtime-b",
  "author_kind": "human_edited_after_candidate_gen",
  "source_sha_before_patch": "a1b2c3d4",
  "fixture_set": "fixtures/order_book_v3",
  "fixture_sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "touches_production_paths": ["src/order_book.py"],
  "born_in_patch_pr": false,
  "flake_policy": {
    "first_seen_sha": "0099aa11",
    "freeze_until": null,
    "allow_freeze": false
  }
}

author_process must not equal the process that produced the patch. source_sha_before_patch must be an ancestor of the PR head. born_in_patch_pr must be false for any property that is allowed to vote on merge.

Step 1 — Seal fixtures before the agent runs

Hash the fixture directory on the merge base. Record the digest in the ledger. If the agent rewrites a fixture, the digest changes and the property loses its vote.

# proposed commands; run on the merge base, not on the PR head
git rev-parse MERGE_BASE > /tmp/base.sha
find fixtures/order_book_v3 -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/fixtures.list
sha256sum /tmp/fixtures.list | awk '{print $1}' > /tmp/fixtures.sha256

A fixture that appears only in the agent commit is not a fixture. It is part of the hypothesis. Move it to patch_scratch/ and exclude it from the voting suite.

Step 2 — Draft properties on a different process

Candidate generation can be mechanical. Acceptance cannot. The second process should see the public types, the sealed fixtures, and the bug report. It should not see the agent's patch diff or the agent's newly added tests.

When a team already spends a model session on the patch, a second session on another runtime is the cheapest way to break that correlation. MonkeyCode's free model access and free server option are one way to host that second process without folding it into the patch job. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product claim that matters here is availability of a separate free path, not a model ranking. Do not treat a second runtime as an oracle. Treat it as an author that can be wrong in a different way.

Label every generated candidate unreviewed. A human or a static policy must promote it before it can fail a merge.

Step 3 — Reject self-referential checks

A property that imports private helpers introduced in the patch is grading homework with the answer key. The validator below is proposed Python. It does not execute the agent.

# proposed harness: property_ledger.py
from __future__ import annotations

import ast
import json
from pathlib import Path

PATCH_PREFIXES = ("src/", "lib/")
FORBIDDEN_IMPORT_ROOTS = {"_agent_tmp", "patch_scratch"}


def load_ledger(path: Path) -> dict:
    return json.loads(path.read_text())


def production_names_in_patch(diff_paths: list[str]) -> set[str]:
    names: set[str] = set()
    for rel in diff_paths:
        if not rel.endswith(".py"):
            continue
        tree = ast.parse(Path(rel).read_text())
        for node in ast.walk(tree):
            if isinstance(node, ast.FunctionDef):
                names.add(node.name)
            if isinstance(node, ast.ClassDef):
                names.add(node.name)
    return names


def property_imports(property_path: Path) -> set[str]:
    tree = ast.parse(property_path.read_text())
    found: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                found.add(alias.name.split(".")[0])
        if isinstance(node, ast.ImportFrom) and node.module:
            found.add(node.module.split(".")[0])
    return found


def reject_self_grade(ledger: dict, property_path: Path, patch_paths: list[str]) -> list[str]:
    errors: list[str] = []
    if ledger.get("born_in_patch_pr"):
        errors.append("property born in the patch PR cannot vote")
    if ledger.get("author_process") == ledger.get("patch_process"):
        errors.append("author_process equals patch_process")
    imported = property_imports(property_path)
    if imported & FORBIDDEN_IMPORT_ROOTS:
        errors.append(f"imports scratch module: {imported & FORBIDDEN_IMPORT_ROOTS}")
    new_names = production_names_in_patch(
        [p for p in patch_paths if p.startswith(PATCH_PREFIXES)]
    )
    # A property may call public APIs. It may not reach into new private helpers.
    priv = {n for n in new_names if n.startswith("_")}
    src = property_path.read_text()
    leaked = {n for n in priv if n in src}
    if leaked:
        errors.append(f"references new private names: {sorted(leaked)}")
    return errors

Run this on CI against the ledger files, not against the agent's narrative in the PR body. Narratives are not records.

Step 4 — Encode one invariant as a bounded check

Keep the check boring. Bounded loops beat unbounded search in merge gates. The example targets a quantity invariant. Replace the domain with yours.

# proposed test; unexecuted example
import json
from decimal import Decimal

from order_book import OrderBook, Fill

LEDGER = json.loads(open("ledgers/inv-order-book-qty-nonneg.json").read())


def test_qty_nonneg_on_sealed_fixtures():
    assert LEDGER["fixture_sha256"] == open("/tmp/fixtures.sha256").read().strip()
    book = OrderBook.from_fixture_dir("fixtures/order_book_v3")
    for raw in book.iter_fills():
        fill = Fill(
            order_id=raw["order_id"],
            qty=Decimal(raw["qty"]),
            price=Decimal(raw["price"]),
        )
        book.apply_fill(fill)
        remaining = book.remaining(fill.order_id)
        assert remaining >= 0, (fill.order_id, remaining)

If the assertion needs a sleep, a wall-clock timeout, or a network hop, it is not a property for this gate. Move it to a soak job that cannot freeze production merges.

Step 5 — Rank the test diff, then drop same-PR voters

Classify every test file in the PR. Use four buckets. Do not merge bucket C or D into the voting set.

Bucket	Meaning	Votes on merge?
A	Property with ledger, sealed fixture, independent author	Yes
B	Pre-existing unit test, unchanged	Yes
C	New test added in the same PR as the production diff	No
D	Edited fixture or snapshot without a new ledger hash	No

Bucket C can still ship as documentation of intended behavior. It cannot be the reason the gate is green. If the only new coverage is C, the patch is unproven.

Step 6 — Freeze flakes by first-seen SHA, not by pain

A flake born in the agent PR is a regression with jitter. Freezing it hides the patch. The rule is mechanical.

# proposed policy; unexecuted example
from dataclasses import dataclass


@dataclass(frozen=True)
class FlakeRecord:
    test_id: str
    first_seen_sha: str
    patch_sha: str
    is_ancestor_of_patch: bool
    fail_rate_on_main: float  # fill from CI history; do not invent a threshold cult


def allow_freeze(rec: FlakeRecord) -> tuple[bool, str]:
    if not rec.is_ancestor_of_patch:
        return False, "flake first seen on the patch SHA; treat as regression"
    if rec.first_seen_sha == rec.patch_sha:
        return False, "first_seen_sha equals patch_sha"
    # Freeze is a time-boxed quarantine for known mainline noise, not a merge waiver.
    return True, "pre-existing on main; freeze requires expiry in a later job"

Bind expiry outside the agent. A freeze without an expiry job is a deleted test. Put the expiry in a calendar ticket or a CI annotation that fails after a date the ledger stores. Do not let the patch process write freeze_until.

Step 7 — Merge gate as a single command

Wire the checks so a reviewer can paste one invocation. Failure text should name the ledger field that broke, not a generic tests failed.

python property_ledger.py \
  --ledgers ledgers/ \
  --patch-paths $(git diff --name-only MERGE_BASE) \
  --patch-process agent-session-17 \
  --require-author-neq-patch \
  --forbid-born-in-pr
pytest -q tests/properties -k 'not bucket_c'

If property_ledger.py exits non-zero, do not run the agent's new tests as a consolation green. That inverts the gate.

Failure modes the ledger is meant to catch

Overfitted snapshots. The agent updates expected JSON to match a bug. The fixture hash moves. Bucket D. No vote.

Private helper leakage. A new _normalize_qty appears in the patch and in a new test. The AST pass flags it. The property never gained independence.

Flake laundering. A race is introduced, then marked xfail in the same PR. allow_freeze returns false because first_seen_sha is the patch.

Author collapse. Patch generation and property generation share a session log. author_process equals patch_process. The ledger refuses the vote even if the invariant text looks wise.

Limitations

This procedure does not measure model quality. It measures whether the accepting tests are independent records.

It will not catch a wrong invariant that was drafted on a second runtime and then blessed too quickly. Independent authors can share a mistaken domain assumption. Review the statement in English before you review the assertion in Python.

Hashed fixtures do not help if the fixture generator is non-deterministic. Seal concrete files. Do not seal a script that talks to the network.

AST import checks are shallow. A property can still reach new behavior through a public function that the patch quietly redefined. Pair this gate with a review of public signatures on src/.

The free second runtime, when used, can be unavailable, slow, or empty-bodied. An empty candidate list is a skip, not a pass. Do not auto-approve because property generation returned no files.

Who should not use this

Skip the ledger if the change is a one-line comment, a pin bump with an existing lockfile, or a revert that already has a sealed suite on main.

Skip it if no one on the team will read author_process. A JSON file nobody diffs is ceremony. Ceremony trains agents to emit prettier self-grades.

Skip it for exploratory spikes that will not merge. Independence cost is for code that other people will run.

Teams without CI history cannot compute first_seen_sha honestly. In that case, refuse all flake freezes rather than guessing ancestry.

What to keep in the review template

Paste four questions under the PR. Require answers that point at files, not adjectives.

1. Which ledger IDs vote, and which process authored them?
2. Does /tmp/fixtures.sha256 match the merge base?
3. Which test files are bucket C, and are they excluded from the gate?
4. Is every freeze pointing at a SHA that is an ancestor of the patch?

If question 1 and question 3 collapse to "the agent wrote more tests," the patch is still a hypothesis. Keep it out of main until a different author owns the property that can fail.

If you already isolate patch generation, isolate property generation next. The ledger is the smallest artifact that makes that split visible in review.