Lease-Scoped Replay Sheets for Agent-Patch Property Checks

A flake freeze is void when the worker that saw the miss cannot show a fresh workspace lease. Property checks and fixture pins still have to pass first. The lease is the extra gate that stops a dirty tree from becoming a skip on the merge path.

Agent-patch failures collapse into one red line in CI, but the causes are not interchangeable. A property stopped holding. A fixture pin drifted. A test flickered because another job wrote into the same workspace. Only the third case is even eligible for a freeze, and only after the observation is tied to a lease id.

This is a testing strategy with a decision table and a checker you can review. The Python below is a proposal. It was not executed against a live suite for this draft, and it does not claim a measured flake rate, a model quota, or a server size.

What a replay sheet is

A replay sheet is one JSON object emitted at the end of classification. It is the merge gate's input, not a dashboard and not a chat log. Four fields are mandatory.

1. property_result is pass or fail for every predicate owned by a touched module.
2. fixture_pin records whether the fixture hash matches the pin the patch is allowed to read.
3. lease carries an id, a holder, and a state. Missing, empty, or stale means abort.
4. signature and base_signature hash the test id, assertion id, fixture pin, and normalized failure class.

If a required field is absent, the checker returns abort, not unknown. Unknown is how an unreviewed skip reaches main. Timestamps, hostnames, and model request ids stay out of the signature. Include them and every miss looks unique, which disables the base comparison without anyone editing a test.

An illustrative catalog maps files to predicate ids. It is a shape, not a dump from a real repository.

{
  "modules": {
    "src/parser.py": ["parse.roundtrip", "parse.error_class"]
  }
}

The pin file stores hashes only. Do not commit the fixture bodies beside the pin if those bodies are generated per run.

{
  "fixtures/parser": "sha256:example-pin-not-a-real-digest"
}

Separate the model from the replay worker

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option map to two different jobs here. The free model may draft a candidate predicate after a deterministic miss. The free server may host the replay, but only after it takes a workspace lease. Neither role decides whether the patch is correct. This article does not name models, quotas, hardware, or durations, because those details were not supplied and should not be guessed. Treat both options as access you re-check on the day you enable the job. They are not a capacity plan you can hard-code.

The model is not allowed to write a freeze row. A suggested predicate lands under quarantine/ until a second replay, on a new lease and a clean workspace, evaluates it against the pinned fixtures. Disagreement discards the file. Agreement still does not merge it. A reviewer adds it to the catalog in a separate commit, or they leave it out.

Decision table

Use the single row that matches the evidence in the sheet. Do not blend rows to get a softer outcome.

property_result	fixture_pin	lease	base signature	Decision
fail	match	fresh	any	Reject the patch. Do not freeze.
pass	mismatch	fresh	any	Reject. Update the pin in a reviewed commit.
pass	match	missing or stale	any	Abort. Do not record a freeze.
pass	match	fresh	differs from base	Reject. The miss is new on the patch.
pass	match	fresh	same as base	Freeze, with expiry and this lease id.
quarantine only	match	fresh	any	Hold for a second leased replay.

Only the fifth row writes a freeze. A property failure is a product defect or a bad predicate. A pin mismatch is a fixture change. A stale lease is an invalid observation. Those three are not flakes, and they must not share an exit code with a recorded freeze.

Numbered workflow

Run the steps in order. Spending a model call before the local checks finish is a process bug, not a shortcut. The command lines describe an intended interface. They are not a released CLI. Only decide is specified in full later. The other subcommands are placeholders until you implement them.

1. Map touched paths to property owners

Start from the files the patch changes. Each path needs an owner in the property catalog. An unowned path stops the run. That stop is a coverage gap, not an intermittent test.

git diff --name-only origin/main...HEAD > /tmp/touched.txt
python3 replay_sheet.py bind \
  --touched /tmp/touched.txt \
  --catalog properties/catalog.json

bind should exit 2 when a path has no catalog entry. Keep that code distinct from test failures so the CI summary does not invite a freeze request. If the diff is empty, exit 0 and skip the rest. There is nothing to classify.

2. Evaluate properties against the pin

A property is a predicate a reviewer can falsify. Prefer invariants: accepted input still round-trips, or rejected input still yields the same error class. Full equality on generated prose is a weak stand-in. It becomes a tautology when the predicate never reads the fixture input.

python3 replay_sheet.py properties \
  --catalog properties/catalog.json \
  --fixtures fixtures/pin.json \
  --patch-ref HEAD

A fail here ends the workflow. Do not send the log to a model in the hope of reclassifying a broken invariant as a flake. Record property_result: fail on the sheet and stop before lease acquire. A remote worker is irrelevant once a local predicate has already failed.

3. Acquire a workspace lease before replay

The replay worker must start from an empty workspace and record the holder. A shared checkout can fail because another job wrote a build directory or a fixture cache. That failure must not become a freeze row.

python3 replay_sheet.py lease-acquire \
  --worker "$REPLAY_WORKER" \
  --workspace "$WORKSPACE_ID" \
  --ttl-seconds 900

The 900-second TTL is a local example policy, not a vendor limit. Choose a window your queue can honor. If acquire fails, stop. Re-running on the same dirty tree manufactures false flakes.

If you point this step at a free server, the lease command has to be real. A no-op acquire makes every later fresh check theater. Prove that a second concurrent holder is rejected before you trust the gate. Store the lease id in the sheet. A prose note that "the server looked idle" is not a lease.

4. Replay on the patch ref and on the merge base

Run the same test id on HEAD and on the merge base, same pin, same lease generation. A freeze requires the same signature on both. A miss that exists only on the patch is a regression, even if it appeared once.

python3 replay_sheet.py replay \
  --lease-file /tmp/lease.json \
  --pin fixtures/pin.json \
  --test-id "$TEST_ID" \
  --ref HEAD \
  --ref-base origin/main

Normalize the failure class before hashing. Strip timings and absolute paths. Keep the assertion id and the fixture pin. Over-normalizing hides real regressions. Under-normalizing blocks every legitimate freeze. Review the normalizer in the same pull request as any catalog change.

5. Draft a predicate only after the sheet is otherwise clean

Spend a model call only when a reviewer asks for a stronger predicate after a coverage gap, or when you already have an unexplained miss that matches base and you want a tighter check for next time. The prompt is narrow: one predicate, it must reference fixture input, and it must not restate a constant true. Write the response to quarantine/. Do not append it to catalog.json.

python3 replay_sheet.py propose \
  --catalog properties/catalog.json \
  --failure /tmp/miss.json \
  --out quarantine/candidate.json

Discard empty bodies, non-success statuses, and predicates that do not mention an input field. Do not loop retries inside the merge job. A later job can try again. The gate does not wait on a completion, and a failed draft is not evidence for or against the patch.

6. Emit one decision

python3 replay_sheet.py decide \
  --sheet /tmp/replay-sheet.json \
  --out artifacts/decision.json

decide prints reject, abort, hold, or freeze. A freeze object includes lease_id, signature, expires_at, and test_id. A row missing lease_id is an abort even if a person pasted the other fields into the artifact. CI should upload artifacts/decision.json next to the test log so the row is auditable without opening a transcript.

Checker sketch

This module is a review sketch for the sixth step. It is not a published library, and it was not run in CI for this article. Add a test per table row before you wire it into a required check.

#!/usr/bin/env python3
"""Proposal: classify a replay sheet. Not executed for this draft."""
import json
import sys

REQUIRED = (
    "property_result",
    "fixture_pin",
    "lease",
    "signature",
    "base_signature",
    "test_id",
)

def decide(sheet: dict) -> dict:
    missing = [key for key in REQUIRED if key not in sheet]
    if missing:
        return {"decision": "abort", "reason": "missing_fields", "fields": missing}
    lease = sheet["lease"] or {}
    if not lease.get("id") or lease.get("state") != "fresh":
        return {"decision": "abort", "reason": "lease_not_fresh"}
    if sheet["property_result"] != "pass":
        return {"decision": "reject", "reason": "property_fail"}
    pin = sheet["fixture_pin"] or {}
    if pin.get("state") != "match":
        return {"decision": "reject", "reason": "fixture_drift"}
    if sheet.get("predicate_state") == "quarantine":
        return {"decision": "hold", "reason": "second_replay_required"}
    if sheet["signature"] != sheet["base_signature"]:
        return {"decision": "reject", "reason": "miss_not_on_base"}
    if not sheet.get("expires_at"):
        return {"decision": "abort", "reason": "freeze_missing_expiry"}
    return {
        "decision": "freeze",
        "lease_id": lease["id"],
        "signature": sheet["signature"],
        "expires_at": sheet["expires_at"],
        "test_id": sheet["test_id"],
    }

def main() -> int:
    sheet = json.load(sys.stdin)
    json.dump(decide(sheet), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Two stdin objects cover rows that must not freeze. They are synthetic. They are not logs from an agent run. By inspection, decide returns reject for the first object and abort for the second. Run them locally before you depend on that behavior.

printf '%s\n' '{"property_result":"fail","fixture_pin":{"state":"match"},"lease":{"id":"L1","state":"fresh"},"signature":"sig-a","base_signature":"sig-a","expires_at":"2026-10-16T00:00:00Z","test_id":"parse.roundtrip"}' \
  | python3 replay_sheet.py

printf '%s\n' '{"property_result":"pass","fixture_pin":{"state":"match"},"lease":{"id":"","state":"stale"},"signature":"sig-a","base_signature":"sig-a","expires_at":"2026-10-16T00:00:00Z","test_id":"parse.roundtrip"}' \
  | python3 replay_sheet.py

Add a third fixture for the fifth table row, and assert that lease_id is copied into the freeze object. If that assertion is missing, the checker can look green while still accepting a hand-edited row. The sample expiry 2026-10-16T00:00:00Z is only an input shape. It is not a recommended freeze length.

Limits of the evidence

A passing property suite only shows that the predicates you wrote still hold. It does not show that you wrote the right ones. A freeze row shows that one signature matched on one leased replay of base and of the patch. It does not show that the test is nondeterministic in production, and it does not justify deleting the test.

Expiry is a field in your artifact. This workflow does not renew it. After expires_at, the next merge replays. If the miss is gone, delete the row. If the miss remains only on the patch, the earlier freeze was wrong, and the patch should be rejected.

Isolation is only the isolation you encoded. If lease-acquire always returns fresh, the table's third row never fires, and contaminated workers will keep writing freezes. Schedule a negative test: two acquires, one workspace, and expect the second to fail.

Quarantined predicates can still be vacuous. Reject any candidate whose body does not reference a field from the fixture pin. A second leased replay that passes a vacuous predicate is not confirmation. Keep that file out of the catalog until a person can name an input that would falsify it.

Who should skip this

Do not use the remote steps if the repository must stay air-gapped. Run the catalog map, the property checks, the base replay, and decide on internal workers. Omit propose.

Do not send fixtures to a model endpoint if they contain secrets, customer payloads, or unredacted production logs. Pin hashes of synthetic fixtures. A tighter predicate is not worth a data leak.

Do not adopt this if the team expects a model to approve merges. No accept path in the table starts from a model response. The only non-reject outcome that can touch main is a time-boxed freeze after deterministic checks, and that outcome is an exception.

Do not use the fifth row on a repository with no merge base. Without base_signature, a freeze cannot be justified, and the checker should abort. New suites need owners and pins first, not a freeze ledger.

What to store with the build

Keep the sheet beside the test log. A reviewer should see which table row fired without opening a transcript. If the only artifact of a green build is a completion payload, this gate did not run.

Count the decisions for a week before you tighten the TTL or the normalizer. A healthy sheet mix is mostly reject on real property misses, a few abort values while the lease client is still shaky, and rare freeze rows. A week of only freeze means the other rows are unreachable, which is a broken gate, not a stable suite. That count is a local audit you run yourself. It is not a benchmark this draft performed.

If free model access and a free server are already on the account you use for this kind of experiment, keep the split: draft under quarantine on the model, replay only under a lease on the server, and leave decide on a machine you control. Read the current access terms on the day you enable the job. Availability is something you verify per run. It is not a constant a test plan can freeze.