Reject, Undecided, or Review: A Lane Verdict for Agent Patches

An agent patch should stay off a shared eval host until three local lanes agree. The property oracle still holds, the fixture digest matches the locked set, and any flaky assertion sits behind an unexpired freeze that does not switch the other lanes off. Reversing that order spends host time on diffs a laptop can already reject.

The steps below are a proposal. Commands and snippets are unexecuted examples, meant to be copied into a branch and adapted. Nothing in this note is a measured pass rate, a quota, or a claim about how long any hosted option remains available.

What each lane is allowed to decide

Lane one asks whether a behavior contract broke. Lane two asks whether fixture bytes changed without review. Lane three asks whether a red result is a known unstable assertion or a new failure.

These questions are not substitutes. A green suite can hide a contract break when the fixture was rewritten. A freeze that also skips the property hides the same break. The gate in this note refuses both shortcuts.

Lane	Input	Pass condition	Shared host	Freeze
Property	Pure functions plus a pinned seed	Every listed property holds	Do not use	Not allowed
Fixture	Bytes under the fixture root	Observed SHA-256 equals the lock	Do not use	Not allowed
Replay	Cases left undecided by the first two lanes	Receipt matches the uploaded slice	Bounded slice only	One named assertion, with expiry
Full suite	The CI job you already trust	Out of scope for this gate	Not on this path	Not as a blanket skip

Keep the table beside the YAML the checker reads. A review comment that paraphrases the table is not a record. If a lane has no files in a given patch, write not_applicable and say why. Silence is not a pass.

Step 1. Classify the diff before any model call

Start from the patch, not from a generated explanation. A summary that repeats the commit message adds no evidence. List files first, then assign lanes.

git fetch origin main
git diff --name-only origin/main...HEAD > /tmp/agent-patch-files.txt
git diff --unified=0 origin/main...HEAD -- '*.py' > /tmp/agent-patch.diff
wc -l /tmp/agent-patch-files.txt /tmp/agent-patch.diff

A path under tests/fixtures/ forces the fixture lane. A path under src/ forces the property lane. An edit that only changes a test assertion may enter the freeze lane, and only after properties still pass on the locked fixtures.

# review-map.yml — proposal, edit per repository
lanes:
  property:
    paths: ["src/billing/", "src/quota/"]
    module: "checks.billing_props"
  fixture:
    paths: ["tests/fixtures/billing/"]
    lock: "tests/fixtures/billing/LOCK.json"
  freeze:
    registry: "tests/flaky-freeze.yml"
    max_age_days: 14

The value max_age_days: 14 is an example team policy, not a limit from any host. Choose a window reviewers will actually enforce. If the diff touches none of the mapped paths, stop and extend the map. An empty map must not be treated as a pass.

Step 2. Run property checks locally

Property checks stay deterministic and offline. They take a seed, a case count, and functions the diff can affect. They do not call a model.

# checks/billing_props.py — unexecuted example
import random
from billing import apply_refund  # function under test; do not reimplement it here

def prop_refund_never_exceeds_charge(seed: int, n: int = 200) -> None:
    rng = random.Random(seed)
    for i in range(n):
        charge = rng.randint(0, 10_000)
        refund = rng.randint(0, 12_000)
        result = apply_refund(charge_cents=charge, refund_cents=refund)
        assert 0 <= result.net_cents <= charge, (seed, i, charge, refund)
        assert result.rejected is (refund > charge), (seed, i, charge, refund)

python -m checks.runner --map review-map.yml --seed 20260924 --cases 200 \
  --out /tmp/property-report.json

Record the seed in the review note. The seed makes this local run repeatable. It does not replace the fixture lock, and it does not authorize a shared-host job.

Stop when a property fails. Do not open a freeze for that failure. Do not upload the patch. Keep the counterexample: seed, case index, and the arguments that broke the bound.

Add a property when review finds a contract the list cannot see. Do not add one that copies the implementation line by line. A check that inlines the same formula as the code under test stays green while that formula is wrong. That check is a tautology, and it does not belong in this lane.

Step 3. Lock fixture bytes

Matching filenames miss silent example edits. Hash the fixture bytes and compare them with a lock a human has reviewed. A minimal lock looks like this:

{
  "files": [
    {
      "path": "tests/fixtures/billing/invoice_min.json",
      "sha256": "replace-with-64-hex-chars"
    }
  ]
}

The checker itself is standard-library Python. It does not need a network, and it should run before any host upload.

python - << 'PY'
import hashlib, json, pathlib, sys
root = pathlib.Path("tests/fixtures/billing")
rows = []
for path in sorted(root.rglob("*")):
    if path.is_file() and path.name != "LOCK.json":
        rows.append({
            "path": str(path),
            "sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
        })
lock = json.loads((root / "LOCK.json").read_text())
observed = {r["path"]: r["sha256"] for r in rows}
expected = {r["path"]: r["sha256"] for r in lock["files"]}
report = {
    "missing": sorted(set(expected) - set(observed)),
    "extra": sorted(set(observed) - set(expected)),
    "changed": sorted(
        p for p in expected.keys() & observed.keys() if expected[p] != observed[p]
    ),
}
print(json.dumps(report, indent=2))
sys.exit(1 if any(report.values()) else 0)
PY

A non-empty changed list is a review event, not an automatic forever-reject. Restore the bytes, or commit a new lock with a one-line reason. Re-run the property lane after that commit. A new lock does not grandfather a broken property.

Canonicalize JSON before hashing when key order carries no meaning. Otherwise equivalent files flap the lane. Reviewers then ignore it, and the lane is dead.

Step 4. Freeze one assertion, never the oracle

Flaky tests exist. Retrying until the suite is green hides them. A freeze is a named exception with an owner, an expiry, and a property that remains mandatory.

# tests/flaky-freeze.yml — unexecuted example
freezes:
  - id: billing-clock-skew
    nodeid: "tests/test_billing.py::test_invoice_timestamp_order"
    owner: "qa-rotation"
    reason: "wall-clock assertion races the sandbox clock"
    expires: "2026-10-08"
    required_properties:
      - prop_refund_never_exceeds_charge
    fixture_lock: "tests/fixtures/billing/LOCK.json"

2026-10-08 is an example expiry for a review dated 2026-09-24. It is not a vendor calendar. Reject the freeze record when any rule below fails.

1. expires is missing, or it is earlier than the review date.
2. required_properties is empty.
3. A required property failed in step 2.
4. The fixture lock failed in step 3.
5. The node id appears in neither the diff nor the current failure log.

python -m checks.freeze_audit \
  --registry tests/flaky-freeze.yml \
  --today 2026-09-24 \
  --property-report /tmp/property-report.json \
  --fixture-report /tmp/fixture-report.json

A passing freeze audit allows one assertion to be skipped in the replay slice. It does not mark the patch safe. Contract checks and input hashes stay in force. Extending expires without a new failure log is a review defect. Cap extensions, or the file becomes a permanent skip list.

Step 5. Replay only what the local lanes could not decide

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product's free model access and free server option belong in this step, not in the local lanes. A free model may draft a candidate property, or restate a counterexample already reproduced on the laptop. A free server may execute only the seeded slice those lanes left undecided.

Treat both as availability options supplied for this workflow. This note does not state quotas, hardware, model names, uptime, or duration. Read the current product terms before you depend on either option.

The verdict function is the piece to keep under test. It is pure, local, and does not need a host to be falsified.

# checks/verdict.py — unexecuted example
def lane_verdict(prop_ok: bool, fixture_ok: bool, freeze_ok: bool, replay: str) -> str:
    if replay not in {"pass", "fail", "not_run"}:
        raise ValueError(replay)
    if not (prop_ok and fixture_ok and freeze_ok):
        return "reject"
    if replay == "not_run":
        return "undecided"
    if replay == "fail":
        return "reject"
    return "accept_for_review"

def test_freeze_cannot_save_a_broken_property():
    assert lane_verdict(False, True, True, "pass") == "reject"

def test_missing_host_is_not_a_pass():
    assert lane_verdict(True, True, True, "not_run") == "undecided"

Build the slice on the laptop. Do not assume a transport. On a given day the free server option might be a job submit, a container start, or another interface. The contract is the slice file plus a receipt, not a vendor-specific command.

python -m checks.select_undecided \
  --map review-map.yml \
  --property-report /tmp/property-report.json \
  --out /tmp/replay-slice.txt
python -c "import hashlib,pathlib; p=pathlib.Path('/tmp/replay-slice.txt'); print(hashlib.sha256(p.read_bytes()).hexdigest())"

If the host is unreachable, leave the verdict at undecided. Do not weaken a property to save a slot. Queue the slice, or run it on a trusted CI runner you already operate. Missing capacity is not permission to merge.

Ask the remote job for a receipt in this shape:

{
  "seed": 20260924,
  "slice_sha256": "hash-of-replay-slice.txt",
  "failed_nodeids": [],
  "skipped_by_freeze": ["tests/test_billing.py::test_invoice_timestamp_order"],
  "host_role": "free-eval"
}

Reject the receipt when slice_sha256 differs from the file you uploaded. Also reject it when a skipped node id is absent from the freeze registry. A screenshot of a green log is not a receipt.

How to read a red result

Use one order. Do not average the lanes, and do not let a later green lane erase an earlier red one.

1. Property report red: treat the patch as incorrect. Ignore suite retries.
2. Fixture report red: treat the inputs as unreviewed. Decide the lock, then re-run properties.
3. Only a frozen node id is red, and the freeze has not expired: record a skip. Do not delete the test.
4. A non-frozen node id is red on the slice: keep the patch off the merge queue. Add a property if the failure shows a missing contract. Do not add a freeze to force a green slice.
5. Local lanes are green and no host ran: report undecided, not pass.

The fifth state is the one short reviews omit. Undecided blocks promotion until a matching receipt exists.

Who should not use this gate

This gate does not replace a full suite. A listed property is silent about contracts you never wrote down. prop_refund_never_exceeds_charge says nothing about tax rounding until a separate property exists.

Do not adopt the ritual when the code under review has no pure function a property can call, when fixtures are generated during the run and cannot be hashed first, or when the defect is a multi-service race a seeded local check cannot represent. In those repositories the three files create false confidence. Isolate one pure function, or keep the suite on trusted CI and say that plainly.

Skip remote replay when the slice contains secrets, customer payloads, or fixtures you are not allowed to upload. A local hash is not upload permission. Redact first, or keep the slice on a private runner.

Four fields, then stop

Close the review with four fields, and do not add a fifth paragraph of model prose.

• Property seed and case count.
• Fixture lock result, or the one-line reason for a lock update.
• Freeze ids with expiries, or none.
• Replay receipt hash, or undecided.

A missing field means the patch was commented on, not reviewed. Draft text from a model does not fill the fields. A person comparing the three reports does.

If standing up a private runner is more work than this slice deserves, use the free server option already cited, and send only the undecided file. Confirm the upload rules on the day you submit. Free access is not a reason to move fixture bytes the lock lane already settled.