Block the Agent Patch Until a Second Process Replays the Witness

An agent patch stays blocked until a second process replays a seed-locked property campaign and matches a witness file. A local green log is only a candidate observation. It is not the merge ticket.

The witness stays small on purpose. It records the fixture digest, property ids, seed, trial budget, trials spent, and whether replay called a model. Reviewers can regenerate those fields. They cannot regenerate a chat transcript.

Why the authoring log is the wrong artifact

Agent diffs often ship a passing run from the same context that wrote the change. That run shares imports, environment variables, and sometimes the test file. Shared context inflates confidence.

Separate three measurements before the diff debate starts. Compare trials spent with the budget. Compare the fixture digest with the pre-generation pin. Compare the replay with the witness. If one measurement is missing, the patch stays blocked.

This is a testing strategy, not a score. Zero failures inside a finished budget means the campaign completed. It does not mean the property set was sufficient.

Fields the witness must carry

Use a fixed schema so reviews do not depend on log formatting. The example is a proposal. It is not a log from a production gate.

{
  "schema": "agent-patch-witness/v1",
  "fixture_digest": "sha256:<64 hex chars from step 1>",
  "properties": ["balance_non_negative", "transfer_conserves_sum"],
  "seed": 170924,
  "trial_budget": 200,
  "trials_spent": 200,
  "failures": 0,
  "counterexample": null,
  "model_called_during_replay": false
}

Digest bytes, not filenames. A rename with identical bytes is the same fixture. A one-byte edit starts a new campaign, even when property names stay put.

Keep generated cases out of the fixture tree. Cases come from the seed. Fixtures are the pinned inputs those cases may touch. Mixing the two makes the digest chase the generator, which hides drift.

Workflow

Commands below assume fixtures/, properties/campaign.py, and witness/latest.json. They are illustrative. No pass rate is claimed.

1. Pin the digest before generation

Hash the fixture tree before any model call and before the patch is applied. If the diff also edits fixtures/, split that edit into its own change. The patch should consume the pin, not replace it.

python3 - <<'PY'
import hashlib, pathlib
root = pathlib.Path("fixtures")
h = hashlib.sha256()
for p in sorted(x for x in root.rglob("*") if x.is_file()):
    h.update(p.relative_to(root).as_posix().encode())
    h.update(b"\0")
    h.update(p.read_bytes())
print("sha256:" + h.hexdigest())
PY

Store the printed line in the review note. A later mismatch means the campaign measured a different tree than the one you pinned. Discard that run. Do not average it with a newer digest.

2. Admit only predicates a reviewer can restate

A property enters the campaign only after a reviewer restates it as a relation on inputs and outputs. The restatement cannot depend on private helpers introduced by the patch. Drafts may come from a person or a model. Both stay proposals until that sentence exists.

Drop predicates that do not constrain behavior. assert True, identity comparisons, and checks that only confirm a return value spend budget without evidence. They can replay cleanly and still teach nothing. Remove them before the ledger opens.

3. Spend a fixed trial budget

The budget is an input. Trials spent is an output. The runner stops at the budget even when every trial passed. An early stop is allowed only to record a counterexample.

# Proposal sketch. Not an executed measurement.
import hashlib, random

def run_campaign(props, seed, budget, fixture_bytes):
    rng = random.Random(seed)
    spent = 0
    failures = []
    digest = "sha256:" + hashlib.sha256(fixture_bytes).hexdigest()
    for _ in range(budget):
        spent += 1
        case = {"n": rng.randrange(0, 50), "delta": rng.randrange(-10, 10)}
        for name, fn in props:
            ok, detail = fn(case)
            if not ok:
                failures.append({"property": name, "case": case, "detail": detail})
                return {
                    "seed": seed,
                    "trial_budget": budget,
                    "trials_spent": spent,
                    "failures": failures,
                    "fixture_digest": digest,
                }
    return {
        "seed": seed,
        "trial_budget": budget,
        "trials_spent": spent,
        "failures": failures,
        "fixture_digest": digest,
    }

You still supply props. The sketch does not invent a domain oracle. Its job is the stop rule: a finished pass spends the full budget, and a failure keeps the case that broke the relation.

Shrink the counterexample before filing it. Replay the failing case, then reduce n and delta one step at a time while the same property still fails. Store the smallest case. A wide dump is harder to review and no stronger as evidence.

4. Serialize the ledger, not the transcript

Write witness/latest.json from ledger fields only. Leave prompts, tokens, and chat text out of the file. Those strings are not functions of the seed, so a second process cannot reproduce them.

python3 properties/campaign.py --seed 170924 --budget 200 \
  --fixtures fixtures/ --out witness/latest.json
python3 -m json.tool witness/latest.json > /tmp/witness.pretty.json

If failures is non-empty, attach the reduced case and stop. Keep the failure as a property result. Do not rewrite the witness so the same case disappears.

5. Replay where the model cannot participate

Run the same campaign in a clean process. Unset prompt and credential variables first. Then compare digest, seed, budget, spend, and failures with the witness.

env -u OPENAI_API_KEY -u MODEL_PROMPT \
  python3 properties/campaign.py --replay witness/latest.json --fixtures fixtures/

# Proposal check. Equality is exact, not approximate.
def replay_matches(witness, actual):
    keys = ("fixture_digest", "seed", "trial_budget", "trials_spent")
    if any(witness[k] != actual[k] for k in keys):
        return False
    if actual.get("model_called_during_replay"):
        return False
    return witness.get("failures", []) == actual.get("failures", [])

Four equalities decide the replay.

1. Fixture digest equals the pre-generation pin and the witness field.
2. Seed and trial_budget equal the recorded values.
3. trials_spent equals the budget, unless a recorded counterexample explains an earlier stop.
4. No model client was imported during replay.

A hosted runner is optional capacity, not a source of truth. A timeout or a transport success that does not print a match is a missing replay. Missing replay stays blocked.

6. Apply a decision table

Observed state	Digest	Spend	Action
Replay match, no failures	Equals pin	Equals budget	Evidence complete; human diff review still required
Counterexample reproduced	Equals pin	Stops at the case	Block; keep the reduced case
Digest drift	Differs	Any	Discard; re-pin; rerun
Replay absent	Unknown	Unknown	Block
Budget field missing	Any	Absent	Block

Evidence complete is not approval. The reviewer still reads the diff for deleted checks, loosened bounds, and edits outside the claimed fix. The table only answers whether the testing evidence can be regenerated.

Where drafting help and a replay host fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access fits step 2 as a drafting aid for candidate predicates, which remain proposals until a reviewer restates them. The free server option fits step 5 as an optional clean replay host when one is already available.

Neither option names a model, a quota, a machine size, or a duration, and this article does not add those figures. If either option is down, run the same commands on a local interpreter. Availability is not evidence.

Do not upload fixtures that contain secrets. A digest does not redact bytes you copied to a host. Strip secrets first, or replay on a machine that already holds them.

What to record on the pull request

Put four observations in the review comment. They are counts, not a grade.

• trials_spent against trial_budget. A stop at 12 of 200 with an empty failure list means the campaign did not finish.
• Digest equality with the step 1 pin. Hex after sha256: is 64 characters. A shorter string is truncation, not a match.
• failures length. Zero is required for a complete campaign. One counterexample is enough to block.
• Whether replay imported a model client. The only acceptable value here is no.

Across a queue, tally how many patches produced a witness, how many mismatched on replay, and how many offered only an authoring-session log. Those tallies describe your reviews. They are not a product benchmark, and this article reports none.

Limitations

A witness proves regeneration, not completeness. A predicate that returns true on every input will match on replay and still miss defects. Read the predicate before you trust the spend.

random.Random is convenient. It is not a portability promise. Different interpreters can diverge for the same seed. Pin the interpreter used for replay, or replace the generator with an integer recurrence you control.

Hashing the fixture tree ignores clocks, network calls, and process globals. Properties that read them overstate isolation. Inject those inputs as fixture fields, or reject the property.

A free hosted option can be withdrawn, throttled, or unusable. The strategy has to survive that. If the only passing path is a remote runner you cannot repeat, you do not have a witness. You have a session.

Who should not use this

Skip documentation-only edits with no behavioral claim. Skip work where you cannot state an invariant that would still make sense if the patch were reverted. Skip an emergency rollback whose validation is an already witnessed build, rather than a new agent diff.

Do not use a hosted replay for secret-bearing fixtures. Do not accept a drafted predicate as the oracle for a patch from the same uninterrupted model session unless step 2's restatement is in the review. Convenience is not independence.

Closing

Pin the fixture digest. Spend a visible trial budget. Write the witness from the ledger, then replay it where the model cannot help. The authoring log can stay attached. It does not carry the patch.

If a free MonkeyCode server is already in your setup, point step 5 at it and attach witness/latest.json to the review. The server is a place to regenerate the witness. It is not a reason to merge.