If You Cannot Replay the Agent, Do Not Merge

You should refuse any agent patch you cannot replay. A scrolling demo is not a build artifact. Merge the replay file first, then consider the diff.

Take this position

Unreproducible agents are not production-grade coding assistants. They behave like unrecorded pair programmers with total amnesia. You would not accept that from a human contractor.

You would demand logs from a flaky integration test. Demand the same evidence from every coding-agent session. Otherwise you are shipping a one-time stage performance.

Opinion, not a hedged framework dump: replay or reject. Clever plans cannot replace a missing tool transcript. Your future self cannot debug a vanished chain of tool calls.

What replay actually means

Replay is not running the same prompt and hoping. Replay means stored inputs reconstruct the same side effects. You keep enough evidence to rebuild the job later.

A useful replay file must answer six concrete questions.

1. You stored the task prompt and every attached file hash.
2. You stored the tool schemas the process could legally call.
3. You stored each call, argument blob, and result digest.
4. You stored the model identifier as a pinned opaque string.
5. You stored resulting diffs as content-addressed patch files.
6. You stored the human decision: merge, reject, or isolate.

If any row is missing, you do not have a replay. You have a diary entry with extra punctuation. Diary entries do not belong on the main branch.

Cheap retries make the lie easier

Free or cheap tokens hide the cost of wasted loops. You retry until something compiles, then forget the path. Cheap retries without transcripts create irreproducible success stories.

Teams then paste a green terminal into Slack threads. Nobody can name the tool order that produced files. The next intern cannot reconstruct the change two weeks later.

That is how "the agent just knows the repo" myths start. The agent does not actually know your repository. It sampled a path you failed to record.

Practice the contract in a sandbox

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that sandbox to practice the replay contract, not to skip it.

Keep the same checker when you later pay for inference. Free sandboxes do not lower the merge bar. They only keep experiments off your invoice.

Hunt these failure modes

Watch for patches that exist only inside a chat pane. Watch for tool traces truncated to look "readable" in demos. Watch for model labels copied from a pricing page.

If the transcript was edited by hand, discard the run. If file hashes drifted after cleanup, discard the run. If the PR description is only vibes, demand the file.

Pin a replay contract

Pin a JSON contract before you write more agent glue. The schema below is an example, not a shipped standard. Adapt field names to your orchestrator without adding poetry.

{
  "schema_version": "1",
  "task_id": "ISSUE-1842",
  "task_hash": "sha256:REPLACE_WITH_PROMPT_FILE_HASH",
  "model_id": "pinned-opaque-string",
  "base_git_sha": "abc1234",
  "allowed_tools": ["read_file", "write_file", "run_tests"],
  "tool_calls": [
    {
      "seq": 1,
      "name": "read_file",
      "args_digest": "sha256:ARGS",
      "result_digest": "sha256:BYTES"
    }
  ],
  "patch_digest": "sha256:GIT_DIFF",
  "human_decision": "review",
  "secrets_redacted": true
}

Every tool_calls item needs a digest, not a novel. Store stdout hashes when the bytes are large. Store raw text only when the payload is tiny and non-secret.

Keep a tight allowlist beside that JSON file. Unknown names fail the run. Broad shell access does not belong in the first draft.

# tools.allow  (example)
read_file
write_file
run_tests

Validate before you argue about the diff

Validate the file in CI before you discuss the patch. The checker below is an example, not a product. Do not treat a passing checker as proof of model quality.

#!/usr/bin/env python3
"""Example only: unexecuted replay-file checker. Not a benchmark."""
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = {
    "schema_version",
    "task_hash",
    "model_id",
    "base_git_sha",
    "allowed_tools",
    "tool_calls",
    "patch_digest",
    "human_decision",
    "secrets_redacted",
}


def sha256_bytes(data: bytes) -> str:
    return "sha256:" + hashlib.sha256(data).hexdigest()


def fail(message: str) -> None:
    print("REPLAY_FAIL:", message)
    raise SystemExit(2)


def load_allowlist(path: Path) -> set[str]:
    names = set()
    for line in path.read_text(encoding="utf-8").splitlines():
        text = line.strip()
        if text and not text.startswith("#"):
            names.add(text)
    if not names:
        fail("allowlist is empty")
    return names


def main(argv: list[str]) -> None:
    if len(argv) != 3:
        fail("usage: check_replay.py replay.json tools.allow")
    replay_path = Path(argv[1])
    allow_path = Path(argv[2])
    payload = json.loads(replay_path.read_text(encoding="utf-8"))
    missing = REQUIRED - set(payload)
    if missing:
        fail(f"missing keys: {sorted(missing)}")
    if payload.get("secrets_redacted") is not True:
        fail("secrets_redacted must be true")
    if payload.get("human_decision") not in {"review", "reject", "isolate"}:
        fail("human_decision is not an allowed value")
    if not str(payload.get("patch_digest", "")).startswith("sha256:"):
        fail("patch_digest is not a sha256 digest")
    allow = load_allowlist(allow_path)
    declared = set(payload.get("allowed_tools") or [])
    if declared - allow:
        fail(f"tools not in allowlist file: {sorted(declared - allow)}")
    calls = payload.get("tool_calls") or []
    if not isinstance(calls, list) or not calls:
        fail("tool_calls must be a non-empty list")
    for item in calls:
        name = item.get("name")
        if name not in allow:
            fail(f"tool call not allowed: {name}")
        if not str(item.get("args_digest", "")).startswith("sha256:"):
            fail(f"args_digest missing for {name}")
        if not str(item.get("result_digest", "")).startswith("sha256:"):
            fail(f"result_digest missing for {name}")
    print("REPLAY_OK", sha256_bytes(replay_path.read_bytes()))


if __name__ == "__main__":
    main(sys.argv)

Run it on fixtures that should fail. A validator that never fails is only decoration. You want red on missing hashes, unknown tools, and empty diffs.

Merge with a boring table

Use a boring table instead of another agent self-eval. A human still owns merge after the table returns. The table exists to block unreproducible theater.

Signal	Action	Rule you enforce
Replay file missing from the PR	Block	No transcript means no merge
Tool name outside tools.allow	Block	The run is already dirty
Digest does not match saved bytes	Block	You cannot replay this job
human_decision is empty or merge	Block	The agent does not own merge
Tests arrived in the same agent burst	Hold	A human rewrites those tests
Replay valid and tests are human-owned	Review	You still read the diff

Notice merge is absent from allowed decisions. Review, reject, or isolate are the only honest exits. Auto-merge is how unreproducible patches escape.

Wire the gate into CI

Put the replay file next to the patch in the PR. Reject the PR when the checker exits non-zero. Reject the PR when hashes and diffs disagree.

# Example only: local gate before you open the PR.
python3 check_replay.py replay.json tools.allow
git rev-parse HEAD
git diff --binary "$BASE_SHA" > /tmp/agent.patch
python3 - <<'PY'
from pathlib import Path
import hashlib, json, sys
payload = json.loads(Path("replay.json").read_text())
raw = Path("/tmp/agent.patch").read_bytes()
digest = "sha256:" + hashlib.sha256(raw).hexdigest()
if digest != payload["patch_digest"]:
    print("PATCH_DIGEST_MISMATCH", digest)
    sys.exit(2)
print("PATCH_DIGEST_OK")
PY

Do not let the agent write this checker. You write the checker and you own it. The agent may propose patches, never the gate.

A debugging workflow when replay fails

When check_replay.py fails, stop generating more code. You are now in forensics, not in authoring. Follow this order and do not skip steps.

Order of operations

1. Confirm replay.json parses as UTF-8 JSON without comments.
2. Confirm task_hash matches a sha256 of the prompt file.
3. Confirm every tool name exists in the allowlist file.
4. Confirm each result_digest matches the saved artifact bytes.
5. Confirm patch hashes match git diff against the base sha.
6. Only then rerun the agent, with the same pinned model string.

If step six still drifts, you do not have replay yet. You have a non-deterministic tool or an unpinned runtime. Freeze the runtime before you blame the prompt.

What drift usually means

Drift usually means an unpinned tool, not a "creative" model. Network time, random files, and live clocks leak entropy. You stub those before you tune prompts again.

Record the stub, not the live socket. Live search and live HTTP are not replay sources. If a tool must touch the network, isolate that step from merge.

Limitations

This contract does not make a weak model strong. It does not prove the patch is correct, only reconstructable. Correctness still needs tests you wrote without the agent.

It will not capture GPU nondeterminism you refused to pin. It will not redact secrets you stuffed into prompts. It will not help if you never review the diff.

Floating-point tools, network calls, and clocks break naive replay. You must stub those tools or record their exact bytes. If you cannot stub them, isolate that step from merge.

Who should not use this approach

Do not use this as a permit for unattended production writes. Do not use this in regulated systems without a real audit log. Do not use this to rubber-stamp generated tests.

Skip the JSON file if you never invoke tools. A single-shot edit in your own editor needs a review, not theater. This gate is for looped agents with side effects.

If you cannot store transcripts safely, do not start. Prompts include secrets, customer names, and private URLs. You need a redaction step before any shared sandbox.

Close

You do not need a smarter agent to ship safer diffs. You need a replay file that survives a cold restart. If you cannot replay the agent, you do not merge.

Comments

[Raju Dandigam]

@airs_6907, the strongest implication here is that replay and reproduction are different. For live web tools and nondeterministic services, the honest merge artifact is often pinned inputs, tool schemas, recorded results, digests, and the resulting patch—not a promise that rerunning will recreate the outside world. I’d classify each step as replayable, stub-replayable, or evidence-only, then fail closed when a PR depends on a step whose evidence cannot be verified. That keeps the merge bar high without overstating what a transcript can reproduce.

[Raknaos]

Item 3 in your replay list is the one people skip: storing the tool call and result digest separately. We log full tool transcripts for a fleet of agents and the first incident that actually mattered was a diff that looked wrong but the transcript said the tool returned something the model then paraphrased incorrectly. Without the raw result blob we would have blamed the model.

One honest limitation we hit: replay gives you the record, but non-deterministic tools (anything that fetches live web state) still can't be re-executed faithfully. We treat the stored digest as evidence, not as a reproduction. Curious whether your checklist distinguishes those two.