Replay Is Not Rerun: A Session-Receipt Glossary, a Four-Leaf Tree, and a Worked Example at Every Leaf

The patch arrived in Slack as a zip. Tests were green on the author's laptop. The author said an agent had finished the work on a spare machine overnight.

You asked for the starting commit. They sent another zip. No .git directory. No prompt file. Three chat bubbles in a screenshot. That object is not a rerun. It is a souvenir.

Cheap remote runtimes make this failure more common, not less. When a session is free to start, it is also free to forget. The missing artifact is not intelligence. It is a receipt.

This article is a classification tool. You get a glossary, a four-leaf tree, a worked example at every leaf, and a small checker. The checker does not grade the model. It grades whether you could defend the session in review.

The problem, stated as two records

A coding-agent session has two independent records. The first is the input bundle: the tree you started from, the prompt, the tool policy, and the environment claims. The second is the transcript: every model message and every tool result, in order.

Green tests after the fact reconstruct neither record. A later green run on your laptop only says the final tree happens to pass now. It does not say how the tree was produced, or whether you could produce it again.

Treat those two records as bits. Present or absent. That gives four leaves. Do not add a fifth leaf for “the model sounded confident.” Confidence is not a record.

Session-receipt glossary

Use these terms as defined here. Nearby English will lie to you.

1. Input bundle — A pinned starting state: a git SHA (or a tarball with a known hash), the exact prompt file, the tool-policy file, and an environment manifest (language version, OS family, package manager). If any of those four is missing, you do not have a bundle.
2. Transcript — An ordered log of model messages, tool names, tool arguments, and tool results. A screenshot of the last three bubbles is not a transcript.
3. Replay — Re-executing the recorded tool-call sequence against the pinned tree without new model sampling. Replay answers “did this path happen?” It does not answer “would the model do it again?”
4. Rerun — New sampling against the same input bundle. A rerun can diverge. Divergence is data. It is not a bug in the definition.
5. Receipt — The bundle plus a classification of the transcript: complete, partial, or absent. Attach the receipt to the pull request, not the chat vibe.
6. Orphan run — Files on disk with neither bundle nor transcript. Reviewers should treat orphan output as untrusted scratch.
7. Residue — Processes, virtualenvs, temp files, or secrets left on a remote server after the session. Residue is not a receipt. It is a cleanup problem.
8. Divergence point — The first tool call that differs between two reruns of the same bundle. Record the step index. Do not argue about “style.”
9. Pin — A content-addressed pointer: git rev-parse HEAD, plus hashes of the prompt and policy files. “Latest main” is not a pin.
10. Souvenir — Any human-facing leftover (screenshot, zip, “it said done”) that cannot reconstruct a pin or a transcript.

The four-leaf tree

Ask two questions, in this order.

Q1. Is the input bundle pinned?
Q2. Is the transcript complete?

Q1 pinned bundle?
├─ yes → Q2 complete transcript?
│         ├─ yes → Leaf A: Replay audit
│         └─ no  → Leaf B: Fresh rerun
└─ no  → Q2 complete transcript?
          ├─ yes → Leaf C: Forensic read-only
          └─ no  → Leaf D: Orphan scratch

Stop there. Do not invent a leaf for “the remote box was slow” or “the names were nicer.” Those are not evidence classes.

Shared setup for the four leaves

Label: unexecuted example. The task is small on purpose.

Repo at 9f3c1a2. File client.py has fetch_user(user_id) with no timeout. prompt.md says: add a five-second timeout, raise TimeoutError, do not change the return schema, do not edit tests. policy.json allows Read, Edit, and Bash(python -m pytest) only. No network install.

You will classify four endings of that same story.

Leaf A — Replay audit (bundle + transcript)

You have HEAD = 9f3c1a2, prompt.md, policy.json, env.txt, and transcript.jsonl with every tool result.

1. Verify the pin: git rev-parse HEAD matches the receipt.
2. Hash the prompt and policy: sha256sum prompt.md policy.json.
3. Walk the transcript. Confirm every Edit path is under the repo. Confirm no pip install or npm i slipped through.
4. Replay the edits onto a clean worktree at 9f3c1a2. Do not call a model.
5. Run the test command recorded in the transcript, not a new command invented at review time.

A replay is a patch applicator plus a log matcher. The matcher below is runnable. It only answers completeness, not correctness.

# replay_check.py — classify whether a transcript is mechanically complete
import json
import sys

REQUIRED_TOOL_KEYS = ("name", "args", "result")


def load_jsonl(path):
    rows = []
    with open(path, encoding="utf-8") as handle:
        for index, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            try:
                rows.append(json.loads(line))
            except json.JSONDecodeError as exc:
                raise SystemExit(f"transcript line {index}: {exc}")
    return rows


def complete_transcript(rows):
    if not rows:
        return False, "empty transcript"
    tool_rows = 0
    for index, row in enumerate(rows):
        if row.get("type") != "tool":
            continue
        tool_rows += 1
        missing = [key for key in REQUIRED_TOOL_KEYS if key not in row]
        if missing:
            return False, f"step {index}: missing {missing}"
    if tool_rows == 0:
        return False, "no tool rows"
    return True, "ok"


def main(transcript):
    rows = load_jsonl(transcript)
    ok, msg = complete_transcript(rows)
    print(f"complete={ok} detail={msg} steps={len(rows)}")
    return 0 if ok else 2


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Worked ending: the transcript shows Edit client.py, then Bash python -m pytest tests/test_client.py. Replay produces the same diff. You still review the diff. The receipt only proves the path is inspectable.

Leaf B — Fresh rerun (bundle, no transcript)

You have the pin. The chat UI discarded the log. This is common on disposable servers.

1. Recreate the worktree at 9f3c1a2.
2. Feed the same prompt.md and policy.json.
3. Capture a new transcript this time. If you cannot capture, stop. You are about to manufacture another souvenir.
4. Diff the new tree against the author's zip, if they still have one: diff -ru author_tree/ rerun_tree/.
5. Record the divergence point. If the first difference is an extra refactor in utils.py, the original zip is a different program, not a noisy rerun.

A rerun is evidence about stability of the bundle, not about the missing original path. Do not say “we reproduced it” unless the diffs match. Say “we sampled again from the same pin.”

git fetch --quiet
git switch --detach 9f3c1a2
git status --porcelain   # must be empty before the rerun
sha256sum prompt.md policy.json
# start the agent here, writing transcript.jsonl
diff -ru author_tree/ rerun_tree/ | head

If git status --porcelain prints anything, you do not have a pin. You have Leaf C wearing Leaf B's clothes.

Leaf C — Forensic read-only (transcript, no bundle)

You have a beautiful log. You do not have the SHA. The author was on dirty main.

1. Read the transcript. Extract every edited path and every shell command.
2. You may reconstruct a candidate patch. You may not claim it started from a known tree.
3. Refuse merge until someone produces a pin or rebuilds the bundle from a known tag.
4. If the transcript contains cat .env or git push, treat it as an incident, not a style issue.

Forensics can tell you what was attempted. They cannot tell you what else was already dirty in the worktree. Hidden local changes are an unrecorded input.

Proposal: extract paths only. Do not execute them on a machine that holds secrets.

# forensic_paths.py — read-only inventory of a transcript
import json
import sys


def paths_from(transcript):
    seen = []
    with open(transcript, encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            if row.get("type") != "tool":
                continue
            args = row.get("args") or {}
            for key in ("path", "file", "command"):
                if key in args:
                    seen.append((row.get("name"), key, args[key]))
    return seen


if __name__ == "__main__":
    for item in paths_from(sys.argv[1]):
        print("\t".join(map(str, item)))

Worked ending: the log shows an edit to client.py and a pytest command. The parent tree is unknown. You can read intent. You cannot replay.

Leaf D — Orphan scratch (neither)

A zip. Green tests on one laptop. A screenshot. Stop.

1. Do not merge.
2. Ask for a pin. If the author cannot get one, recreate the task from main with a receipt from step one of Leaf A.
3. Delete residue on any remote box you used: temp clones, virtualenvs, prompt files that contain customer names.
4. Do not estimate “how likely the agent was right.” Likelihood is not a leaf.

Orphan output can still be a useful sketch in a personal branch. It is not review evidence.

A receipt file you can attach

Keep this next to the PR. Label: example schema, not a vendor format.

{
  "task": "timeout on fetch_user",
  "prompt_path": "prompt.md",
  "policy_path": "policy.json",
  "bundle": {
    "git_sha": "9f3c1a2",
    "prompt_sha256": "replace-with-real-hash",
    "policy_sha256": "replace-with-real-hash",
    "env": {
      "python": "3.12",
      "os": "linux"
    }
  },
  "transcript": {
    "path": "transcript.jsonl",
    "complete": true
  }
}

Checker that maps files to a leaf. Runnable.

# leaf_from_receipt.py
import json
import os
import sys


def classify(bundle_ok, transcript_ok):
    if bundle_ok and transcript_ok:
        return "A_replay_audit"
    if bundle_ok and not transcript_ok:
        return "B_fresh_rerun"
    if not bundle_ok and transcript_ok:
        return "C_forensic_readonly"
    return "D_orphan_scratch"


def exists(path):
    return bool(path) and os.path.isfile(path)


def main(receipt_path):
    with open(receipt_path, encoding="utf-8") as handle:
        receipt = json.load(handle)
    bundle = receipt.get("bundle") or {}
    transcript = receipt.get("transcript") or {}
    bundle_ok = all(
        [
            bundle.get("git_sha"),
            exists(receipt.get("prompt_path")),
            exists(receipt.get("policy_path")),
        ]
    )
    transcript_ok = bool(transcript.get("complete")) and exists(transcript.get("path"))
    print(classify(bundle_ok, transcript_ok))
    print(
        "bundle_ok="
        + str(bundle_ok)
        + " transcript_ok="
        + str(transcript_ok)
    )


if __name__ == "__main__":
    main(sys.argv[1])

Wire it before you argue about the patch:

git rev-parse HEAD
git status --porcelain
sha256sum prompt.md policy.json
python leaf_from_receipt.py receipt.json
python replay_check.py transcript.jsonl

If porcelain is not empty, you do not have a pin. Dirty trees are Leaf C or D waiting to happen.

Where a free remote loop fits

You can practice Leaves A and B without buying a dedicated box. The constraint is discipline, not spend.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two facts matter here for one reason: they make reruns cheap enough that a team can refuse souvenirs and still move. Cheap is not the same as captured. If the session does not give you a transcript file and a git SHA, you are in Leaf D with nicer UX.

A reasonable use: clone at a pin, drop prompt.md and policy.json beside the repo, run the session, write transcript.jsonl, run leaf_from_receipt.py, then pull the diff back to your laptop for review. The review still happens locally. The remote box is a sampler.

If you only need a sandbox to rehearse that loop, the free server option is enough to try the receipt checker on a throwaway clone. Do not send production secrets there. Do not assume the box keeps residue for you. Copy the receipt off the machine before you log out.

Limitations, stated plainly

This tree does not measure model quality. Two Leaf A sessions can still implement the timeout with different internals. Review the diff.

Replay is only as faithful as the tool sandbox. If the transcript says tests passed, but the sandbox had network, you replayed a contaminated path.

Reruns of non-deterministic samplers diverge. That is expected. Use divergence as a signal that the bundle under-specified the task, not as proof that either run is “the real” one.

Who should not use this as a merge gate: teams that cannot pin dependencies; workflows that forbid sending code to a shared free server; work that requires bit-identical generation; and any task whose prompt includes credentials. Those belong on an isolated runner you control, with a secrets manager, not on a disposable session.

The checker scripts classify completeness. They do not detect a prompt that asks the agent to ignore the policy file. Policy bypass is a different tree.

Close

If the author cannot point to a SHA and a log, you do not have a rerun. You have a zip. Classify the leaf before you classify the code.

The four questions that matter are small. Was the bundle pinned. Was the transcript complete. Can you replay without sampling. Can you rerun with sampling and name the divergence point. Answer those, then argue about timeouts.