The Replay Packet: Turning an Agent Session Into Evidence a Reviewer Can Re-run

A patch is only reviewable when a reviewer can reproduce its evidence without the author sitting next to them. That sentence ended a pairing session that had stalled for forty minutes on a simple question: what exactly did the agent run before it declared the fix complete? The senior in the session refused to read the diff first, because a diff shows what changed while a session log shows what was believed. The outcome was a small artifact called a replay packet, and it changes the reviewer's job from trusting a narrator to re-executing a record.

The pairing session, and the four questions it started with

The scenario is common enough to be boring: an agent edits three files, adds one test, and reports success. The author then summarizes that work in a pull request body of six confident sentences. The senior treated the summary as a claim set rather than a description and asked four questions in sequence.

1. Which command produced the green result, and what was its exit code at each step?
2. Which of the six sentences in the PR body maps to a command that actually ran?
3. If the reviewer runs the same commands on a clean checkout, will the output match?
4. What happens to this review if the author is unavailable for the next two days?

Question four is the one that reframed everything. A review process that depends on the narrator being reachable is not a review process; it is a handoff with extra ceremony. The senior's position was that the author should deliver a record, not a performance.

Three dead ends the session walked into

The first proposal was to paste the full transcript into the PR. That failed immediately on size and on signal: the transcript ran past four thousand lines, most of it file reads, and a reviewer cannot execute a wall of text. The second proposal was to have the reviewer re-run the agent on the same prompt. That failed on determinism and cost, since the second run explores a different path and produces a different patch that nobody asked for. The third proposal was to trust the diff plus green CI, which sounds mature until someone notices the CI job ran the union of all tests rather than the specific discovery path the agent followed.

The session kept exactly one decision: every claim in the PR body must cite a numbered step in a machine-readable command log, and that log must replay cleanly in a fresh checkout. Everything else, including the transcript, was discarded as supporting material rather than evidence.

The decision kept: four parts of a replay packet

The packet has a command recorder, a normalizer that removes inevitable noise, a replayer that compares exit codes and output hashes, and a linter that refuses unsupported sentences. The format is deliberately dull, because dull formats survive contact with reviewers.

Step 1: record each command as one JSON line

A sourced shell function wraps every command the agent runs and appends one line to session.jsonl. The recorder stores the command, the exit code, a hash of normalized output, and a duration for human context only.

# tools/record.sh — source this before the session starts
LOG="${SESSION_LOG:-session.jsonl}"
__seq=$(wc -l < "$LOG" 2>/dev/null || echo 0)

rec() {
  local cmd="$1" start end code out sha
  start=$(date +%s%3N)
  out=$(eval "$cmd" 2>&1); code=$?
  end=$(date +%s%3N)
  sha=$(printf '%s' "$out" | python3 tools/normalize.py | sha256sum | cut -d' ' -f1)
  __seq=$((__seq + 1))
  python3 - "$__seq" "$cmd" "$code" "$sha" "$((end - start))" "$LOG" <<'PY'
import json, sys
seq, cmd, code, sha, ms, path = sys.argv[1:7]
with open(path, "a") as f:
    f.write(json.dumps({"seq": int(seq), "cmd": cmd, "exit": int(code),
                        "out_sha": sha, "duration_ms": int(ms)}) + "\n")
PY
  printf '%s\n' "$out"
  return $code
}

rec "pytest -q tests/test_router.py"   # seq 1: expected failure before the fix
rec "pytest -q tests/test_router.py"   # seq 2: expected pass after it

Step 2: normalize output before hashing it

Raw output never matches twice, because timestamps, temporary paths, and pytest durations move on every run. The normalizer strips those classes of noise and leaves the structural text intact, which is what a reviewer actually cares about.

# tools/normalize.py
import re, sys

RULES = [
    (re.compile(r"0x[0-9a-fA-F]+"), "0xADDR"),
    (re.compile(r"\d{4}-\d{2}-\d{2}T[\d:.]+Z?"), "<TS>"),
    (re.compile(r"/tmp/[^\s:]+"), "<TMP>"),
    (re.compile(r"\b\d+(?:\.\d+)?(?:ms|s|us)\b"), "<DUR>"),
]

def normalize(text: str) -> str:
    for pattern, replacement in RULES:
        text = pattern.sub(replacement, text)
    return "\n".join(l.rstrip() for l in text.splitlines() if l.strip())

if __name__ == "__main__":
    sys.stdout.write(normalize(sys.stdin.read()))

Step 3: replay in a clean checkout

The replayer reads the log, re-executes each command, and compares both the exit code and the normalized hash. Three verdicts are possible: PASS, DIVERGED, or UNRUNNABLE for an empty or malformed log. Notice that a step is allowed to have a non-zero exit code, because the failing-then-passing pair is often the most valuable evidence in the packet.

#!/usr/bin/env python3
# tools/replay.py
import hashlib, json, subprocess, sys
from tools.normalize import normalize

def out_hash(text: str) -> str:
    return hashlib.sha256(normalize(text).encode()).hexdigest()

def main(log_path: str, timeout: int = 300) -> int:
    rows = [json.loads(line) for line in open(log_path) if line.strip()]
    if not rows:
        print("VERDICT: UNRUNNABLE (empty log)")
        return 2
    diverged = 0
    for row in rows:
        proc = subprocess.run(row["cmd"], shell=True, capture_output=True,
                              text=True, timeout=timeout)
        ok = (proc.returncode == row["exit"] and
              out_hash(proc.stdout + proc.stderr) == row["out_sha"])
        diverged += 0 if ok else 1
        print(f"[{'ok' if ok else 'DIVERGED'}] seq={row['seq']} "
              f"exit={proc.returncode}/{row['exit']} {row['cmd']}")
    verdict = "PASS" if diverged == 0 else "DIVERGED"
    print(f"VERDICT: {verdict} ({diverged}/{len(rows)} steps differ)")
    return 0 if diverged == 0 else 1

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Step 4: lint the PR body against the log

The final piece is the part the senior cared about most, because it converts prose into citations. Sentences in the PR body carry markers such as [seq:2], and the linter rejects any marker that points at a missing step or at a step that exited non-zero without being described as an expected failure.

# tools/claims.py  — usage: python3 tools/claims.py PR_BODY.md session.jsonl
import json, re, sys

body = open(sys.argv[1]).read()
log = {r["seq"]: r for r in (json.loads(l) for l in open(sys.argv[2]) if l.strip())}
refs = {int(m) for m in re.findall(r"\[seq:(\d+)\]", body)}
missing = sorted(refs - log.keys())
unproven = sorted(s for s in refs & log.keys() if log[s]["exit"] != 0)
for s in missing:
    print(f"claim cites a step that never ran: seq={s}")
for s in unproven:
    print(f"claim cites a failing step; describe it as an expected failure: seq={s}")
sys.exit(1 if missing or unproven else 0)

A gated CI job then becomes three lines of workflow, and it fails the build on DIVERGED rather than on style. The check is cheap, deterministic for anything that does not touch the network, and readable by a reviewer who has never seen the agent.

- run: python3 tools/replay.py session.jsonl
- run: python3 tools/claims.py PR_BODY.md session.jsonl

When a replay packet is worth the overhead

Situation	Replay packet	Cheaper alternative
Patch changes runtime behavior across several modules	Yes	None that preserves evidence
Agent added or edited its own tests	Yes, the discovery path is the interesting part	Diff review only misses it
Docs, comments, or formatting only	No	Ordinary diff review
Steps depend on live third-party APIs	Partial, pin responses first	Recorded fixtures
Session log contains credentials or customer data	No, redact before recording	Manual review with a witness

Where the free model access and the free server option fit

Two parts of this workflow are annoying to provision, and that is the honest place where a hosted coding assistant earns its mention. The free model access is useful for one narrow job: turning a long session.jsonl into a first-draft claims table that a human then corrects, which is summarization rather than reasoning. The free server option matters for the reviewer side, because replaying a heavyweight environment on a laptop is exactly the step reviewers skip when they are busy.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability of free model access and a free server option is supplied by the operator rather than independently verified here; the outreach brief cites a ten-million-token allowance, and that number should be treated as provisional until the current terms are read on the project's own page. Free tiers move, quotas expire, and hardware specifications are not promised by this article. Anyone planning capacity around them should confirm the present terms first and keep a fallback runner in CI, because a review gate that depends on a promotional tier is not a gate.

A reasonable next step for a team already keeping a command log is to test the replay runner on the free server for one week, compare divergence reports against the local runs, and adopt it only if the two agree.

Limitations and who should skip this

Replay is not reproducibility. Commands that hit a network, a clock, or an unstable dependency will diverge for reasons unrelated to the patch, and normalizing those differences away can hide real bugs. Logs also become a liability when they capture tokens, personal data, or internal hostnames, so redaction has to happen at record time rather than at publish time. Finally, the packet proves what ran, not that what ran was the right thing to test, and no harness fixes a weak test suite.

Teams that ship documentation-only changes, solo maintainers with a fast deterministic CI pipeline, and anyone working under a policy that forbids transcript retention should not adopt this. The packet is for the middle ground: multi-file behavioral patches where the reviewer was not present, the session was long, and the author might be asleep when the questions arrive.