FAQ: Five Myths About Replaying an Agent Run

You hit retry and the model claimed a fix. Did you actually replay the same experiment this time? You probably replayed a sentence, not an experiment.

This FAQ is about that gap in your evidence. It is not about prettier prompts.

Why replay keeps lying to us

Agent loops feel like scripts you can rerun. They are not pinned scripts. A script freezes inputs. An agent run borrows them, then improvises.

Free models make retries feel cheap. A free server makes them cheaper still. Cheap retries hide drift you never named.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I reach for MonkeyCode when a free model and a free server share one loop. The receipt workflow below still matters if you run elsewhere. The product is optional. The drift is not.

Myth 1: Same prompt means same experiment

Wrong. A prompt is one input among many. It is not the experiment.

The experiment also includes things you rarely paste:

• git SHA and dirty paths
• working directory at tool time
• files already written by prior tools
• env names the process actually saw
• which files the model read this pass

Did retry restore those, or only the chat box? If you never snapshot them, you did not replay. You started a cousin run with a familiar sentence.

Corrected mental model

Treat the prompt as a comment on the run. Treat the receipt as the test vector. No receipt, no honest claim of "same run."

Myth 2: The transcript is a replay button

The chat log is a story about work. It is not history | bash.

On retry the model may skip a tool call. It may read a different file. It may describe a patch that never reached disk. Can you replay a story? You can reread it. You cannot compile it.

Capture checkable lines instead

Keep the list boring and numbered:

1. git rev-parse HEAD
2. git status --porcelain
3. pwd and uname -srm
4. exact argv of each tool call
5. exit codes, not vibes
6. hashes of files you actually care about

If a line cannot be hashed, it is not evidence. It is narration.

# proposed: identity of the tree, not a novel
git rev-parse HEAD
git status --porcelain
pwd
uname -srm

Myth 3: Equal stdout means equal workspace

Stdout is theater. The workspace is the set. Two runs can both print OK. One wrote a lockfile. One did not.

Did you diff the tree, or trust the last line? Agents love trailing success text. Disk does not care about tone.

Tiny proposed check

# proposed: hash paths the run was allowed to touch
# run on a throwaway clone first
git ls-files -o --exclude-standard | sort | xargs -r sha256sum
git diff --name-only HEAD | sort | xargs -r sha256sum

Label that proposed until you try it. If the hashes move, OK lied to you. Would you merge a green line that cannot name a file?

Myth 4: Dirty git is the only drift

Clean trees still drift. Env drifts. Clocks drift. Caches drift. node_modules may be gitignored. The server still used it.

You do not know without a receipt. A clean git status answers one question only. It does not answer "same binary on PATH."

Env is part of the test

Do not dump secrets. Dump names and shapes.

# proposed: names only, never token values
env | awk -F= '{print $1}' | sort
command -v python3
command -v pytest
command -v node

If PATH changed, your "same pytest" is a different binary. That is not a myth. That is an ordinary Tuesday on a free box.

Myth 5: Free compute makes receipts optional

This one is social, not technical. Free models invite extra retries. A free server invites "just run it again." Cost drops. Discipline drops faster.

Is replay free? The compute might be. The confusion is not. I still catch myself skipping the receipt. Then I argue with a ghost that shares my prompt.

Corrected mental model

Free is a reason to log more, not less. Retries without receipts are new experiments wearing old names. Cheap is not the same as comparable.

Artifact: a run receipt you can diff

Below is a proposed collector. It writes run-receipt.json. It talks to your repo, not a vendor API. Copy it. Run it after a loop. Compare it next time you say "same prompt."

#!/usr/bin/env bash
# proposed workflow — review before you point it at real secrets
set -euo pipefail

out="${1:-run-receipt.json}"
root="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$root"

sha="$(git rev-parse HEAD 2>/dev/null || echo not-a-git-repo)"
branch="$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo n/a)"
dirty="$(git status --porcelain 2>/dev/null | wc -l | tr -d ' ')"

mapfile -t env_names < <(env | awk -F= '{print $1}' | sort)
env_json="$(printf '%s\n' "${env_names[@]}" | python3 -c 'import json,sys; print(json.dumps([l.strip() for l in sys.stdin if l.strip()]))')"

python3 - "$out" "$sha" "$branch" "$dirty" "$PWD" "$env_json" <<'PY'
import hashlib, json, os, sys, time
out, sha, branch, dirty, cwd, env_json = sys.argv[1:]
watch = []
for path in ("README.md", "package-lock.json", "go.sum", "Cargo.lock"):
    if os.path.isfile(path):
        h = hashlib.sha256(open(path, "rb").read()).hexdigest()
        watch.append({"path": path, "sha256": h})
receipt = {
    "schema": "agent-run-receipt/v1",
    "recorded_at_unix": int(time.time()),
    "git_sha": sha,
    "git_branch": branch,
    "dirty_path_count": int(dirty),
    "cwd": cwd,
    "env_names": json.loads(env_json),
    "watched_files": watch,
    "prompt_note": "store a prompt hash here; do not paste secrets",
}
open(out, "w").write(json.dumps(receipt, indent=2) + "\n")
print(f"wrote {out}")
PY

Hash the prompt. Do not paste the prompt if it holds tokens.

printf '%s' "$PROMPT" | sha256sum

Compare two receipts without a story

# proposed: compare_receipts.py
import json, sys

a = json.load(open(sys.argv[1]))
b = json.load(open(sys.argv[2]))

keys = ["git_sha", "git_branch", "dirty_path_count", "cwd"]
print("field\tA\tB\tmatch")
for k in keys:
    print(f"{k}\t{a.get(k)}\t{b.get(k)}\t{a.get(k)==b.get(k)}")

sa = {x["path"]: x["sha256"] for x in a.get("watched_files", [])}
sb = {x["path"]: x["sha256"] for x in b.get("watched_files", [])}
print("\nwatched files")
for p in sorted(set(sa) | set(sb)):
    print(p, sa.get(p), sb.get(p), sa.get(p) == sb.get(p))

na, nb = set(a.get("env_names", [])), set(b.get("env_names", []))
print("\nenv names only in A", sorted(na - nb)[:20])
print("env names only in B", sorted(nb - na)[:20])

No scores. No leaderboard. Just mismatches a human can read. If you need a one-liner after that:

python3 compare_receipts.py before.json retry.json

Decision table: retry or new experiment?

You observed	Receipt?	Call it
Same prompt, hashes match, env names match	yes	replay attempt
Same prompt, git SHA moved	yes	new experiment
Same prompt, lockfile hash moved	yes	new experiment
Same prompt, no receipt	no	anecdote
Stdout matched, watched files did not	yes	false pass
Dirty count dropped to zero after "fix"	yes	candidate success

Print the table. Argue with the table. Do not argue with the model. Which row is your last retry actually in?

A twenty-minute drill

Do this on a throwaway clone. Not on production credentials. Not on a repo with live tokens.

1. Write a failing test you already understand.
2. Run the collector. Save before.json.
3. Let the agent try one loop only.
4. Run the collector. Save after.json.
5. Diff with compare_receipts.py.
6. Re-run the same prompt once.
7. Save retry.json. Compare again.

What moved? If only recorded_at_unix moved, you got close. If the lockfile moved, you did not replay. You mutated the tree and reused a sentence.

Ask the awkward question out loud. Would you merge retry.json without reading it? If no, stop calling that click a replay.

What a mismatch usually means

Keep this list next to the table:

• git_sha changed: you are on another commit, period
• dirty_path_count changed: tools wrote, or you did
• cwd changed: the command is not where you think
• lockfile hash changed: the solver ran, not just the tests
• env names appeared: a wrapper injected a tool you did not pin
• env names vanished: the free server image is not your laptop

None of those prove the model "got worse." They prove the experiment moved. That is a kinder diagnosis than yelling at a transcript.

Limitations

This receipt is shallow on purpose. It does not freeze the model. It does not freeze hardware. It does not freeze network indexes.

It also does not record tool payloads. Those payloads can contain secrets. Keep them out of json sitting in the repo.

Unix time is in the file. That field will never match across retries. Ignore it on purpose when you compare. env names are not env values. Two boxes can share names and still differ in PATH order.

Watched files are an allow-list. Your bug may live in an unwatched path. I am not claiming bit-for-bit agent replay. That claim would be false, and I will not make it.

Who should not use this

Skip it if you are not running commands against a tree. Chat-only drafts do not need a disk receipt.

Skip it if policy forbids writing json next to the repo. Put the file outside. Or skip the file.

Skip it if you need cryptographic attestation of the run. This is a notebook, not a hardware module. Skip it if you will not read the diff. Unread receipts are clutter with a schema.

What I want you to keep

Replay is a claim. Claims need artifacts. The prompt is not the artifact. The transcript is not the artifact. A trailing OK is not the artifact.

A small json file is ugly. It is also honest. If you already pair a free model with a free server, generate the receipt on that box. MonkeyCode is one way to get that pairing. The comparison still works when you leave.

Now look at your last retry. Was it a replay? Or only a sequel with better confidence?