FAQ: Five Myths About Proof in an Agent Loop

Did your agent finish with a confident smile today? Did any process actually assert on your tree?

I keep hearing the same five claims after demos. They show up right after cheap model drops.

People paste a transcript and call it CI. Then a hopeful merge happens on pure vibes.

This FAQ is a receipt drill, not a product tour. You can run the whole drill on a laptop.

Why green loops fool careful reviewers

Agents now call tools inside a tight loop. From far away that looks like a test runner.

Is a function call the same as a failing assert? The honest answer is not even close here.

A loop can stop because the model got bored. A real test stops on process exit codes.

Chat text is not a JUnit XML report. A cheerful thumbs-up is not git status either.

Want something closer to proof on disk? Capture a receipt before you trust green.

The five fields I refuse to skip

I want five fields written onto local disk. I do not want five chatty paragraphs.

• host and user from the executing machine
• cwd plus the value of git rev-parse HEAD
• the exact command vector that ran
• numeric exit code from the process
• a UTC timestamp for start and end

If any field is missing, the loop only told a story. You should treat that story as gossip.

A wrapper you can actually run

The script below is a labeled example. It is not a captured production run log.

#!/usr/bin/env bash
# loop_receipt.sh — labeled example, not a captured production run
set -euo pipefail

receipt_dir="${RECEIPT_DIR:-.loop-receipts}"
mkdir -p "$receipt_dir"
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
file="$receipt_dir/${stamp}.txt"

{
  echo "host=$(hostname)"
  echo "user=$(id -un)"
  echo "cwd=$(pwd)"
  echo "head=$(git rev-parse HEAD 2>/dev/null || echo NO_GIT)"
  echo "dirty=$(git status --porcelain 2>/dev/null | wc -l | tr -d ' ')"
  echo "cmd=$*"
  echo "started_utc=$stamp"
} > "$file"

set +e
"$@"
code=$?
set -e

{
  echo "exit=$code"
  echo "ended_utc=$(date -u +%Y%m%dT%H%M%SZ)"
} >> "$file"

echo "receipt=$file"
exit "$code"

Wrap the command your agent wanted executed. Keep the receipt file around afterward for review.

chmod +x loop_receipt.sh
./loop_receipt.sh python -m pytest -q
cat .loop-receipts/*.txt

No pytest in that tree right now? Use a tiny stand-in drill command instead.

./loop_receipt.sh python -c "import sys; sys.exit(2)"

You should see exit=2 in the receipt. If you only see prose, you have no proof.

Add the directory to gitignore before the next commit. Receipts are local evidence, not source code.

echo ".loop-receipts/" >> .gitignore

Myth 1: The model said pass, so tests passed

Here is the claim developers keep repeating in reviews. The transcript contains the words all tests passed.

What evidence usually exists for that sentence? You get a paragraph, maybe a pasted snippet.

Did you store the test runner's numeric exit code? Did you store the commit HEAD beside it?

The corrected mental model is blunt and useful. Prose is a claim, and exit is evidence.

If the agent summarized pytest output in chat, demand the receipt file. With no file, there is still no merge.

Would you merge on a Slack lgtm with no CI link? Then why merge on a smiling agent loop?

I also reject screenshots of a terminal with no command. A crop can hide the failing lines.

Myth 2: A tool call means the command ran here

Here is the second claim I see in pull requests. The tool named run_terminal_cmd touched your dirty tree.

Maybe the command ran in a workspace you never saw. Maybe that command never ran at all.

Free remote boxes make this confusion much worse. The host field suddenly decides the whole argument.

A tool name is not a hostname you can trust. Read host= and cwd= before you relax.

You get the same binary, different machine, different dirty files. That pattern is not a replay you can ship.

Who executed it, you or the box? Ask that question out loud during review.

Paste the host line into the PR description. Then the word local finally has a referent.

Myth 3: More tools in the loop means more verification

People treat a pile of tools like a suite. Search plus shell plus browser equals tests, they still say.

Those tools are actuators, not oracles at all. Who actually checks the output they just produced?

A loop can invoke pytest and ignore exit codes. Treat that ignored failure as a red run.

Count asserts that a program parsed, not tools. One ignored failure is still fully red.

Which tool's output was parsed by a program? Which tool's output was narrated by a model?

A browser screenshot is not an HTTP contract. A recursive grep is not a typechecker either.

Myth 4: I can replay the run from the chat log

Developers say the log is already a script. They copy a few commands and call it reproduction.

Chat logs skip cwd, env, and failed attempts. They also skip the file the model overwrote.

Then the model fixes the overwrite with more prose. You still cannot replay the original tree.

The log is a highlight reel with missing takes. The receipt is the take you can rerun.

If you cannot rerun cmd on head, you cannot replay. That is the whole myth in one line.

# labeled replay check — fill HEAD and CMD from a receipt
git switch --detach "$HEAD"
./loop_receipt.sh $CMD

Dirty working trees lie hard during replay drills. Detach first, and then talk about reproduction.

If dirty was not zero on the receipt, stop. You are arguing about an unpublished tree.

Myth 5: Free model plus free server means free proof

This claim appears whenever inference gets cheaper again. People say verification got cheaper in the same breath.

Cheaper tokens do not create extra exit codes. They mostly create more confident summaries of runs.

A free box can still lose the workspace tomorrow. A free model can still hallucinate a green run.

When I need a throwaway loop, I use free model access. A free server is enough to practice the drill.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. I use those for throwaway loops, then keep the receipt.

That is the whole product role here. The product does not replace exit on a runner.

Free compute is a sandbox, and proof is a file. Keep secrets off that box on purpose.

Do not rename a sandbox into CI. A spare box is for the diff, not for gating merges.

Decision table I keep beside the PR

I use this table when the chat sounds sure. Each row is a claim, a required field, and a fallback.

Claim in the chat	Field you must have	If missing, do this
tests passed	exit=0 from the runner	Re-run under loop_receipt.sh
I ran it locally	host and user	Do not merge
same commit	head matches the review SHA	Detach and replay
clean tree	dirty=0	Run git status yourself
reproducible	cmd plus head	Treat the demo as theater

Print the table next to the review comments. Review arguments get shorter when fields are missing.

A thirty-minute workflow you can copy

1. Pick one agent task you already trust too much.
2. Write the intended shell command on paper first.
3. Run that command through loop_receipt.sh on your laptop.
4. Run the same command on a throwaway server.
5. Diff the two receipt files field by field.
6. Only then let the model narrate what happened.

What should match across hosts for a real test? The cmd, exit, and head fields should match.

What may differ without killing the whole claim? The host and user fields may differ.

If exit differs, you do not have a test. You only have two competing stories then.

Need a spare box for step four this afternoon? A free server is enough to practice the diff.

Do not copy .env onto that box. Do not call the box your CI system.

Limitations

This receipt file is not cryptographic provenance at all. Anyone can fake a local text file.

It does not pin tool versions or environment variables. It will not stop a model lying about unwrapped commands.

It also will not turn a free sandbox into production. If you need attested builds, use real CI.

Keep this drill as a pre-check before review. Do not retire your required GitHub checks.

The wrapper also cannot see tools the agent called privately. Unwrapped calls stay outside the receipt on purpose.

Who should skip this FAQ

Skip this if required checks already gate every pull request. You already have a better oracle in CI.

Skip this if your agent cannot wrap a shell command. The wrapper would never see a real exit.

Skip this if you must not send code to a shared free server. Use the laptop path only in that case.

Skip this if you came here looking for a benchmark. I did not run one, on purpose.

This FAQ is for people merging on transcripts. That failure mode is the whole point.

What I want you to keep

A loop is only a process with tools. A test is an exit code on a commit.

A chat log is marketing copy with extra steps. Which artifact did you file for the merge?

Keep the receipt on disk after every demo. Question the green color in the chat transcript.

Ask who asserted it, the model or the shell? Then do not ship the smile anyway.