Did your agent really finish the job this time?
Or did the remote box just go quiet instead?
Those two outcomes look identical in a chat window.
I keep hearing the same five claims on reviews.
They sound like careful engineering rather than hope.
They are mostly comfort stories with extra syntax.
This FAQ treats each claim as a testable myth.
I want evidence, not a confident last message.
Then I want a mental model you can reuse.
What this FAQ is not
This is a verification habit, not a product tour.
I sometimes canary prompts on free model access.
I sometimes run that loop on a free remote server.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option.
I treat both as scratch capacity for unsigned canaries.
I do not treat them as a signed runtime contract.
I will not name hosted models in this FAQ.
I will not invent quotas, hardware, or benches.
If a number is missing from your logs, drop it.
Myth 1: One green agent run is a contract
Claim: The agent said done, so the prompt is safe.
Why it spreads: Chat UIs reward a tidy, final sentence.
Humans hate leaving an expensive loop visibly open.
Evidence to collect: Two fingerprints, not one transcript.
Same checks, same repo SHA, two different moments.
Would your CI accept a screenshot of a terminal?
Mine would not accept that as a release signal.
Neither should a chat ending that merely looks calm.
Better model
A single agent pass is only a sample.
Samples can inform you. They do not bind you.
Write the fingerprint down before you quote "done."
Myth 2: The free box PATH matches CI PATH
Claim: The tools resolved, so the toolchain matches prod.
Why it spreads: command -v feels like hard proof.
It only proves something exists on this box now.
Evidence to collect: Hash PATH. Record interpreter versions.
Compare those strings with the CI image, not memory.
Ask this: which python3 did the agent actually invoke?
Then ask which python3 CI will invoke tomorrow morning.
Those answers diverge more often than people admit.
command -v python3
python3 -V
readlink -f "$(command -v python3)" || true
command -v node; node -v 2>/dev/null || true
printf '%s\n' "$PATH" | tr ':' '\n' | nl
printf '%s' "$PATH" | sha256sum
Better model
A resolved binary is local luck, not parity.
Pin the image, or admit the drift in writing.
Do not let a helpful agent hide the mismatch.
Myth 3: Tool JSON means the side effect landed
Claim: The tool returned JSON, so the file exists.
Why it spreads: Agents narrate success in structured blobs.
Narration is not stat, and JSON is not a write.
Evidence to collect: The bytes on disk, right now.
The git index. The process exit, not the prose.
Did the patch land, or did the model describe one?
Those are different events with similar English vocabulary.
Your eyes will prefer the story. Distrust that preference.
git rev-parse HEAD
git status --porcelain=v1
stat -c '%n %s %y' src/app.py tests/test_app.py 2>/dev/null || true
Better model
Treat every tool payload as an unverified claim.
Verify claims with the filesystem and with git.
If stat fails, the transcript does not get a vote.
Myth 4: A retry is the same experiment
Claim: Hit run again and you reproduced the pass.
Why it spreads: The prompt text did not change at all.
The box, the clock, and the worktree probably did.
Evidence to collect: Fingerprint before and after the retry.
If the hash moved, you ran a brand new experiment.
Was the temp directory empty the second time around?
Did a partial write survive the first failed attempt?
Retries feel cheap. They are not identical trials.
# proposed commands — label them unexecuted until you run them
date -u +%Y-%m-%dT%H:%M:%SZ
ls -la /tmp | sed -n '1,20p'
git status --porcelain=v1 | wc -l
Better model
Retries are new runs with leftover gravity.
Name them as such in the notes you keep.
Never file two fingerprints under one experiment id.
Myth 5: The session is a durable team brain
Claim: The model remembers the repo across sessions.
Why it spreads: Long chats feel like institutional memory.
A free session is not your wiki, runbook, or ticket.
Evidence to collect: What already lives in git today.
What lives in a file somebody wrote on purpose.
Can a teammate recover state without that chat window?
If not, you do not have memory. You have fog.
Fog does not survive a laptop reboot or a new box.
Better model
Chat is volatile working memory, nothing more.
Git is the contract. Files are the only evidence.
If it is not committed, it is not team knowledge.
The artifact: unsigned-run checks
Do not trust my adjectives about any of this.
Run a fingerprint, then a side-effect checker, twice.
Label this whole section as a proposed workflow.
I am not reporting production timings or pass rates.
Copy the files. Then run them on your own box.
Step 1: fingerprint the box
Save this as fingerprint_agent_box.sh.
Keep the output boring, line-oriented, and diffable.
#!/usr/bin/env bash
# Proposed environment fingerprint for an agent scratch box.
# Label: unexecuted example. Adapt paths before you run it.
set -euo pipefail
out="${1:-./agent-fingerprint.txt}"
{
echo "utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "host=$(hostname)"
echo "pwd=$(pwd)"
echo "user=$(id -un)"
echo "kernel=$(uname -srm)"
echo "python=$(command -v python3 || true)"
echo "python_v=$(python3 -V 2>&1 || true)"
echo "node=$(command -v node || true)"
echo "git=$(command -v git || true)"
echo "git_head=$(git rev-parse HEAD 2>/dev/null || echo none)"
echo "dirty_count=$(git status --porcelain=v1 2>/dev/null | wc -l | tr -d ' ')"
echo "path_hash=$(printf '%s' "$PATH" | sha256sum | awk '{print $1}')"
} > "$out"
echo "wrote $out"
Run it twice. Diff the two files without mercy.
chmod +x fingerprint_agent_box.sh
./fingerprint_agent_box.sh /tmp/fp1.txt
# ... agent does work, or you think it did ...
./fingerprint_agent_box.sh /tmp/fp2.txt
diff -u /tmp/fp1.txt /tmp/fp2.txt
If path_hash moved, your toolchain moved with it.
If git_head moved, you tested two different commits.
If dirty_count moved, the worktree is the real story.
Step 2: make the agent emit a manifest
The agent should emit a manifest, not a vibe.
Use this shape. Keep the keys boring and strict.
{
"claimed_files": ["src/app.py", "tests/test_app.py"],
"claimed_exit": 0
}
Tell the agent to write claimed.json before it stops.
No manifesto, no "done." That rule is the whole point.
Step 3: verify claimed side effects
Then run this proposed checker against that file.
It does not grade the patch. It grades the claim.
#!/usr/bin/env python3
"""Proposed side-effect checker. Unexecuted example. Adapt before running."""
from __future__ import annotations
import json
import sys
from pathlib import Path
def main(manifest_path: str) -> int:
data = json.loads(Path(manifest_path).read_text(encoding="utf-8"))
claimed = data.get("claimed_files") or []
claimed_exit = data.get("claimed_exit")
if claimed_exit not in (0, "0"):
print("claimed_exit was not 0; treat as failed run")
return 2
missing = [rel for rel in claimed if not Path(rel).is_file()]
if missing:
print("missing files the agent claimed:")
for rel in missing:
print(f" {rel}")
return 1
print("all claimed files exist on disk")
return 0
if __name__ == "__main__":
if len(sys.argv) != 2:
print("usage: check_claimed_files.py claimed.json")
raise SystemExit(64)
raise SystemExit(main(sys.argv[1]))
python3 check_claimed_files.py claimed.json
echo "checker_exit=$?"
If the checker exits non-zero, the transcript lied.
The chat can still look friendly after that lie.
Ignore the friendliness. Keep the exit code.
Step 4: a decision table you can print
| Claim you heard | Minimum evidence | If evidence is missing |
|---|---|---|
| One green "done" | Two fingerprints | Sample, not a contract |
| PATH is "the same" | PATH hash plus versions | Drift, not parity |
| Tool JSON success |
stat plus git porcelain |
Claim only |
| Retry reproduced it | Unchanged fingerprint | New experiment |
| Session remembers | File committed in git | Fog |
Print the table. Stick it near the terminal.
I mean that part literally, on paper or a wiki.
A table beats another pep talk in the chat.
A loop worth copying, not quoting
This is a procedure, not a published benchmark.
I am not claiming I timed it in a shared lab.
- Fingerprint the box before the agent starts.
- Give the agent a manifest path, not a vibe.
- Let it work. Do not coach the final sentence.
- Fingerprint again. Diff. Then run the checker.
- Only then read the chat for clues and gaps.
Why that order, instead of reading the chat first?
Because the chat will bias your eyes immediately.
Should you skip the diff when you are late?
That is how unsigned runs become team folklore.
Limitations
This habit does not pin a hosted model for you.
Free model access can change without a local tag.
This habit does not freeze a free server image.
I am not claiming hardware, uptime, or duration.
The scripts do not prove correctness of the patch.
They only catch a "done" that left no files.
Clock skew can still fool naive local timestamps.
Use date -u, then compare with your laptop clock.
I have not published numbers from a shared fleet here.
If you need numbers, collect them on your own box.
Do not import someone else's adjective as a metric.
Who should not use this approach
Do not send production secrets to a scratch box.
A free server is not your vault or your HSM.
Do not use this for regulated release evidence.
Auditors want pinned images and pinned model ids.
This FAQ will not satisfy that bar, and should not.
Do not use this if you cannot inspect the worktree.
If you only have the chat, you have nothing yet.
Skip it if your agent cannot write a manifest file.
Fix that interface first. Then fingerprint the box.
What I want you to remember
"Done" is a sentence with no signature attached.
A contract is a fingerprint plus files plus git.
Did the agent finish, or did the box just stop?
Make the box sign the work before you believe it.
If this FAQ missed a claim you still hear, name it.
Top comments (0)