FAQ: Five Myths About Agent-Reported Test Passes

Did the agent really pass your tests tonight?
The chat looks green. The summary sounds sure.
I do not buy a smile in prose. I want a receipt.

Why this keeps biting people

Remote agent boxes feel like CI. They are not.
A free model can narrate a pass with perfect grammar.
Your shell can still return 1. Who did you believe?

I write this as a skeptical checklist. Not a victory lap.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option. I treat both as a scratch lab, not a merge gate.

The receipt I actually want

I want three numbers. Not a paragraph.

• process exit code
• a content hash of the tree
• a timestamp I control

If the model omits any of those, I assume theater.
Would you merge on theater? I would not.

Myth 1: The model said PASS, so tests passed

Agents love wrapping pytest in a bedtime story.
"All 41 tests passed" is easy to emit. Cheap, even.
Did pytest run? Did it finish? Did it exit 0?

Evidence you can collect tonight

Do not parse the chat for the word passed.
Parse a file the process wrote. Then stop talking.

# example: proposed local or remote receipt
set -euo pipefail
python -m pytest -q --tb=no > /tmp/pytest.out 2>&1
echo "exit:$?" > /tmp/pytest.receipt
wc -l /tmp/pytest.out >> /tmp/pytest.receipt
sha256sum /tmp/pytest.out >> /tmp/pytest.receipt
cat /tmp/pytest.receipt

Corrected mental model

The model is a narrator. The kernel is the witness.
A sentence is not an exit code. Ever.
If you cannot show the receipt file, you have a rumor.

Myth 2: A pasted exit code in chat is the real one

The agent can type exit: 0 after a failure.
Copy-paste is not a syscall. You know that, right?
So why trust a digit inside a markdown fence?

Evidence

Wrap the command yourself. Refuse model-authored status lines.

# example harness: verify_agent_run.py
import hashlib, json, subprocess, sys, time
from pathlib import Path

def run(cmd, cwd="."):
    t0 = time.time()
    p = subprocess.run(cmd, cwd=cwd, text=True,
                       capture_output=True)
    blob = (p.stdout + "\n" + p.stderr).encode()
    receipt = {
        "cmd": cmd,
        "exit_code": p.returncode,
        "seconds": round(time.time() - t0, 3),
        "stdout_sha256": hashlib.sha256(blob).hexdigest(),
        "stdout_bytes": len(blob),
    }
    Path("run_receipt.json").write_text(
        json.dumps(receipt, indent=2) + "\n"
    )
    return p.returncode

if __name__ == "__main__":
    sys.exit(run(sys.argv[1:]))

Run it like this. Notice who writes the JSON.

python verify_agent_run.py python -m pytest -q
cat run_receipt.json

If the model did not spawn that wrapper, ignore its claim.
Would you let a intern hand-write a JUnit XML? Same energy.

Corrected mental model

Status lives in a file your wrapper created.
Chat is commentary. Commentary can lie without malice.

Myth 3: The box installed it, so you have a lockfile

"I pip installed the extras. We are unblocked."
Unblocked for whom? For this boot? For tomorrow?
A free remote box is a rented kitchen. Pans walk away.

Evidence

Hash the lockfile before and after the agent touches deps.
If there is no lockfile, you do not have a build. You have weather.

# example: proposed dependency receipt
ls -l uv.lock poetry.lock package-lock.json pnpm-lock.yaml 2>/dev/null || true
sha256sum uv.lock 2>/dev/null || echo "NO_LOCKFILE"
python -c "import sys; print(sys.executable)"

Ask one rude question after every install step.
Did the agent edit the lockfile, or only the live environment?

Corrected mental model

A successful install is not a pinned graph.
The graph lives in the lockfile you commit.
The live environment is a rumor that dies on reboot.

Myth 4: A free remote box is a private CI runner

CI runners have isolation stories. Queues. Cache policies.
A shared scratch server has whatever landed last.
Did the previous session leave a .env? A node_modules?

I do not claim a specific vendor layout here.
I claim a habit. Assume neighbors. Assume leftovers.
Would you run secret tests on a cafe laptop? Then why here?

Evidence

Start every session with an inventory. Then decide.

# example: session inventory, not a security audit
pwd
id
env | awk -F= '{print $1}' | sort
ls -la
git status --porcelain || true
git rev-parse HEAD 2>/dev/null || echo "NO_GIT"

If you see files you did not create, stop.
Do not "just continue the agent loop." That is how leaks start.

Corrected mental model

Treat the box as disposable glassware.
Bring your own lockfile. Bring your own tests.
Leave with hashes, not with vibes.

Myth 5: A green transcript is merge permission

This is the expensive myth. It ships.
The PR description quotes the agent. Reviewers relax.
Nobody reruns the suite on a machine they own. Why not?

Evidence

Replay is the product. The transcript is a trailer.

# example: cheap local replay after a remote story
git diff --stat
git stash push -u -m "agent-tree" || true
git checkout -- .
python verify_agent_run.py python -m pytest -q
# compare run_receipt.json exit_code to the chat claim

If local replay disagrees, the remote story loses.
Always. No debate. No "but the model was confident."

Corrected mental model

Merge permission is a command you ran.
On a tree you hashed. On a machine you trust.
The agent can draft. It cannot bless.

Artifact: a one-page decision table

Use this before you believe a remote pass. Print it. Argue with it.

Claim in chat	What I check	If missing, I do
"tests passed"	run_receipt.json exit_code == 0	rerun via wrapper
"exit: 0" inside markdown	wrapper-authored file, not chat	discard the digit
"installed deps"	lockfile hash changed in git	restore and pin
"clean workspace"	git status --porcelain empty	inventory, then wipe
"safe to merge"	local replay receipt	block the PR

I keep the table next to the wrapper.
The wrapper is boring. Boring is the point.

A tiny workflow I actually follow

1. Inventory the box. Abort on surprise files.
2. Pin deps from a committed lockfile only.
3. Run tests through verify_agent_run.py only.
4. Commit the receipt or attach it to the PR.
5. Replay locally before any review ping.

That is the whole ritual. No dashboard required.
Can you do it on a free scratch server? Yes, as a drill.
Should you skip step 5 because the drill looked pretty? No.

Limitations, said plainly

This receipt proves a process exited.
It does not prove the tests were meaningful.
It does not prove isolation from other tenants.

It does not freeze a free server into dedicated CI.
Availability can change. I will not invent quotas here.
I will not invent hardware, model names, or duration claims.

Hashes do not catch flaky tests. Rerun policy still matters.
A wrapper can be deleted by a later tool call. Guard the file.
Stdout truncation can still hide a useful traceback. Keep artifacts small.

Who should not use this approach

Do not park production secrets on a shared scratch box.
Do not treat this as SOC2 evidence. It is a habit, not a control.

Skip it if you already have real CI on every push.
Use CI. Let the agent propose. Let CI swear.

Skip it if you cannot replay locally at all.
No laptop, no runner, no hash? Then you have no merge signal.

What I want you to argue with

Which myth still lives in your team chat?
The narrator, the fake exit, the missing lockfile?
The leftover workspace, or the merge-from-transcript move?

Pick one. Add a receipt file this week.
If you want a scratch place to practice the wrapper, MonkeyCode's free model access and free server option are one lab for that drill. Bring the inventory commands with you.

I still want the JSON. Not the smile.