FAQ: Did the Agent Run the Tests, or Narrate Them?

Did the agent run the tests, or only narrate them?
Chat windows still reward a confident summary every time.
Process tables reward boring receipts over pretty adjectives.

I keep a short FAQ for this mix-up.
This is not another clever prompt trick.
You can check the model in ten minutes.

Why this FAQ exists

Agent loops look a lot like ordinary development now.
A model proposes a full shell command first.
A tool maybe runs that command next.

The chat then narrates a tidy result.
Where did the actual truth live then?
Did it live in argv, cwd, and status?

Teams blur those three layers every week.
They blur them hardest on cheap remote loops.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option.
That pairing makes another turn feel almost free.
It does not make a missing receipt true.

The corrected mental model

Treat every agent turn as three separate objects.

1. The proposal: text the model wanted to run
2. The invocation: bytes the tool actually executed
3. The observation: exit code, streams, and artifacts

If object two is missing, object three is fiction.
If object three has no hash, you cannot replay it.
If the suite never collected tests, green is a costume.

Myth 1: A command in chat means the command ran

The model can emit pytest -q as prose.
That line is not a real tool call.
That line is a suggestion wearing a prompt font.

Ask one rude question after every green claim.
Where is the tool payload for that command?

Look for these fields before you relax:

• a tool name
• a complete argv array
• a working directory
• a start timestamp
• an end timestamp

No payload means nobody actually ran pytest, only a draft.

# labeled example: inspect a tool log, not the chat
test -f .agent/tool.jsonl || echo "no tool log"
tail -n 5 .agent/tool.jsonl

Prose is only intent until a tool runs.
The JSONL from the tool is the execution.

Myth 2: A traceback in chat is a crash on the box

Models copy stack traces from context and training.
They also invent frames that look painfully local.
A path like /tmp/app/test_api.py feels fully earned.

Is that file on the server right now?
Did Python even start on that box?
Check the box itself, not the paragraph.

# labeled example: prove the frame exists
ls -l /tmp/app/test_api.py
python3 -c "from pathlib import Path; print(Path('/tmp/app/test_api.py').exists())"

If the file is missing, the traceback is literature.
If Python never launched, the crash is a costume.

A stack trace without a file is only a claim.
A file plus an exit code is evidence.

Myth 3: "Exit code 0" in the summary is process status

Narrators really love the number zero here.
Tool wrappers sometimes swallow a real process status.
Shells with set +e keep smiling anyway after failure.

Did you read status from the process table?
Or only from the model's closing sentence?

# labeled example: never trust a paraphrased status
set -o pipefail
pytest -q --tb=short
echo "pytest_exit=$?"

Capture PIPESTATUS whenever you chain output filters.
pytest | tail can hide a red suite.

Process status is an integer from wait(2).
It is not a vibe in a paragraph.

Myth 4: "All tests passed" means your suite ran

Empty collection is the quiet failure mode here.
A wrong directory is its louder cousin.
Cached bytecode also lies in smaller, meaner ways.

The chat says green with a straight face.
Pytest may have collected zero tests underneath.
That result is not success; it is a skipped map.

# labeled example: demand a collection count
pytest -q --collect-only | tee /tmp/collect.txt
pytest -q --tb=line | tee /tmp/run.txt

Then assert the numbers, not the adjectives.

# labeled example: parse collection, reject empty green
from pathlib import Path
import sys

text = Path("/tmp/collect.txt").read_text()
if "test session starts" not in text:
    sys.exit("pytest never started")
if "0 tests collected" in text:
    sys.exit("empty suite is not green")

Ask these questions before you merge anything:

• How many tests got collected?
• Which directory was cwd?
• Which marker expression actually ran?
• Did -k filter the whole room?

Green always needs a denominator you can count.
With no count, you have no claim.

Myth 5: A cheap loop makes verification optional

Free model access lowers the cost of another turn.
A free server lowers the cost of another shell.
Neither one stamps a receipt for you.

Cheap iteration is useful during early exploration.
Cheap trust is not useful during review.

Price does not merge proposal, invocation, and observation.
Skip the log because the box was free?
You also skip the only audit trail.

Cost and truth remain independent axes here.
Pay nothing for the loop if you want.
Still record everything you plan to claim.

Artifact: a command receipt you can replay

Do not argue with the chat transcript alone.
Wrap the command and keep the wrap.

This is a proposed workflow on your box.
Treat the script as an example, not a benchmark.

#!/usr/bin/env bash
# receipt.sh — labeled example, unexecuted here
# usage: ./receipt.sh -- pytest -q --tb=short
set -euo pipefail

if [[ "${1:-}" != "--" ]]; then
  echo "usage: $0 -- <command>..." >&2
  exit 2
fi
shift

mkdir -p .receipts
stamp=$(date -u +%Y%m%dT%H%M%SZ)
id="${stamp}-$$"
out=".receipts/${id}.json"

start=$(date -u +%s)
set +e
"$@"
status=$?
set -e
end=$(date -u +%s)

python3 - "$out" "$status" "$start" "$end" "$@" <<'PY'
import json, os, sys, hashlib, pathlib
out, status, start, end, *argv = sys.argv[1:]
cwd = os.getcwd()
env_keys = ["PATH", "VIRTUAL_ENV", "PYTEST_ADDOPTS", "HOME"]
env = {k: os.environ.get(k, "") for k in env_keys}
blob = json.dumps({"cwd": cwd, "argv": argv, "env": env}, sort_keys=True)
digest = hashlib.sha256(blob.encode()).hexdigest()[:16]
record = {
    "id": pathlib.Path(out).stem,
    "cwd": cwd,
    "argv": argv,
    "env": env,
    "exit": int(status),
    "start_unix": int(start),
    "end_unix": int(end),
    "input_sha": digest,
}
pathlib.Path(out).write_text(json.dumps(record, indent=2) + "\n")
print(out)
PY

exit "$status"

Make it executable, then drive pytest through it.

chmod +x receipt.sh
./receipt.sh -- pytest -q --tb=short
ls -l .receipts | tail

Now the agent can still talk in chat.
You then compare that talk against .receipts/.

Review table

Chat claim	Evidence required	If missing, treat as
"I ran pytest"	argv in a receipt	proposal only
"Tests passed"	exit 0 plus collected > 0	unknown
"Fixed the traceback"	file exists and suite reran	story
"Same environment"	PATH and VIRTUAL_ENV in receipt	guess
"Took two seconds"	start_unix and end_unix	filler

Print one receipt and then read it slowly.
Does the argv match the sentence in chat?
Does cwd match the service you meant to test?

# labeled example: reject a green story with a red file
python3 - <<'PY'
import json, glob, sys
files = sorted(glob.glob(".receipts/*.json"))
if not files:
    sys.exit("no receipts; no claim")
row = json.load(open(files[-1]))
print(row["argv"], "exit", row["exit"], "cwd", row["cwd"])
if row["exit"] != 0:
    sys.exit("last receipt is red")
PY

Labeled example: chat versus receipt

Chat says: "I ran pytest and everything passed."
Receipt says cwd is /tmp and argv is ["pytest"].
Exit is 0. Collect log says 0 tests collected.

Do you ship that result to main?
No. You ship a denominator, not a smile.

False greens that show up in traces

Watch these substitutes for a real suite:

• pytest launched with no tests directory
• python -m pytest started from /
• -k not slow dropping the only regression
• exit 0 from a wrapper around a skipped module
• running ruff and calling it the test suite
• a second run that reused a dirty cache

Each one can look green in a paragraph.
Each one fails the receipt plus collection check.

A ten-minute check you can run

Use this sequence after any agent claims a green build.

1. Find the last receipt, not the last paragraph.
2. Confirm cwd is the service directory.
3. Confirm argv includes the suite you meant.
4. Confirm collected tests is not zero.
5. Confirm exit is an integer from the wrapper.
6. Only then read the chat summary.

Skip a step and you are reviewing fiction.

What this does not prove

A receipt is not a real security boundary.
It is not a production deploy either, period.
It is not a flake detector for hidden races.

It will not stop a model from lying in prose.
It only makes the lie expensive to miss.

Which readers should skip this wrapper approach?

• People who already gate merges on CI artifacts
• People who never let an agent shell out
• People running one-off snippets in a notebook

If CI already stores junit XML, start there.
Do not duplicate a weaker local log instead.

Limitations of this first wrapper:

• It does not snapshot the full environment
• It does not hash test artifacts yet
• It trusts the local clock
• It will not catch a mocked pytest shim

Extend it if you need those teeth.
Do not pretend the first version is complete.

How the cheap loop still helps

Iterate on the wrapper with a free model.
Run the wrapper on a free server.
That pairing is convenient for tight loops.
It is not magical, and it is not proof.

The model can draft receipt.sh in minutes.
The free server can execute that draft immediately.
You still open the JSON file yourself.

If the model forces exit zero, the cheat is visible.
The next human command can cat the file.
Ask the agent to hide the file.
You just learned the real threat model.

The loop stays useful when it stays inspectable.

Closing

So did the agent actually run the tests?
Please show me the receipt file before adjectives.
Also show me the test collection count next.
Then show me the integer exit status.

With no file, there is no claim.
Already on a free model and free box?
Start a .receipts/ directory tonight, not tomorrow.
Then argue with bytes, not with confidence.