FAQ: Did the Shell Run, or Did the Model Type?

Did the test suite actually run on a machine? Or did the model only narrate a passing pytest log?

I keep hitting this mix-up in review threads. A chat pane shows green dots and a zero exit. Then CI fails on the same commit hash.

Why this FAQ exists

Coding agents blur two jobs that used to be obvious. One job is generating tokens that look like a log. The other job is starting a real process with a pid.

Those jobs can line up on a good day. They can also drift apart without a red flag.

I treat a transcript as prose until a receipt appears. A receipt looks boring on purpose, and that is the point.

I sometimes run the receipt on MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The receipt, not the chat copy, decides if a process ran.

Claim: visible pytest output means pytest ran

What claim keeps getting repeated in agent chats? "I saw the dots, so the suite ran."

You hold tokens in a scrollback buffer, not a pid. That gap is the entire debugging problem here.

A model can emit a perfect TAP log from memory. It can also emit a log after a real run. Those two strings can look identical in the pane.

Correct the mental model before you merge tonight. A process is an OS object, not a paragraph. Ask for pid, cwd, hostname, and exit code as fields.

If the agent cannot return those fields from a host, you got narration. Narration can still help you draft the next patch.

Probe you can run yourself

This is an example script, not a published benchmark. Run it on the host you think executed the tests.

import os, socket, sys
print("hostname=", socket.gethostname())
print("cwd=", os.getcwd())
print("pid=", os.getpid())
print("executable=", sys.executable)
print("argv=", sys.argv)

No pid from that host means nobody ran pytest there. Do not argue with missing fields.

Claim: a zero in chat is a CI result

Is that zero from a runner you can name? Or is it a number inside a sentence?

CI stores a job id, a runner label, and artifacts. Chat stores a digit next to some prose. Those two systems do not share any database.

Copying 0 from a transcript into a PR is theater. Reviewers cannot replay theater on the merge commit.

Treat a chat exit code as untrusted until a receipt sits beside it. The receipt must include a hostname and a nonce you chose. Your CI system remains the only green check that counts for merge.

Bind a nonce you control

Do not let the agent invent the nonce. You mint the nonce on your own machine. The candidate machine must echo that nonce back.

NONCE="$(python3 -c 'import secrets; print(secrets.token_hex(8))')"
echo "$NONCE" > /tmp/run-receipt.nonce
echo "nonce=$NONCE"
echo "hostname=$(hostname -f 2>/dev/null || hostname)"
echo "pwd=$(pwd)"
echo "date_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "exit=$?"

A missing nonce match means no run happened. A matching nonce means that host ran that argv.

Claim: a server install lands on your laptop

The agent ran npm install, so you think the tree is ready. But which disk is actually ready now?

A free server has one filesystem of its own. Your laptop has a second, separate filesystem. The chat pane has no filesystem at all.

Package trees do not teleport because a model named a dependency. Name the host that grew node_modules before you celebrate.

If that host is a scratch box, your laptop still needs its own install. If CI must install, commit the lockfile from your clone. Do not ship node_modules from a throwaway server.

That tarball is not your shipping product. It is a side effect on one disk.

echo "host=$(hostname)"
echo "node=$(command -v node || echo missing)"
echo "python=$(command -v python3 || echo missing)"
echo "npm_root=$(npm root 2>/dev/null || echo missing)"
ls -ld node_modules 2>/dev/null || echo "node_modules=absent"
ls -ld .venv 2>/dev/null || echo "venv=absent"

Ask the agent to run that block. Compare the host= line with your laptop hostname. A hostname mismatch means you are in two worlds.

Claim: the log clock is your laptop clock

The log says 14:02, so you just ran it. But whose 14:02 does that log show?

Models often invent clocks from training patterns. Servers have clocks, and laptops have clocks too. Those three drift, and they lie in different ways.

Trust a timestamp only with timezone, hostname, and your nonce. UTC is less ambiguous than a casual now.

from datetime import datetime, timezone
import time, socket
print("utc=", datetime.now(timezone.utc).isoformat())
print("unix=", time.time())
print("tz=", time.tzname)
print("host=", socket.gethostname())

If the agent returns a pretty time without the nonce, ignore the time. A pretty clock is not proof of execution.

Claim: pasting the thread proves tests passed

You pasted the chat, so reviewers can see the pass. Can they replay that pass on the merge commit?

Reviewers only hold a screenshot of prose. They cannot rerun that prose on demand. Later turns can also rewrite the story.

Proof for a PR is a replayable job. Point at the CI URL, or attach receipt.json as an artifact. If you only have a transcript, say that plainly.

"The agent claims tests passed" is the honest sentence. Saying tests passed is not honest yet.

Artifact: a run-receipt workflow

Use this when an agent says a command ran. It is a method, not a vendor scorecard. I am not publishing timings or pass rates here.

1. Mint the nonce on your side

python3 -c 'import secrets; print(secrets.token_hex(8))'

Keep that value in your own notes. Do not paste a nonce the model invented.

2. Demand a fixed receipt shape

Refuse extra storytelling around these eight fields.

• nonce
• hostname
• cwd
• pid
• executable
• utc
• argv
• exit

3. Run a tiny script on the candidate host

Save this file as receipt.py on the host. Treat it as an example you execute yourself.

#!/usr/bin/env python3
"""Run receipt helper. Example only. Not a benchmark."""
import json, os, socket, sys
from datetime import datetime, timezone

nonce = sys.argv[1] if len(sys.argv) > 1 else "MISSING_NONCE"
doc = {
    "nonce": nonce,
    "hostname": socket.gethostname(),
    "cwd": os.getcwd(),
    "pid": os.getpid(),
    "executable": sys.executable,
    "utc": datetime.now(timezone.utc).isoformat(),
    "argv": sys.argv,
}
print(json.dumps(doc, indent=2))

python3 receipt.py "$NONCE"
echo "exit=$?"

4. Read the result with a table

What the chat shows	Extra evidence	Call it
pytest-like prose, no nonce	none	narration
exit=0 with no hostname	none	narration
nonce match plus pid	receipt.json	process on that host
nonce match on a free server	receipt.json	ran there, not on your laptop
nonce match inside CI logs	job URL	merge evidence
receipt from your laptop cwd	local dirty tree	local only

5. Map the hostname to a purpose

• Laptop host: fine for scratch. Not merge proof.
• Free server host: fine for a canary command. Still not CI.
• CI runner host: the only merge gate I will accept.

Do not chain those hosts in your head. Say the hostname out loud every time.

What a fake log looks like

The next block is only an illustrative fake log. It is not captured from a vendor.

======================= 47 passed in 3.14s ========================

That line has no nonce, no pid, and no host. I treat it as fiction until a receipt appears.

What a receipt does not prove

A receipt proves a process started on one host. It does not prove the tests are correct.

It does not prove the suite is complete. It does not prove a free server matches production. It does not prove the model understood the failure.

Flaky tests still flake after a pretty JSON blob. A missing env var still misses the same way. Wrong cwd still skips half the suite.

Who should skip this approach

Skip this if you already have a locked-down CI path. You do not need a chat receipt to merge.

Skip this if you cannot run any command on a host. A model-only session cannot mint a pid.

Skip this for any production deploy path. A free server is the wrong target for that work.

Skip this if you need formal attestation. The receipt script is only a debugging aid. It is not signed build provenance at all.

Limits I will not paper over

I did not attach quotas, hardware, or model names here. Those claims go stale, and I will not invent them.

I also did not claim a free server keeps files after you leave. Persistence belongs to a different debugging problem.

This method adds a minute of friction on purpose. That friction is the feature, not a bug.

The corrected picture

1. Chat text is a story about a command.
2. A process is an OS object on one host.
3. A nonce binds the story to that object.
4. Your laptop, a free server, and CI are three hosts.
5. Only CI is merge evidence.

Still tempted to paste dots into the PR? Then ask which hostname printed those dots.

If you try the receipt on a free server, paste the JSON fields. Leave the narrative sitting in the chat.