FAQ: The Agent Said Done. What Was the Exit Code?

The agent typed done in the last line. Did the process behind that sentence actually succeed?

I treat that cheerful sentence as a hypothesis now. Chat text and shell integers live in different layers.

Why this FAQ exists

A free model can narrate a perfect run. A free server still returns an integer.

Do you merge those layers into one story? That merge hides failed tests.

Chat text is a summary. The shell is a process. They can disagree without drama.

MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pairing is convenient for a cheap loop. It still does not merge those two runtimes.

This FAQ remains useful if you strike that name. Capture exit codes outside the prose.

The mental model worth stealing

Keep three buckets. Do not pour them together.

1. The model proposes text.
2. The server starts a process.
3. The chat paraphrases both.

Which bucket produced the word done? Ask that every time.

If you cannot point at an exit code, you do not have a result. You only have a story.

FAQ: If the chat says done, did the process succeed?

Claim

A closing done means exit code zero. The vibe itself is treated as proof.

Evidence you can collect

Ask the shell, not the paragraph. Wrap the command before anyone summarizes it.

Treat the next script as a proposed logger. It is not a CI system.

#!/usr/bin/env bash
# record_cmd.sh — proposed logger, not a CI replacement
set -u
log_dir="${RECORD_LOG_DIR:-./.cmd-records}"
mkdir -p "$log_dir"
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
id="${stamp}-$$"
stdout_file="$log_dir/$id.stdout"
stderr_file="$log_dir/$id.stderr"
meta_file="$log_dir/$id.meta"

"$@" >"$stdout_file" 2>"$stderr_file"
code=$?

{
  printf 'id=%s\n' "$id"
  printf 'cwd=%s\n' "$(pwd)"
  printf 'argv=%s\n' "$*"
  printf 'exit=%s\n' "$code"
  printf 'utc=%s\n' "$stamp"
} >"$meta_file"

echo "recorded $id exit=$code" >&2
exit "$code"

Run a deliberate failure. Then read the meta file.

chmod +x record_cmd.sh
./record_cmd.sh python3 -c 'raise SystemExit(1)'
cat .cmd-records/*.meta
echo "wrapper=$?"

Did the chat mention exit=1? Or did it say it handled things?

Corrected mental model

The word done is punctuation inside a paragraph. Zero is a number the kernel reported.

Only that number counts as a result. Prose cannot replace wait.

FAQ: Is the fenced block what actually ran?

Claim

The markdown fence is a transcript of the process. What you read is what ran.

Evidence you can collect

Fences get edited for readability. Shells do not perform that cleanup.

Quotes move around. cd lines vanish. Long paths get shortened.

Compare three artifacts after any so-called quick fix.

• the fenced block in the chat
• the argv= line in your meta file
• ps or shell history, if the session still exists

# proposed checks; run them on the server, not in your head
ps -o pid,ppid,etime,cmd
fc -l -20 2>/dev/null || true
python3 - <<'PY'
from pathlib import Path
root = Path(".cmd-records")
for p in sorted(root.glob("*.meta")):
    print("====", p.name)
    print(p.read_text())
PY

Do those three strings match character for character? They often do not.

Corrected mental model

The fence is a draft the model can rewrite. argv is the evidence of execution.

If they disagree, believe the wrapper. Ignore the prettier block.

FAQ: Does a quiet chat mean quiet stderr?

Claim

If the model stayed calm, stderr was empty. No quote means no noise.

Evidence you can collect

Models truncate long streams. They also translate noise into soft vibes.

Fill stderr on purpose. Watch what the chat actually reports.

./record_cmd.sh bash -c 'echo ok; echo boom >&2; exit 2'
wc -c .cmd-records/*.stderr
tail -n 20 .cmd-records/*.stderr

Did the chat quote boom? Did it say minor warning instead?

Pipefail is another quiet trap. The last command can hide the first failure.

./record_cmd.sh bash -c 'set +o pipefail; false | true; echo inner=$?'
./record_cmd.sh bash -c 'set -o pipefail; false | true; echo inner=$?'
cat .cmd-records/*.meta

Which run did the chat call a success? Check exit= anyway.

Corrected mental model

Silence in chat is not silence on file descriptor two. Calm prose is not an empty stream.

Read the stderr file first. Ask the model second, if at all.

FAQ: Can a later turn reuse the same process?

Claim

I fixed it means the same PID recovered. The broken process waited for coaching.

Evidence you can collect

Most agent commands are new processes. Death after a non-zero status is normal.

A failed pytest does not sit around waiting. The next turn starts another process.

Maybe another cwd. Maybe another interpreter. Maybe another PATH.

./record_cmd.sh bash -c 'echo PID=$$; pwd; command -v python3'
./record_cmd.sh bash -c 'echo PID=$$; pwd; command -v python3'
python3 - <<'PY'
from pathlib import Path
metas = sorted(Path(".cmd-records").glob("*.meta"))
for p in metas[-2:]:
    print("====", p.name)
    print(p.read_text())
PY

Are the PIDs equal across those two turns? They should not be.

Did cwd change between the two meta files? Your chat may never stress that.

Corrected mental model

Each command is a new witness with a new PID. There is no rehab for a dead process.

A later done talks about a later process. Do not glue those stories together.

FAQ: Does exit zero mean the intended command ran?

Claim

Zero proves the tests ran. Green means the suite you meant.

Evidence you can collect

Zero only describes the process you actually spawned. The wrong argv can still succeed.

true exits zero. echo tests passed exits zero. Empty collection can exit zero.

./record_cmd.sh true
./record_cmd.sh bash -c 'echo tests passed'
./record_cmd.sh python3 -m pytest -q no_such_dir 2>/dev/null || true

Now plant one failing test. Wrap the real runner, not a description of it.

# test_exit_myth.py — proposed fixture, unexecuted until you run it
def test_truth():
    assert 1 + 1 == 3

./record_cmd.sh python3 -m pytest -q test_exit_myth.py
grep '^exit=' .cmd-records/*.meta | tail -n 1

Did the agent say the suite passed? Did exit= agree with that sentence?

Score the mismatch with a tiny checker. Label this unexecuted until you run it.

# score_chat_vs_exit.py — proposed scorer, not a shipped metric
from pathlib import Path

chat_claim = Path("chat_last_line.txt").read_text().strip().lower()
latest = sorted(Path(".cmd-records").glob("*.meta"))[-1]
meta = dict(
    line.split("=", 1) for line in latest.read_text().splitlines() if "=" in line
)
exit_code = int(meta["exit"])
chat_says_pass = any(w in chat_claim for w in ("done", "passed", "success"))
print("argv", meta.get("argv"))
print("exit", exit_code)
print("chat_says_pass", chat_says_pass)
print("mismatch", chat_says_pass and exit_code != 0)

printf '%s\n' 'All tests passed. Done.' > chat_last_line.txt
python3 score_chat_vs_exit.py

If mismatch prints True, the chat is not a runner. It is a narrator.

Corrected mental model

Exit zero is local to the argv you recorded. It is not a badge for the intended suite.

No wrapper, no result. A described run is not a run.

A one-hour decision table

Use this table after any agent run you might ship.

Question	If yes	If no
Do I have exit= on disk?	Trust the number	Re-run under the wrapper
Do argv and the fence match?	Review the real diff	Treat the fence as fiction
Is stderr empty in the file?	Continue	Read it before the chat
Was this a new PID?	Compare fresh outputs	Discard the recovery story
Did the intended runner start?	Keep the result	Zero came from the wrong command

Print the table. Stick it near your terminal. Yes, really.

Would you merge a PR because a coworker said looks good? You would still open CI.

Why is an agent different from that coworker? Because the prose is smoother?

A tiny test plan you can repeat

Label this unexecuted until you actually run it. Then keep the files.

1. Create a repo with one failing test.
2. Ask the agent to run tests and fix them.
3. Wrap the test command with record_cmd.sh.
4. Save the chat's final sentence into chat_last_line.txt.
5. Open the newest .meta file and read exit=.
6. Run score_chat_vs_exit.py and record the mismatch flag.

Repeat after the agent claims the assert is fixed. Do not skip the wrapper on the second run.

Want a stricter bar? Fail the review if argv does not contain pytest.

grep -E 'argv=.*pytest' .cmd-records/*.meta | tail -n 1 || echo 'no pytest argv'

That one grep catches a surprising amount of theater. Try it once on a real session.

Limitations

This wrapper is not CI. It is a note-taking tool for one host.

It does not redact secrets. Do not log commands that print tokens.

It does not freeze the filesystem. Another process can still race you.

It does not prove tests are meaningful. A zero can still be the wrong suite.

Working directories on a free server may disappear later. Copy .cmd-records off the box if you need the files.

I am not claiming any particular retention window. I am not claiming hardware, quotas, or model names.

The model can still ignore your wrapper. You have to invoke it yourself.

Who should not use this

Skip this if you already capture exit codes in real CI. You do not need a second diary.

Skip this if you need a security audit. Logging argv can leak credentials.

Skip this if you cannot run any commands on the server. Chat-only sessions will not help.

Skip this if you want the model to be the source of truth. This FAQ argues the opposite.

What I do now

I read exit= first. I read the chat second.

I refuse to ship a story with no integer beside it. That habit is cheaper than another mystery failure.

Keep the three buckets. Keep the wrapper. Keep your skepticism.

If you already pair a free model with a free server, wrap the shell before you trust the story. That pairing exists in MonkeyCode; the wrapper still matters on any other host.