FAQ: A Chat PASS Is Not a Test Artifact

Did your agent just declare the suite green?
I see that claim in almost every transcript.

A PASS in prose is not a test artifact.
It is a sentence. Sometimes it is a guess.

So what did the scratch box actually run?
What would remain if the chat window vanished?

This FAQ attacks five claims I still hear.
Each one dies under a small, boring command.

The check I run before I believe PASS

I do not debate the model in the thread.
I make it write files I can open later.

I ask for commands. I keep their outputs.
The chat is a sticky note, not a build.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I use MonkeyCode's free model access and free server option as a scratch box for that capture loop. That is the product role here: a place to run commands and leave files. It is not my CI system.

Then I copy the files into the real repo.
CI stays the judge. The scratch box is a witness.

Myth 1: "It printed PASS, so pytest exited zero"

Who told you that, the model or the shell?
Those two speakers do not share a contract.

Models summarize. Shells return integers.
A paragraph cannot replace $?.

Corrected mental model: stdout is narrative. The exit code is the result.

Run this. Keep both files. Open them yourself.

pytest -q --junitxml=junit.xml
echo $? | tee pytest.exit

If pytest.exit is missing, you proved nothing.
If junit.xml is missing, you proved almost nothing.

A chat line that says "all tests passed" is not replayable.
A junit file names cases. An integer closes the argument.

Still tempted to trust the prose?
Ask where the integer lives. Then stop talking.

Myth 2: "The scratch box uses my Python"

Does it?
Did you check, or did you just feel aligned?

python on a scratch box is a PATH accident.
Your laptop Python is a different accident.

Corrected mental model: a toolchain is a triple: binary, version, platform.

uname -sm
command -v python3
python3 -V
python3 -c "import sys; print(sys.executable); print(sys.platform)"

Write those lines into toolchain.txt.
Park that file next to the patch under review.

If versions differ, a green run is a different product.
You tested a cousin. You did not test your app.

Node folks, swap in node -v and which node.
Go folks, record go env GOOS GOARCH.
Same myth. Same fix. Different binary name.

Myth 3: "The agent installed the deps, so we are pinned"

Installed where?
Pinned how? In which file that git can see?

pip install requests is not a lockfile.
It is one resolver pass on one afternoon.

Corrected mental model: if it is not in the lockfile, it is gossip.

# labeled example: capture the scene, do not "heal" it
python3 -m pip freeze | tee freeze.txt
test -f requirements.lock && cp requirements.lock lock.copy
wc -l freeze.txt

Ask one rude question after every install claim.
Which file now constrains the next install?

No file? Then you cannot reproduce tomorrow.
The model will not remember the wheel set.

freeze.txt is a snapshot of what landed once.
It is not a promise about the next machine.

Myth 4: "It compiled there, so it will compile here"

Compiled for whom?
Which architecture? Which C library? Which platform tag?

A scratch box is not your laptop.
It is not your CI image either.

Corrected mental model: a build is tied to uname -m and the loader.

uname -m
python3 -c "import platform; print(platform.machine()); print(platform.platform())"
python3 -c "import sysconfig; print(sysconfig.get_platform())"

Shipping a wheel? Record the platform tag.
Shipping Go or Rust? Record the target triple.

"It built" without a triple is a tourist photo.
Pretty. Useless at release time.

C extensions make the lie louder.
A pure Python pass can still explode on manylinux.

Myth 5: "Skip CI. The agent already ran the suite"

Would you skip CI because a coworker used a random VM?
Then do not skip it for a chat window.

Corrected mental model: a scratch-box run is a canary, not a gate.

CI has the pinned image, the secrets policy, the matrix.
The free server has whatever it has today. I am not claiming that image is stable.

Use the agent run to find dumb failures fast.
Do not use it to bless a merge.

Green on a witness host is still only a hint.
Hints are cheap. Merges are not.

Artifact: capture_run.sh plus a decision table

Here is the script I paste first.
Treat it as a proposal. Run it on your box.

#!/usr/bin/env bash
# capture_run.sh — evidence, not a story
set -u
mkdir -p evidence
{
  echo "date_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "pwd=$(pwd)"
  echo "uname=$(uname -sm)"
  echo "python=$(command -v python3)"
  python3 -V
  python3 -c "import sys,sysconfig; print(sys.executable); print(sysconfig.get_platform())"
} | tee evidence/toolchain.txt

if [[ -f requirements.lock ]]; then
  cp requirements.lock evidence/lock.copy
elif [[ -f poetry.lock ]]; then
  cp poetry.lock evidence/lock.copy
elif [[ -f package-lock.json ]]; then
  cp package-lock.json evidence/lock.copy
elif [[ -f go.sum ]]; then
  cp go.sum evidence/lock.copy
else
  echo "NO_LOCKFILE" | tee evidence/lock.copy
fi

python3 -m pip freeze > evidence/freeze.txt 2>/dev/null || true

set +e
if [[ -f pyproject.toml || -f pytest.ini || -d tests ]]; then
  python3 -m pytest -q --junitxml=evidence/junit.xml
  echo $? | tee evidence/pytest.exit
else
  echo "NO_PYTEST_LAYOUT" | tee evidence/pytest.exit
fi
set -e

ls -la evidence

Read this table before you trust any PASS.

You have this file	You proved	You did not prove
Chat sentence "PASS"	Someone claimed success	Any test ran
evidence/pytest.exit is 0	This interpreter exited zero	Your CI image would
evidence/junit.xml	Named tests ran here	Stability or coverage
evidence/toolchain.txt	Which binary ran	That it matches prod
Lockfile copy	Inputs were at least named	The resolver was honest
Only freeze.txt	What landed this once	What will land next time

If the left column is empty, stop.
You are reviewing fiction with nice formatting.

Four questions I paste into the chat

I want files. I do not want apologies.

1. Write $? to evidence/pytest.exit.
2. Write python3 -V and uname -sm to evidence/toolchain.txt.
3. Copy the lockfile, or write NO_LOCKFILE.
4. Do not say PASS unless those files exist.

Then I open the files myself.
I do not let the model paraphrase XML.

Want a brutal extra check?
Diff evidence/toolchain.txt against your CI image.

# labeled example: compare if you already have a CI-like image
# docker run --rm your-ci-image python3 -V

Mismatch? Then the scratch PASS is a different experiment.
Useful signal. Not a merge ticket.

I also glance at timestamps inside junit.xml.
A suite that "ran" in zero milliseconds is a story.

What this does not fix

This workflow does not turn a scratch box into CI.
It does not pin a model. It does not pin a VM image.

I am not claiming hardware, quotas, or uptime.
I am not claiming the free server lasts, or that it matches last week.

The script assumes a Python-ish layout.
Translate it for Go, Node, or Rust. Keep the same evidence idea.

The model can refuse to run the script.
The model can invent a junit file. Open the XML tags.

junit.xml can be forged in a workspace.
Your real gate is still CI on a known image.

Networked tests lie in a second way.
A blocked egress can look like a skipped module.

Who should not use this

Do not use a scratch box for secret-bearing tests.
Do not point it at production databases or live queues.

Do not skip required reviewers because a file says zero.
Do not treat freeze.txt as a license audit.

If you need bit-for-bit builds, this venue is wrong.
If you need change-control theater for auditors, this venue is wrong.

If you cannot read $?, this FAQ will not help you.
Run pytest locally first. Learn that integer. Come back.

Teams with a strict matrix belong in CI from the first push.
A witness host is optional. The gate is not.

The mental model I want stuck in your head

Ask one question when the chat yells PASS.
Where is the exit code file?

No file means no result.
A result on a scratch box is still not CI.

Keep the evidence. Merge with your real gate.
The chat can disappear. The files should not.

If you already have a scratch box, run capture_run.sh before the next PASS argument.
Then argue from pytest.exit, not from a paragraph.