FAQ: When the Agent Says Green, Whose Runtime Was That?

Did your coding agent just print a green pytest line?
Did that one line almost become a merge decision?
I still catch that reflex on tired afternoons.

The chat log looks extremely sure of itself.
It even pasted a clean test summary block.
That summary still never names the actual runtime.

Why this FAQ exists

I am not here to ban coding agents.
I am here to kill one lazy shortcut.
Green chat text is not a provenance record.

A free model can explain a failing assertion.
A free server can execute your test command.
Those two are still not your CI image.

I use those tools without worshipping their logs.
The rest of this FAQ is my checklist.

How I read each claim

Every section follows the same four simple beats.

• The claim I keep hearing
• Why the claim spreads so fast
• What I actually inspect on disk
• The mental model I want instead

Skip any section that does not match your stack.
Do not paste secret values into a model prompt.

Claim 1: Green means the same interpreter

The claim

"The agent ran pytest, so my Python matches."
I hear that after almost every green agent run.
It sounds reasonable until you print sys.executable.

Why it spreads

The chat output shows a command you already recognize.
Your brain then fills in your laptop's interpreter.
The log never shows the real binary path.

What I inspect

I ask for a runtime fingerprint, not a vibe.

python3 -c "import sys,platform; print(sys.version); print(sys.executable); print(platform.platform())"
node -p "process.version + ' ' + process.execPath"

Then I compare three columns, not vibes.

• Laptop
• Agent box
• CI image

If any column drifts, the green check is local to that column.

Corrected model

A passing test names a command you typed.
It still does not name your full toolchain.
Treat interpreter identity as evidence you must print.

Claim 2: The agent inherited my PATH

The claim

"It sourced my nvm, asdf, or pyenv setup."
Did the agent actually source those shims though?
Shell plugins live on your machine, not in a prompt.

Why it spreads

You use version managers every single day.
The agent typed python and something ran.
That something is not automatically your shim.

What I inspect

echo "$PATH" | tr ':' '\n'
command -v python3
command -v pytest
type python3
ls -l "$(command -v python3)"

Look for shims, containers, and plain system packages.
Look for missing version files such as .python-version.

Corrected model

PATH is an environment, not a personality trait.
If you did not export it, the box guessed.

Claim 3: Skipped extras will skip in CI too

The claim

"Optional deps failed here, CI will skip them."
Platform wheels lie about this more than people admit.
musl, glibc, Darwin, and CPU flags often disagree.

Why it spreads

The install log said "skipped" or "using cache".
That one sentence feels portable to every machine.
It is usually just a platform footnote.

What I inspect

uname -s -m
python3 -c "import sysconfig; print(sysconfig.get_platform())"
python3 -m pip debug 2>/dev/null | sed -n '1,40p'

Ask one blunt question before you merge.
Would CI install the same wheel tag?

If you cannot answer, you do not have parity.

Corrected model

Each installer skip is a platform-specific event.
Copy the skip into CI only after the tags match.

Claim 4: Lint in chat used my config file

The claim

"Ruff, eslint, or prettier already agreed."
Which working directory did that linter actually see?
Which config file actually won the lookup walk?

Why it spreads

The tool name matches tools in your repo.
The agent printed zero warnings in that chat.
Zero warnings can mean the wrong root.

What I inspect

pwd
git rev-parse --show-toplevel
git status -sb
# pick the linter you actually run in CI
ruff check . --show-settings 2>/dev/null | sed -n '1,20p'
npx eslint --print-config . 2>/dev/null | sed -n '1,20p'

Confirm three things before you relax.

• Config path
• File list
• Commit SHA

Corrected model

A linter result is rooted in a directory.
No directory proof means no style proof.

Claim 5: Install succeeded, so the lockfile won

The claim

"Dependencies resolved. Ship it."
Did the installer freeze hard to the lockfile?
Or did it solve the whole graph again?

Why it spreads

npm i and pip install both look boring.
Those boring logs still hide registry drift well.
A free server may see a different index.

What I inspect

# Node
test -f package-lock.json && echo "lockfile present"
npm ci --dry-run 2>/dev/null || true

# Python
test -f poetry.lock && echo "poetry.lock present"
test -f uv.lock && echo "uv.lock present"
test -f requirements.txt && echo "requirements.txt present"

Prefer the same install command CI uses.
Dry-run beats a hopeful install from chat.

Corrected model

The word installed is not the word reproduced.
The lockfile plus the identical installer command matters.

A labeled example of a false green

This is a reconstructed scenario, not a measured incident.
Treat it as a walkthrough, not as a benchmark.

The agent ran pytest and printed twelve passes.
The box had Python 3.12 on PATH.
CI still used an image pinned to 3.11.

A typing change passed on 3.12.
It failed on 3.11 over a stdlib difference.
The chat never printed sys.version.

Would the fingerprint have caught it?
Yes. The interpreter lines would have drifted.
That is the whole point of the dump.

The artifact: a three-column fingerprint

I keep one script at the repo root.
I run that same script in three places.
Then I diff the files, not my memory.

#!/usr/bin/env bash
# runtime_fingerprint.sh
# Label: example workflow. Review before you run it.
set -euo pipefail

out="${1:-runtime-fingerprint.txt}"
{
  echo "=== runtime fingerprint ==="
  echo "utc: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "host: $(hostname)"
  echo "uname: $(uname -a)"
  echo "user: $(id)"
  echo "pwd: $(pwd)"
  echo "shell: ${SHELL:-unset}"
  echo
  echo "=== interpreters ==="
  command -v python3 && python3 --version || echo "python3: missing"
  command -v node && node --version || echo "node: missing"
  command -v git && git --version || echo "git: missing"
  echo
  echo "=== git ==="
  git rev-parse --is-inside-work-tree 2>/dev/null || echo "not a work tree"
  git rev-parse HEAD 2>/dev/null || true
  git status -sb 2>/dev/null || true
  echo
  echo "=== env names only (no values) ==="
  env | awk -F= '{print $1}' | sort
} > "$out"

echo "wrote $out"

Run it like this.

chmod +x runtime_fingerprint.sh
./runtime_fingerprint.sh laptop.txt
# same command where the agent ran tests
# same command inside CI on the same SHA
diff -u laptop.txt agent-box.txt || true
diff -u agent-box.txt ci.txt || true

I sometimes read those diffs with a free model.
I sometimes capture them on a free server.

I use MonkeyCode when I want that pairing in one place.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free model access helps me narrate a diff.
A free server option gives me a scratch shell.
Neither one becomes my real release pipeline.

Decision table

Claim you heard	Print this	Green still does not prove
Same Python	sys.executable plus version	CI will import the same C extensions
Same Node	process.execPath plus version	Native addons compiled for this OS
Same PATH	command -v and type	Your nvm shims exist on the box
Same lint	config path plus SHA	The linter walked the same file set
Same deps	lockfile hash plus install command	The registry served the same tarball

If a cell is empty, I do not merge.
I rerun the fingerprint first, every time.

Limitations

This workflow only compares captured runtime dumps across boxes.
It does not prove your tests are correct.
It does not prove the agent understood the bug.

The dump can leak hostnames and tool versions.
That is already more than some teams should share.
Redact before you paste anything into a model.

A free server is not a sealed builder.
Images change, disks vanish, and clocks still drift.
I never pin a release to that box.

Do not dump raw environment values into chat.
Print variable names only, and never secret values.
Secrets in values are how chats become incidents.

Who should not use this

Skip this workflow if you sign artifacts.
Skip this if policy blocks third-party hosted models.
Skip this if you need bit-identical builds.

Release engineers should keep a real CI path.
So should anyone in a regulated build environment.
This FAQ is a brake pedal, not a factory.

What I do instead of trusting chat green

1. Run the fingerprint on my laptop.
2. Run it where the agent executed the tests.
3. Run it in CI on the same SHA.
4. Diff the three files without guessing.
5. Only then reread the test output.

Did the agent help with that failure?
Did it inherit my laptop environment today?
Did CI agree on the same commit SHA?

Already have a free model and a free server?
Run the fingerprint before you trust chat green.