DEV Community

Jordan Huang
Jordan Huang

Posted on

The Agent Transcript Is Not a Test Report

#ai #testing #python #agents

Did the agent run the suite?

Did your coding agent actually run the test suite? Or did it only narrate a green run to you?

I keep hitting this mix-up on cheap stacks. A remote shell plus a chat model looks like CI. It is not CI in any serious sense. It is a narrator that might call a shell.

This piece is a myth FAQ for evals. It is a corrected mental model, nothing fancier.

What I actually check

I use a free remote server as a scratch box. I also use free model access for short agent loops.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is one place that pairing exists. You can swap the host and keep the hygiene. I do not treat either piece as a merge gate. I treat them as cheap rehearsal before real CI.

Green in chat, red in pytest?

Claim developers repeat: The agent said all tests passed, so they passed.

Evidence I collect: I ignore the story in the transcript. Then I rerun the exact pytest command.

Who actually executed pytest during that agent session? Was it the model, or a real runner with an exit code?

# proposed check — label this unexecuted on your tree
git rev-parse --short HEAD
python -m pytest -q --tb=line
echo "exit:$?"
Enter fullscreen mode Exit fullscreen mode

The chat log is a story about work. The process exit code is evidence of work.

Corrected mental model: Transcripts are only claims from a narrator. Runners produce facts you can store.

Watch for fake greens in the transcript too. Did the agent pass a collect-only flag? Did it use a filter that matches nothing?

# proposed: inspect the alleged command
grep -E 'collect-only|--nofail|-k |--lf' eval/claimed.json || true
Enter fullscreen mode Exit fullscreen mode

A collected suite is not a passed suite. An empty test selection is not success.

Did the files even land?

Claim developers repeat: The agent edited files, so my laptop has them.

Evidence I collect: I print git status on both machines. After that I compare both HEAD commit values.

Does the remote tree match your working branch? Did anyone commit, or only talk about commits?

# proposed: pin the tree before any agent loop
git fetch origin
git checkout --detach "$SHA"
git status --porcelain
Enter fullscreen mode Exit fullscreen mode

A dirty remote worktree is not your feature branch. A model cannot teleport the index to your laptop.

Corrected mental model: That free server is still a stranger checkout. You should pin a SHA or stop.

I also dump pwd and a directory listing. Coding agents often love the wrong working directory. Green tests under tmp do not save your repo.

pwd
ls -la
git diff --stat
Enter fullscreen mode Exit fullscreen mode

If the diff is empty, the story was empty. I believe the diff, not the speech.

Was that one lucky loop an eval?

Claim developers repeat: I watched it succeed once, so the task is solved.

Evidence I collect: I store inputs, tools, and the independent runner output.

Was that prompt pinned in git yet? Was the tool schema pinned beside it? Did you keep the raw pytest file after the loop died?

eval/
  sha.txt
  prompt.md
  tools.json
  claimed.json
  pytest.txt
  verdict.txt
Enter fullscreen mode Exit fullscreen mode

A vibe of success is not a dataset. An eval is a folder you can replay tomorrow.

Corrected mental model: If you cannot replay it, you did not evaluate it.

I refuse to argue from memory here. Memory is how we bless flaky runs. The replay folder is the only argument.

Will the next model fail the same way?

Claim developers repeat: It failed on the free model, so paid fails too.

Evidence I collect: I treat model class as an input, not as weather.

Did you record which model endpoint actually answered? Did you keep temperature and the tool results next to the SHA?

I do not invent quotas or model names here. I only record whatever the client already prints.

# proposed logger — unexecuted example
import os
record = {
    "sha": open("eval/sha.txt").read().strip(),
    "endpoint": os.environ.get("MODEL_ENDPOINT", "unset"),
    "claimed": claimed_payload,
    "pytest_exit": pytest_exit,
}
Enter fullscreen mode Exit fullscreen mode

Different chat models often fail for different reasons. A free rehearsal is not a production oracle.

Corrected mental model: Model identity is part of the fixture. Log that identity or you are only guessing.

Is tool JSON a committed fixture?

Claim developers repeat: The function call looks structured, so it is durable.

Evidence I collect: I copy tool results into files under eval. Chat history rot is real, and it is fast.

Can you open last week's tool trace as a file? Or did the UI swallow the only copy?

# proposed: persist every tool result
mkdir -p eval/tools
printf '%s\n' "$TOOL_JSON" > "eval/tools/$(date -u +%Y%m%dT%H%M%SZ).json"
Enter fullscreen mode Exit fullscreen mode

A pretty JSON blob in a sidebar is not source control. Fixtures live in git, or they drift.

Corrected mental model: If it is not in the repo, it already drifted.

I also hash the tool file after each copy. Then I can see silent overwrites much later.

sha256sum eval/tools/*.json > eval/tools.sha256
Enter fullscreen mode Exit fullscreen mode

Those hashes look boring, and that is the point.

Does SSH make it CI?

Claim developers repeat: The agent can run commands, so it is the pipeline.

Evidence I collect: I ask who can merge, and what artifact is required.

Can that box block a merge request? Does it publish junit for the job? Does it use a secrets policy you could audit?

CI has identity, artifacts, and a gate. A free scratch server has a prompt and a shell.

Corrected mental model: Rehearsal boxes do not ship software, pipelines do.

If your gate is reading a chat, you still have no gate.

The artifact: claimed versus actual

Here is the workflow I want in the loop. Treat every snippet as a proposal unless you run it.

Pin the world first 

#!/usr/bin/env bash
# proposed eval_gate.sh
set -euo pipefail
SHA="$(git rev-parse HEAD)"
mkdir -p eval
printf '%s\n' "$SHA" > eval/sha.txt
git status --porcelain > eval/dirty.txt
if [ -s eval/dirty.txt ]; then
  echo "dirty tree, refusing to grade" >&2
  exit 2
fi
Enter fullscreen mode Exit fullscreen mode

If that dirty file is not empty, I stop. Dirty trees mint false greens every time.

Let the agent speak, then distrust it

The agent may write eval/claimed.json for me. I still do not believe a single word.

{
  "claimed_pass": true,
  "claimed_command": "pytest -q",
  "notes": "agent narrative only"
}
Enter fullscreen mode Exit fullscreen mode

That file is an allegation from the agent. That JSON file is not the test suite.

Run the suite with no narrator 

set +e
python -m pytest -q --tb=line > eval/pytest.txt
echo $? > eval/pytest_exit.txt
set -e
Enter fullscreen mode Exit fullscreen mode

I keep the same SHA and the same tree. There is no narrator in that path.

Compare claim and fact 

# proposed compare.py — unexecuted example
import json
from pathlib import Path

claimed_path = Path("eval/claimed.json")
exit_path = Path("eval/pytest_exit.txt")
verdict_path = Path("eval/verdict.txt")

if not claimed_path.exists() or not exit_path.exists():
    verdict_path.write_text("INVALID\n")
    raise SystemExit("missing claim or runner output")

try:
    claimed = json.loads(claimed_path.read_text())
except json.JSONDecodeError:
    verdict_path.write_text("INVALID\n")
    raise SystemExit("claimed.json is not JSON")

exit_code = int(exit_path.read_text().strip())
said_pass = bool(claimed.get("claimed_pass"))
actual_pass = exit_code == 0
verdict = "match" if said_pass == actual_pass else "LIE"
verdict_path.write_text(
    f"said_pass={said_pass}\nactual_pass={actual_pass}\nverdict={verdict}\n"
)
print(verdict)
Enter fullscreen mode Exit fullscreen mode

That LIE label is the whole point. I want the mismatch loud, not polite.

A table I keep beside the terminal

Situation Trust the transcript? What I run instead
Agent says green, no junit file No pytest on a pinned SHA
Agent edits files, no commit No git status and git diff
One lucky loop on a free model No Replay the folder under eval/
Tool JSON only in the chat UI No Copy it into eval/tools/
Scratch server can SSH No A real CI job with a gate
Claim and pytest_exit match Still maybe Replay tomorrow on a clean tree

I read that table when I get impatient. Impatience is how those fake greens get shipped.

When verdict is LIE

I do not argue with the chat model. I open three files and stay there.

  1. The sha file shows which tree we graded.
  2. The claimed file shows what the agent alleged.
  3. The pytest file shows what the runner printed.

Then I ask three questions out loud, slowly. Did the agent skip the actual pytest command? Did it run under a different project path? Did it invent a skip list for failing tests?

Most lies are skipped commands from the agent. The rest are just wrong working directories.

# proposed: prove cwd and the first errors
pwd
ls -la
sed -n '1,40p' eval/pytest.txt
Enter fullscreen mode Exit fullscreen mode

Short files beat long arguments with a chat model. Keep the ritual boring on purpose for yourself.

Limitations

This workflow does not measure model quality in general. It only catches claim-versus-runner drift on one tree.

It does not pin GPUs, quotas, or vendor SLAs. I will not invent those numbers for you.

It does not replace CI, code review, or a real staging app. A free server is still a scratch box.

Network blips can fail pytest on their own. A red runner is not always a lying agent.

Flaky tests will smear the verdict very fast. Fix flakes before you blame the model.

Who should not use this

Do not use this as a production merge gate. You still need identity, logs, and access policy.

Do not use this if you cannot pin a git SHA. Random trees will make random morals here.

Do not use this pattern for secret-bearing suites. Those scratch boxes are not real secret vaults.

Do not use this if your tests are only screenshots in chat. That is theater, and it is not evaluation.

What I keep from the myths

I still like free models for cheap rehearsal. I still like a free remote server for a throwaway checkout.

I just refuse to merge on a story. The runner writes the report for the tree. The agent only writes a draft report.

If you already have that free pairing, run the comparator on one fixture. Then throw the transcript away if verdict says LIE.

Top comments (0)