FAQ: Five Myths About Agent-Run Tests

Did your coding agent just print green pytest output? That screenshot still is not your real pipeline.

I keep seeing the same five claims in reviews. This FAQ is how I personally tear them apart.

Why this FAQ exists

Agents now run unit tests on scratch boxes. That sudden green output can feel exactly like CI.

I wanted a checklist I can paste into PRs. Feelings from a chat window do not merge code.

The real problem is measurement, not model cleverness. Green output can lie in very boring ways.

What I actually collect

I do not argue with the chat transcript. I collect files that another machine can replay.

Here is the minimum evidence pack I demand:

• interpreter version and platform string
• lockfile hash at the same git HEAD
• pytest JUnit XML with a real test count
• coverage XML only if coverage was claimed
• the exact command line that ran
• git HEAD and a porcelain dirty check

These are proposed local commands after the agent session.

python -V
uname -a
git rev-parse HEAD
git status --porcelain
sha256sum poetry.lock requirements.txt package-lock.json 2>/dev/null || true

Are you still with me on this pack? Keep those files beside the pull request.

pytest -q --junitxml=agent-junit.xml \
  --cov=. --cov-report=xml:agent-cov.xml
echo "exit=$?"

I label that run as a local replay. I never treat the chat as the runner.

Myth 1: A zero exit code means tests ran

The claim

pytest returned zero, so the suite truly ran.

Did collection silently select zero files this time? Did the import path miss your real packages?

What I check

I always parse the JUnit XML test count. Zero cases can still exit zero under some flags.

# proposed checker: parse pytest JUnit XML
import sys
import xml.etree.ElementTree as ET

root = ET.parse("agent-junit.xml").getroot()
suites = [root] if root.tag == "testsuite" else list(root)
total = sum(int(s.attrib.get("tests", 0)) for s in suites)
fails = sum(int(s.attrib.get("failures", 0)) for s in suites)
errors = sum(int(s.attrib.get("errors", 0)) for s in suites)
skipped = sum(int(s.attrib.get("skipped", 0)) for s in suites)
print(
    {
        "tests": total,
        "failures": fails,
        "errors": errors,
        "skipped": skipped,
    }
)
if total == 0:
    sys.exit("no tests collected")

The corrected model

An exit code is necessary but never actually sufficient. Count executed cases before you trust the smile.

Myth 2: Agent-authored tests are independent proof

The claim

It wrote tests, they passed, so the feature is done.

Who wrote the oracle in that session? The same agent that wrote the production change.

What I check

That measurement is circular, and it should bother you. Do these tests measure the change, or the model?

I hunt tautologies during every test review. I do not hunt slogans in the chat log.

Red flags I search for by hand:

1. assert True or empty test bodies
2. mocks that replace the unit under test
3. snapshots of the agent's own printed output
4. tests added only to inflate coverage totals

Proposed questions I paste under the diff:

1. Would this test fail if I revert the feature?
2. Does it hit a public interface, not a self-mock?
3. Did a human name the invariant in review?

The corrected model

Agent tests remain drafts for a human reviewer. Humans still own the oracle and the merge.

Myth 3: The free server is close enough to CI

The claim

It passed on the remote box, so CI will pass.

Will that box really match your pipeline image? What Python, what libc, and what network rules?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use MonkeyCode's free model access and free server option for scratch work only. I treat that box as a whiteboard that happens to have a shell.

I do not treat it as GitHub Actions, GitLab CI, or Jenkins. Those systems keep artifacts I can link.

What I check

Proposed parity checks I run in both places:

python -c "import sys,platform; print(sys.version); print(platform.platform())"
pip freeze | sha256sum

Decision table I keep above my keyboard:

Signal on the scratch box	What it may mean	What it does not mean
pytest exits 0	That environment ran those files	Your CI image will agree
lockfile hash matches HEAD	Dependencies were pinned in git	Native wheels match CI
coverage XML exists	A tracer actually ran	The percentage is honest
the model explained a failure	You received a hypothesis	Someone tested the hypothesis
a screenshot shows green	A person captured a frame	The job is replayable

The corrected model

Scratch boxes are for finding typos fast. Only real pipelines should gate a release.

Myth 4: One green run kills flakiness

The claim

It passed once, so the suite is stable.

Where do order bugs hide in practice? Time bugs and tempdir bugs hide there too.

What I check

I rerun the same command in a tight loop. I want collisions in that loop, not comfort.

# proposed flake probe; not a published benchmark
for i in 1 2 3 4 5; do
  echo "run $i"
  pytest -q -p no:cacheprovider \
    || echo "fail run $i"
done

Did any iteration fail while others passed? Then you do not have a receipt, only a coin flip.

The corrected model

Stability is a distribution of many runs. One lucky sample is only a story.

Myth 5: Coverage percent is a quality receipt

The claim

We hit a high percent, so we can ship.

Percent of which tree, after which generated files? Could empty modules have diluted the ratio on purpose?

What I check

I read the coverage XML myself every time. I refuse to read a badge in isolation.

# proposed reader for coverage.py XML
import xml.etree.ElementTree as ET

root = ET.parse("agent-cov.xml").getroot()
print(root.attrib)
for pkg in root.findall("packages/package"):
    name = pkg.attrib.get("name")
    rate = pkg.attrib.get("line-rate")
    print(name, rate)

Which lines did this change actually need? Vanity totals do not answer that question.

The corrected model

Coverage is only a map of execution. It is not a grade you can cash.

The revert probe nobody runs

Green tests should fail when I undo the feature. Otherwise they do not measure the change.

This stash probe is easy to get wrong. Commit first if you are scared.

# proposed revert probe; unexecuted until you run it
pytest -q; echo "with_change=$?"
git stash push -u -m agent-wip
pytest -q; echo "without_change=$?"
git stash pop

If without_change is also zero, stop merging. Those tests never saw the feature.

I want a failing run on the old tree. Passing on both trees is a tautology.

A proposed script I keep in the repo

This script is a proposal, not a vendor tool. I drop it next to pyproject.toml in the repo.

#!/usr/bin/env bash
# verify_agent_tests.sh — proposed local gate
set -euo pipefail
JUNIT="${JUNIT:-agent-junit.xml}"
COV="${COV:-agent-cov.xml}"

python -V
git rev-parse HEAD
git diff --stat

pytest -q --junitxml="$JUNIT" \
  --cov=. --cov-report=xml:"$COV"

JUNIT="$JUNIT" python - <<'PY'
import os, sys, xml.etree.ElementTree as ET
path = os.environ.get("JUNIT", "agent-junit.xml")
root = ET.parse(path).getroot()
suites = [root] if root.tag == "testsuite" else list(root)
total = sum(int(s.attrib.get("tests", 0)) for s in suites)
fails = sum(int(s.attrib.get("failures", 0)) for s in suites)
errors = sum(int(s.attrib.get("errors", 0)) for s in suites)
print(f"tests={total} failures={fails} errors={errors}")
if total == 0 or fails or errors:
    sys.exit(1)
PY

echo "local rerun ok; still wait for CI artifacts"

Run it once after the agent stops typing. Run it again on your own laptop.

Want a third independent copy of that run? That third copy is your actual CI.

How I close a noisy PR comment

I paste this block and then I stop. Debating feelings wastes the whole afternoon here.

Evidence missing:
- [ ] JUnit test count greater than zero
- [ ] lockfile hash matches current HEAD
- [ ] interpreter version matches the CI image
- [ ] tests fail when the feature is reverted
- [ ] CI job URL, not a chat screenshot

Five boxes beat a paragraph of confidence. Can we just tick them in the PR?

Limitations

This workflow assumes pytest plus XML reports. It does not replace mutation testing at all.

It will not detect a test that asserts the bug. Wrong requirements still sail right through this gate.

The script ignores browsers, devices, and load. API performance needs a different honest harness.

Please do not fake latency work with unit pytest cases. That mix lies to your future self.

I also do not claim any free server remains available. Availability can change without a blog post.

I do not pin model names, quotas, or hardware here. Those numbers go stale by the next morning.

Who should not use this approach

Skip this if your CI already blocks on artifacts. You do not need another sermon from me.

Skip this if your suite is not Python today. Port the checks; do not copy the parser blindly.

Skip this if you need a production soak test. A scratch box will never give you that signal.

Do not use a free remote shell for secrets either. Keep credentials off those scratch boxes, always.

What I want you to remember

The model can help you draft tests. A free server can run a first pass.

Your pipeline still owns the only merge receipt. Would I merge on a chat screenshot today?

No, and I will keep saying no. Would I merge on JUnit plus a CI job URL?

Maybe, but only after the revert check. Ask one sharp question in standup tomorrow morning.

Where is the job URL for that green run? If you already draft on a scratch agent box, keep going.

Steal the checklist, not the pretty screenshot.