The Agent Wrote Tests. Did Anything Actually Run?

Did your coding agent just add tests today?
Then I have one unkind follow-up question.

What process executed those newly added test files?
Which exit code did that process return?

I keep hearing four claims in review threads.
They sound like diligence, but they are theater.

This FAQ busts those four repeated claims.
Then it gives you a claimed-versus-actual proof pack.

The background nobody wants to name

Chat UIs still reward a complete-sounding story.
A model can narrate a green suite in one breath.

You paste that story into a pull request.
Busy reviewers treat the story as a result.

Have you shipped that story this month?

Trend pieces keep arguing about vibe coding lately.
The real engineering failure is smaller and meaner.

We confuse three objects that are not equal.
Text, files, and runner results are different objects.

Myth 1: A new test file means coverage

Claim: The agent added tests, so the change is safe.

What developers repeat: test_api.py exists. Merge it.

Evidence that would actually count:

• Git tracks the test module
• A collector imported the module
• Assertions ran in a process
• The process exit code was zero

Corrected mental model: Authorship of a test is not execution.
A test that never loaded is commented-out hope.

Ask one blunt question at review time.
Did an interpreter import this test module today?

Collect before you trust

# proposed check; not a timed benchmark
git ls-files '*test*.py' '*_test.py' > /tmp/tracked_tests.txt
python -m pytest --collect-only -q | tee /tmp/collected.txt
echo "tracked=$(wc -l < /tmp/tracked_tests.txt)"

If collect-only cannot see the file, stop.
You are reviewing prose that happens to use def.

Parametrize can also lie in the other direction.
One function can explode into twenty node ids.

Count nodes, not functions, when you compare claims.

Myth 2: The recap is the suite result

Claim: "All fourteen tests passed."

Who actually counted those fourteen tests, exactly?
Was it the model, pytest, or a screenshot of neither?

Natural language integers stay cheap to hallucinate.
JUnit XML is boring, and that is the point.

Corrected mental model: You should believe stored artifacts, not chat sentences.
A sentence cannot sign a test report.

Decision table

Chat claim	Evidence?	Why it fails or holds
"All tests passed"	No	No runner identity, no exit code
Pasted pytest summary	Weak	Transcripts are easy to fabricate
junit.xml on disk	Yes	The runner wrote it
CI log URL	Yes	A third system stored it
Pass count without skips	Careful	Skips can inflate a green story
Pass count without errors	Careful	errors are not failures

Pytest treats import crashes as errors, not failures.
Agents often fold errors into failed or hide them.

Did the recap mention skipped tests at all?
Did it mention xfail or collection errors?

If those words never appeared, distrust the integer.

Myth 3: Self-check is a second run

Claim: The agent re-read the tests and confirmed them.

That is one narrator grading its own homework.
Would you let the compiler mark its own exam?

Corrected mental model: The real oracle always lives outside the chat.
A host process is an oracle, and a recap is not.

You need a split, not a pep talk.
Generate fixtures in one place, then execute in another.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A practical split can look like this.
Draft the harness with MonkeyCode's free model access.
Then execute that harness on the free server option.

The value is the split, not the logo.
Any model plus any scratch VM can do this.

Keep generation away from the oracle on purpose.
That split is the whole method here.

Myth 4: Skipping the runner saves time

Claim: The diff is tiny, so tests are overhead.

Tiny diffs still rename symbols under you.
Tiny diffs still invert a boolean in a guard.

The hour you save becomes incident time later.
That trade looks smart until production pages you.

Corrected mental model: The test runner remains the cheapest review you have.
A short pytest run beats a long outage thread.

Also, coding agents retry without telling you.
Retries without an oracle duplicate side effects you never measured.

You do not need a lab for this lesson.
You need an exit code on disk.

Artifact: claimed versus actual

You should stop arguing with the chat transcript.
Serialize both stories, then compare them on disk.

This script is only a proposed oracle.
It is not a vendor benchmark, and it is not timed.

#!/usr/bin/env python3
"""claimed_vs_actual.py — proposed oracle, not production proof."""
from __future__ import annotations

import json
import subprocess
import sys
import xml.etree.ElementTree as ET
from pathlib import Path

CLAIM = Path("claimed.json")
JUNIT = Path("junit.xml")
PROOF = Path("proof.json")


def load_claim() -> tuple[int, int, int]:
    data = json.loads(CLAIM.read_text())
    passed = int(data["passed"])
    failed = int(data["failed"])
    skipped = int(data.get("skipped", 0))
    return passed, failed, skipped


def run_suite() -> int:
    cmd = [
        sys.executable, "-m", "pytest", "-q",
        f"--junitxml={JUNIT}",
    ]
    return subprocess.run(cmd).returncode


def read_junit() -> tuple[int, int, int]:
    root = ET.parse(JUNIT).getroot()
    suites = list(root) if root.tag == "testsuites" else [root]
    passed = failed = skipped = 0
    for suite in suites:
        tests = int(suite.attrib.get("tests", 0))
        failures = int(suite.attrib.get("failures", 0))
        errors = int(suite.attrib.get("errors", 0))
        skipped_n = int(suite.attrib.get("skipped", 0))
        failed += failures + errors
        skipped += skipped_n
        passed += tests - failures - errors - skipped_n
    return passed, failed, skipped


def main() -> None:
    if not CLAIM.exists():
        print("missing claimed.json; copy the agent's counts first")
        sys.exit(2)
    exit_code = run_suite()
    if not JUNIT.exists():
        print("pytest wrote no junit.xml; collection probably died")
        sys.exit(3)
    actual_p, actual_f, actual_s = read_junit()
    claim_p, claim_f, claim_s = load_claim()
    report = {
        "claimed_passed": claim_p,
        "claimed_failed": claim_f,
        "claimed_skipped": claim_s,
        "actual_passed": actual_p,
        "actual_failed": actual_f,
        "actual_skipped": actual_s,
        "pytest_exit": exit_code,
        "counts_match": (
            claim_p == actual_p
            and claim_f == actual_f
            and claim_s == actual_s
        ),
        "runner_green": exit_code == 0,
    }
    PROOF.write_text(json.dumps(report, indent=2) + "\n")
    print(json.dumps(report, indent=2))
    ok = report["counts_match"] and report["runner_green"]
    sys.exit(0 if ok else 1)


if __name__ == "__main__":
    main()

Seed the claim from chat

{
  "passed": 14,
  "failed": 0,
  "skipped": 0,
  "source": "agent-recap",
  "note": "copy numbers from chat; do not trust them"
}

Run the pack

# proposed workflow on any scratch host
python -m venv .venv
. .venv/bin/activate
pip install pytest
printf '%s\n' '{"passed":14,"failed":0,"skipped":0}' > claimed.json
python claimed_vs_actual.py
cat proof.json

If counts_match is false, the chat drifted.
If the runner never wrote XML, you had no suite.

Commit junit.xml and proof.json with the PR.
Now reviewers can argue with files, not vibes.

One-hour checklist

1. Freeze the agent's pass, fail, and skip counts.
2. Write those integers into claimed.json before running.
3. Collect tests with --collect-only and read node ids.
4. Execute the proof pack on a real interpreter.
5. Attach proof.json plus the JUnit file to the PR.
6. Reject the change when runner_green is false.

No dashboard is required for this gate.
No proprietary runner is required for this either.

Limitations, said plainly

Matching counts still do not mean correct assertions.
Wrong tests that run are still wrong tests.

This pack does not measure flake rates.
It also does not measure a mutation score.
It does not measure runtime performance at all.
It does not measure any real security properties.

Free model access is only a convenience.
A free server option is only a convenience.
I will not invent token quotas or hardware claims.

Treat both as optional tools for the split.
Do not treat them as a contract or an SLO.

Who should not use this approach

• Skip this if CI already stores JUnit on every merge.
• Skip this when the code cannot leave a locked network.
• Skip this if secrets must never reach a scratch host.
• Skip this when no human still owns the contract.
• Skip this if you want a recap to replace design work.

If CI already gates merges, do not add theater.
Wire the pack into CI, or skip the extra ritual.

Keep this mental model

Three objects exist after every agent coding session.

1. Text the model wrote in a transcript
2. Files git actually recorded
3. Results a runner produced on a host

Only object three can turn a suite green.
Object one is a story, and object two is an input.

So, did the agent write tests this time?
Good, now make a real process run them.