FAQ: Four Myths About "pytest Already Passed" in an Agent Loop

Did your agent just tell you the tests passed?

I hear that sentence at the end of almost every loop.

It sounds like CI, but it is only narration.

This FAQ names four myths I still hear.

Each myth skips the actual test runner.

I want a receipt, not a pep talk.

Why these myths stick

Agents draft tests in seconds now, and that part is true.

A cheap remote box makes reruns feel optional.

Why rerun what the model already "saw"?

Because the model did not see a process exit.

It predicted a plausible summary, and that is different.

Would you merge from a screenshot of a terminal?

I would not merge that, and neither should you.

Here is the loop I watch go wrong.

1. Agent edits app.py.
2. Agent writes tests/test_app.py.
3. Agent prints a fake pytest summary.
4. Human merges. CI is the first real runner.

Which step produced evidence? Only step four.

Steps one through three were hope with extra files.

Myth 1: The chat summary means pytest exited zero

The claim developers repeat

The model printed 47 passed, 0 failed.

So the suite ran. So the patch is good.

What evidence you actually hold

You hold tokens that look like pytest.

You do not hold an exit code from pytest.

Can a model invent a perfect summary line?

Yes, and it will do that without shame.

A green paragraph is still a paragraph.

It is not a child process. It is not a status code.

Corrected mental model

Treat chat as a narrator, never as a runner.

No process, no result. Do not negotiate that.

Ask this every time a summary appears.

Which command ran, and which PID wrote stdout?

If you cannot answer, you only have prose.

Quick check I want on every turn:

• The exact argv, not a paraphrase.
• The numeric exit code, not an adjective.
• A file on disk the next turn can open.

Missing any one of those? You are not done.

Myth 2: Adding test_foo.py made the suite safer

The claim developers repeat

The agent created tests, so coverage went up.

Green files exist in git. You can relax.

What evidence you actually hold

You hold more Python files. Maybe zero real assertions.

I keep finding this shape in generated tests.

# Example only: this never imports the patch.
def test_create_user_shape():
    user = {"id": 1, "name": "Ada"}
    assert user["id"] == 1
    assert user["name"] == "Ada"

Did those lines call your production function?

No. They asserted on a literal dict you wrote.

Will they fail if create_user returns garbage?

They will not fail. They cannot fail.

Here is the sibling pattern, also useless.

# Example only: tautology dressed as a test.
def test_patch_is_fine():
    assert True

Would you accept that from a junior teammate?

Then do not accept it from a model either.

Corrected mental model

A test is a claim about code you changed.

If it never imports the patch, it is theater.

Count calls into production modules before you smile.

Zero calls means you should delete the file.

A useful generated test does three boring things.

1. It imports the function you just touched.
2. It feeds a realistic input, not a hard-coded trophy.
3. It fails when you break that function on purpose.

If you cannot make it fail, it is not a test.

Myth 3: Pasting the log back preserves the failure

The claim developers repeat

The next turn will "see" the pytest output.

You pasted it. The model will not forget.

What evidence you actually hold

You hold a transcript that will be truncated.

Free model turns drop old tool output constantly.

Did you store the failure on disk?

If not, the next turn is guessing again.

Pasted logs also invite a second lie.

The model may "fix" a failure that never happened.

Or it may ignore a real failure you scrolled past.

Chat is a leaky buffer. Why trust it as memory?

Corrected mental model

Disk outlives chat. Write the result where git can see it.

Do not paste sixty kilobytes of logs on purpose.

Point the agent at test-receipt.json instead.

The next turn should open a file, not a vibe.

If the file is missing, the loop restarts from ignorance.

That ignorance is expensive even when the model is free.

Myth 4: junit.xml in the tree proves pytest ran

The claim developers repeat

The workspace contains junit.xml, so tests ran.

XML is machine readable. XML cannot lie.

What evidence you actually hold

You hold a file the agent could have authored.

Models write XML. They write it confidently.

Look at this worthless "report".

<!-- Example only: an agent can emit this without pytest. -->
<testsuite name="suite" tests="12" failures="0">
  <testcase name="test_create_user" classname="tests"/>
</testsuite>

Did pytest write that? You do not know.

A file named like a report is not a report.

If the model offers to "just write the XML", stop.

That is the tell. Kill that turn.

Corrected mental model

The runner writes the receipt. The model must not.

Wrap pytest. Refuse receipts without the wrapper stamp.

Trust a process you launched, not a document it described.

A reproducible gate you can copy

Here is a small wrapper I want in the repo.

It runs pytest. It writes a receipt. It stamps a nonce.

Label this as a local example. Run it yourself.

Do not ask the model to generate the receipt.

#!/usr/bin/env python3
"""pytest_receipt.py — run pytest, write a stamped receipt.

Example (unexecuted here): python pytest_receipt.py -q
"""
from __future__ import annotations

import hashlib
import json
import os
import secrets
import subprocess
import sys
import time
from pathlib import Path

RECEIPT = Path("test-receipt.json")
JUNIT = Path("junit.xml")


def main() -> int:
    nonce = secrets.token_hex(16)
    started = time.time()
    cmd = [
        sys.executable,
        "-m",
        "pytest",
        "--junitxml",
        str(JUNIT),
        *sys.argv[1:],
    ]
    proc = subprocess.run(cmd, check=False)
    ended = time.time()
    junit_sha = ""
    if JUNIT.exists():
        junit_sha = hashlib.sha256(JUNIT.read_bytes()).hexdigest()
    receipt = {
        "ok": proc.returncode == 0,
        "exit_code": proc.returncode,
        "command": cmd,
        "python": sys.version.split()[0],
        "cwd": os.getcwd(),
        "nonce": nonce,
        "junit_path": str(JUNIT) if JUNIT.exists() else None,
        "junit_sha256": junit_sha or None,
        "duration_sec": round(ended - started, 3),
        "wrapper": "pytest_receipt.py",
    }
    RECEIPT.write_text(json.dumps(receipt, indent=2) + "\n")
    print(f"wrote {RECEIPT} exit={proc.returncode}")
    return proc.returncode


if __name__ == "__main__":
    raise SystemExit(main())

Then add a check the next agent turn must pass.

Human or CI runs this. The model only reads it.

#!/usr/bin/env python3
"""check_receipt.py — fail if the wrapper did not run."""
import json
import sys
from pathlib import Path

p = Path("test-receipt.json")
if not p.exists():
    print("missing test-receipt.json; run pytest_receipt.py")
    sys.exit(2)
data = json.loads(p.read_text())
if data.get("wrapper") != "pytest_receipt.py":
    print("receipt was not stamped by the wrapper")
    sys.exit(2)
if data.get("exit_code") != 0:
    print(f"pytest failed: {data}")
    sys.exit(data.get("exit_code") or 1)
if not data.get("junit_sha256"):
    print("receipt has no junit hash")
    sys.exit(2)
print("receipt ok")

Example commands, also unexecuted here:

python pytest_receipt.py -q tests/
python check_receipt.py

Is this unbreakable? No. Read the limits below.

It still beats a chat bubble that says "passed".

Want a still cheaper smoke check before the wrapper?

Break the production function on purpose. Watch the new test.

If the suite stays green, the test never touched the patch.

Decision table

Use this before you trust a green story.

Signal you have	Proves pytest ran?	Proves the tests mean anything?
Chat said "all passed"	No	No
test_*.py exists	No	No
Agent-written junit.xml	No	No
Wrapper receipt, exit 0	Yes	Not yet
Receipt plus assertions that call the patch	Yes	Maybe
Same command green in CI	Yes	Better

Print that table next to your PR template.

I want reviewers to ask for the receipt path.

No path, no merge. That rule is the whole FAQ.

Where a free model and free server actually help

I iterate cheaply when the runner stays in the loop.

MonkeyCode offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That is useful for redrafting tests without burning a laptop.

It does not replace pytest. It does not stamp receipts.

Run pytest_receipt.py on that host the same way.

Then read test-receipt.json on the next turn.

The free parts lower the cost of another honest rerun.

They do not lower the need for an exit code.

If you try that loop, keep the wrapper in front.

Limits, and who should skip this

This wrapper is a seatbelt. It is not a vault.

An agent with shell can forge the receipt file.

So CI, not the agent, should be the last caller.

Environment drift still exists across images.

I am not claiming your remote box matches GitHub Actions.

Do not invent that match. Measure it when you care.

A hashed junit.xml also does not prove assertion quality.

Garbage tests can exit zero. You still have to read them.

Who should not use this approach?

• Anyone forbidden from sending code to a remote host.
• Suites that need production secrets to boot.
• Teams replacing code review with a JSON receipt.
• Projects with no pytest and no plan to add one.

If your tests are live charges against a vendor API, stop.

Do not aim a free remote loop at that.

If you cannot fail the new test by breaking the patch, stop.

You are collecting files, not signal.

The model I want in your head

The agent proposes a patch. The wrapper runs pytest.

The receipt is the memory. Chat is optional color.

Did the tests pass? Point at test-receipt.json.

Anything else is a myth with syntax highlighting.