The Suite Looked Quiet for 48 Hours. Pytest Had Exited 5.

Have you ever trusted a test run because nothing printed FAILED, and the prompt came back almost immediately? I did, and then I spent two days chasing a broken deploy that was never the real bug. The service on the scratch box kept serving stale behavior, while my notes still said the suite was green. Why would I question a command that refused to scream?

This is a 48-hour field notebook, not a victory lap. I am writing down what I tried, what actually broke, and the small gate I would run again before another push. Every command below is a walkthrough you can reproduce on a throwaway tree; I am not attaching fake timings or a customer story to it.

Hour 0: the app disagreed with my notes

I had asked a coding assistant to add tests around a small Python service, then I pushed the tree to a scratch host so I could hit it from outside my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the tests and its free server option as the remote box that ran them after each push.

The assistant's last message was calm: no failures, suite looks good, safe to deploy. I skimmed the terminal like a human does, hunting for the word FAILED, and I did not print the process exit code. Have you noticed how easy it is to confuse a quiet log with a passing one?

What I tried first, in the wrong order

I treated the remote process as guilty because that is where the browser pain lived. The checklist from those first hours still makes me wince, because every item was reasonable and every item was beside the point.

1. I restarted the service and confirmed the process was still alive with a simple ps check.
2. I compared python3 --version on the laptop against the same command on the scratch host.
3. I reinstalled dependencies from requirements.txt and stared at pip output for a missing wheel.
4. I curled the local route and the remote route, then blamed a stale worker for the mismatch.
5. I asked the assistant to "re-run tests and fix whatever failed," which produced more quiet output.

None of those steps printed echo $? after pytest. That single omission is the whole story, and I still cannot believe I waited a day to type it.

Hour 18: the log was not green, it was empty

I finally reran the suite with collection visible, from the same directory the deploy script used on the server. The output was short enough to miss if you were already looking for a traceback.

cd /srv/app
python -m pytest -q
echo "pytest_exit:$?"
python -m pytest --collect-only -q
echo "collect_exit:$?"

Pytest printed collected 0 items, then it exited 5. Exit 5 is not a pass; it is the documented "no tests were collected" status, and it has been stable in the pytest reference for years. My eyes had been scanning for FAILED, and the assistant had summarized the same silence as success. Who taught both of us that an empty suite is a green suite?

The three files that made collection miss everything

The generated layout looked busy, which is how it survived a tired review. The configuration, though, described a tree that did not exist.

app/
  main.py
tests/
  app_test.py
  helper.py
pytest.ini
deploy.sh

# pytest.ini — this is the broken fixture, labeled as such
[pytest]
testpaths = test
python_files = test_*.py
python_functions = test_*

Three mismatches stacked on top of each other, and any one of them would have been enough.

• testpaths pointed at test/, while the assistant wrote files under tests/.
• The file was named app_test.py, which unittest likes, while this pytest config only loaded test_*.py.
• deploy.sh ran python -m pytest || true, so a 5 became a 0 before the script printed "done".

# deploy.sh — broken on purpose for the walkthrough
#!/bin/sh
set -u
python -m pytest || true
echo "tests complete"
exec python -m app.main

Would you have caught that || true at 1 a.m., after a model said the suite was fine? I did not. I also had PYTEST_ADDOPTS='-k not slow' exported on the server from an earlier experiment, which would have excluded every node even after I fixed the paths. Environment leftovers are not a theory when the same box keeps getting reused.

A gate I can actually rerun

I wanted a check that fails closed when collection is empty, without inventing a plugin ecosystem around it. The snippet below is the artifact from these notes: a tiny runner that prints the exit code, refuses status 5, and refuses a zero collection count even if a wrapper tries to be clever.

# tools/assert_pytest_collected.py
# Reproducible gate; run from the repo root.
from __future__ import annotations

import subprocess
import sys
from xml.etree import ElementTree as ET


def main() -> int:
    junit = "pytest-junit.xml"
    cmd = [
        sys.executable,
        "-m",
        "pytest",
        "--collect-only",
        "-q",
        f"--junitxml={junit}",
    ]
    proc = subprocess.run(cmd, check=False)
    print(f"pytest_collect_exit={proc.returncode}")
    if proc.returncode not in (0, 5):
        print("collection did not complete cleanly")
        return proc.returncode or 1
    if proc.returncode == 5:
        print("no tests collected; treating this as a failed gate")
        return 5
    try:
        root = ET.parse(junit).getroot()
    except (ET.ParseError, FileNotFoundError) as exc:
        print(f"junit xml missing or unreadable: {exc}")
        return 3
    collected = int(root.attrib.get("tests", "0"))
    print(f"collected={collected}")
    if collected == 0:
        print("junit reported zero tests; failing closed")
        return 5
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it the boring way, locally and on the scratch host, before you start the app process.

python tools/assert_pytest_collected.py
echo "gate_exit:$?"
python -m pytest -q
echo "full_suite_exit:$?"

If you prefer to stay inside pytest, a conftest.py hook can keep exit status 5 from being rewritten by a later wrapper. Label this as a proposal you should read before pasting into a huge suite.

# conftest.py
def pytest_sessionfinish(session, exitstatus):
    if session.testscollected == 0 and exitstatus == 0:
        session.exitstatus = 5

That hook is defensive, not magical. It does not fix || true, and it does not fix a deploy script that never inspects $?. The shell still has to care.

Decision table from the notes

I keep this next to the gate because the failure modes look similar in a yellow terminal.

• Quiet log, exit 0, collected N > 0: actual pass; start debugging the app, not the suite.
• Quiet log, exit 5, collected 0: path, name, or -k filter problem; do not deploy.
• Quiet log, exit 0, but || true in the wrapper: you deleted the signal; print $? before ||.
• Ran 0 tests from python -m unittest discover: unittest treats an empty run as OK; do not mix that with pytest status.
• Assistant summary says "no failures": ask it to quote collected and the numeric exit code, not a vibe.

Unittest's empty run is a separate trap. python -m unittest discover can print Ran 0 tests and still return 0, which is why I now refuse to let a model pick the runner in the same patch that adds files.

What broke, in one sentence each

The remote app was not frozen; it was running yesterday's behavior because today's tests never executed. The assistant was not lying in a dramatic way; it was summarizing a log that contained no FAILED lines. My review was not careful; it was pattern matching for the wrong token. The free server was not the villain; it was the first place the deploy script ran without my laptop's accidental local files.

That last point still matters. A local tests/ directory that never got committed will make your laptop collect tests while the scratch host collects none. After this incident I print git ls-files 'tests/*' 'test/*' in the same block as pytest, because collection cannot test a file git does not have.

git ls-files --stage -- 'tests/*' 'test/*' 'pytest.ini' 'conftest.py'
python tools/assert_pytest_collected.py

What I would repeat

I would still use a free model to draft tests, because blank files are worse than awkward files. I would still use a free server as the second machine, because the second machine is where wrapper scripts and leftover environment variables show up. I would not ask the model whether tests passed; I would ask it to paste collected, pytest_exit, and the first twenty lines of --collect-only.

I would also keep the gate in the deploy path, not in a README. A README does not run at the start of deploy.sh. If you already have a scratch box, wire the exit-code print before the next push and look at the number with your eyes.

Limitations, and who should skip this

This gate is wrong for repositories that intentionally collect zero tests on some jobs, such as docs-only packs or generate-then-collect plugins. It is also noisy if you use pytest plugins that create tests during collection in a way the junit file does not represent the way you expect. Do not drop || true removal into a script whose callers already treat pytest 5 as a skipped optional step, unless you want those callers to go red for a reason they never agreed to.

I am not claiming the assistant will always emit app_test.py, and I am not claiming a free server will always inherit PYTEST_ADDOPTS. Those were the failures in this notebook. Your next empty suite will probably be a different spelling of the same mistake: a quiet log, an unread exit code, and a deploy that believed both.

Forty-eight hours is a long time to spend on a missing directory letter. Printing pytest_exit:5 takes one extra line. Why did I wait until the app argued with me before I asked the suite to speak in numbers?