My CI hadn't run a single one of those tests in two months and stayed green the whole time

I added a dependency to a test helper and forgot to declare it in the dev extra. Locally it was already installed, so everything passed. In CI it wasn't, and the tests that needed it did not fail.

They skipped.

Skips exit 0. The checkmark stayed green for two months.

Why it skips instead of failing

The pattern is one line, it's in every codebase, and it's usually correct:

aiosqlite = pytest.importorskip("aiosqlite")

That line means: this test needs something optional, and if it isn't here, move on. Which is right when the dependency is genuinely optional — a test for the Postgres backend on a machine with no Postgres shouldn't fail the build.

The problem is what it looks like from outside. importorskip is a decision that the test is optional, taken at import time, and the only place it surfaces is a number in a summary line nobody reads:

1284 passed, 168 skipped in 42.11s

168 skipped. Two months earlier it was 6. Nothing in that output tells you which number is the anomaly, and the exit code is 0 either way.

After declaring the dependency, the same suite reported 1452 passed. The 168 hadn't been slowly rotting. They'd stopped running all at once, on a specific commit, and the build had been green through every one of them.

The part that stung

I had a suite I trusted. I merged on green. The whole point of the suite was to tell me when I broke something, and for two months it had quietly stopped being able to.

Nothing was misconfigured. No warning. No deprecation. CI did exactly what it was asked. The failure mode was that the number of tests it ran was never something anyone asserted on.

The guard

There is no --min-tests flag. There is a hook, and it's eight lines:

# conftest.py
MIN_TESTS = 1400

def pytest_sessionfinish(session, exitstatus):
    collected = session.testscollected
    if collected < MIN_TESTS and not session.config.option.collectonly:
        session.exitstatus = 1
        print(f"\nFAIL: {collected} tests collected, floor is {MIN_TESTS}. "
              f"Did a dependency stop resolving?")

Three notes on that, all learned by getting it wrong.

testscollected, not passed. You want to catch tests that vanished, and a skipped test still counts as collected — so this alone would not have caught my bug. Which brings me to the second guard, below. Use the floor to catch tests that disappear from collection entirely (a broken import in a conftest, a renamed directory, a bad -k), because that's the failure that turns a 1452-test suite into 3 without a word.

Guard collectonly. Without it, pytest --collect-only in a subset trips your own floor and you'll waste twenty minutes.

Set the floor slightly below current, and raise it. Not at current — every new branch that hasn't added a test yet will fail. Below, and bump it when it drifts far.

For the skip case specifically, the guard is different and simpler:

# pytest.ini
[pytest]
addopts = -rs

-rs prints the reason for every skip. It doesn't fail anything — it just puts SKIPPED [168] tests/conftest.py:14: could not import 'aiosqlite' in the log where a human can see it. Two months of my logs had that line. I never read them, because they were green.

If you want it to actually fail, the strict version:

def pytest_sessionfinish(session, exitstatus):
    skipped = len(session.config.pluginmanager.get_plugin("terminalreporter").stats.get("skipped", []))
    if skipped > MAX_SKIPS:
        session.exitstatus = 1

I don't use that one. It fights with legitimately environment-dependent tests and I got tired of tuning MAX_SKIPS. The floor plus -rs was enough.

The real fix was upstream of all of it

The guards catch the symptom. The cause was that a dependency my tests could not run without was declared as though they could.

# before — aiosqlite nowhere, importorskip papers over it
[project.optional-dependencies]
dev = ["pytest", "pytest-cov", "ruff"]

# after
dev = ["pytest", "pytest-cov", "ruff", "aiosqlite"]

And then delete the importorskip for it. If the suite requires it, importorskip is a lie that makes the requirement look like a preference.

That's the rule I took away: importorskip is for dependencies you have decided the test can run without. If you haven't made that decision deliberately, you've made it accidentally.

Check yours in one command

If you have a CI suite you trust, this tells you the number you've never looked at:

pytest --collect-only -q 2>&1 | tail -1

Compare it to what you'd have guessed. Then run the real suite with -rs and read the skip reasons — all of them, once. Mine had a two-month-old import error sitting in plain text under a green checkmark.

The uncomfortable general version: every green build asserts that the tests that ran, passed. None of them assert that the tests ran.

Comments

[Compound Labs]

One edge case in the strict hook is -p no:terminal: get_plugin("terminalreporter") can return None, so the guard raises while inspecting skips. Did you test the hook with terminal reporting disabled?

[Juan Camilo Auriti]

Good catch, thanks for flagging that. I hadn't tested it in that configuration. To be fair, I never ran that version in production either, the article itself says I dropped it. But the bug you described is real, so here's a safe version:

def pytest_sessionfinish(session, exitstatus):
    reporter = session.config.pluginmanager.get_plugin("terminalreporter")
    if reporter is None:
        return
    skipped = len(reporter.stats.get("skipped", []))
    if skipped > MAX_SKIPS:
        session.exitstatus = 1

Probably one more reason to stick with the test floor approach and -rs. The strict version needs debugging for almost every environment-dependent edge case, and this is exactly the kind of thing that made me abandon it.

[Vlad Z]

Two months of green CI that ran nothing also means two months of CI minutes paid for a job doing no work

Curious whether anyone looked at the runtime trend, a sudden drop in duration is usually the first hint something stopped executing

[Juan Camilo Auriti]

You're right about the cost. Two months of CI minutes wasted is the embarrassing detail I left out of the article. On the runtime trend, good point: a sudden drop in execution time would have been the most visible signal, way more than a skipped count at the bottom of a log. The guard I showed checks how many tests get collected, but the duration trend is a complementary check that needs no code, just look at the pipeline graph. I like it because it's something you can spot without touching the config file.

[Vlad Z]

The pipeline graph is a good free check, and it's easy to automate too: alert when a job's runtime drops more than about 30 percent below its median over the last month

That catches a suite that silently ran nothing without touching the tests themselves

Would you add a runtime floor like that, or is the collected test count guard enough for your setup