Your check says PASS. It looked at zero files.

#testing #automation #python #devops

A verification step that finds nothing looks exactly like a verification step that
finds nothing wrong. Both print PASS. Both exit 0. Only one of them actually did
anything.

I shipped this bug three times in the same codebase before I understood the shape of it.

The bug

A scanner walked a directory and reported leaked secrets:

def main(paths):
    findings = []
    for root in paths:
        for f in glob.glob(root + "/**/*", recursive=True):
            findings += scan(f)
    print(f"files scanned: {len(seen)} | leaks: {len(findings)}")
    if findings:
        sys.exit(1)
    print("PASS")

It worked. For two months it printed PASS before every release.

Then I passed it a list of files instead of a directory. glob("some/file.mp4/**/*")
expands to nothing. Zero files scanned. Zero findings. PASS. Exit 0.

The caller was the release script. Every release for those two months had been
"verified" by a scanner that had looked at nothing at all.

Nothing errored. Nothing warned. The log line even told me the truth —
files scanned: 0 — and I had read past it every single time, because the word next
to it was PASS.

Why this class of bug survives

Most failures announce themselves. This one impersonates success.

Three properties make it durable:

The output is indistinguishable from the good case. A clean scan and a scan-of-nothing produce the same word.
It fails open. The guard is supposed to block. When it breaks, it stops blocking — which is silent by definition.
It gets more trusted over time. Every green run is evidence the check works. Two months of green is two months of accumulating false confidence.

That third one is the dangerous part. The longer it runs, the less anyone looks.

The fix is one line of policy, not one line of code

if scanned == 0:
    print("BLOCK: scanner matched zero files — refusing to report PASS")
    sys.exit(3)

A check that examined nothing must not be allowed to report success.

Note the distinct exit code. 1 means "found a problem". 3 means "could not do my
job". Those are different states and the caller should be able to tell them apart.
Collapsing them into "non-zero" throws away the one signal that would have caught this.

While I was there I added two more:

if missing_paths:
    print("BLOCK: path does not exist:", missing_paths)
    sys.exit(2)

print(f"[verify] files scanned: {scanned} | findings: {len(findings)}")

That last line runs on the success path. The happy path now has to state what it
looked at. If the number is zero, you see it at the moment you would otherwise be
reassured.

Where else this hides

Once you have the shape, you find it everywhere:

Check	Silent-zero failure
Test runner	Pattern matches no tests. "0 passed" is green in several runners.
Linter	Ignore file swallows the whole tree.
Backup verify	Compares a manifest that is itself empty.
Grep-based guard in CI	Typo in the pattern. Nothing matches. Gate opens.
Schema check	Config fails to parse, exception swallowed, empty dict validates fine.

That last one bit me too, separately: a JSON file written by PowerShell carried a
UTF-8 BOM. json.load(open(path, encoding="utf-8")) raised, the exception was
caught and turned into None, and a dashboard rendered an empty table instead of
an error. The table looked fine. It was just empty.

The fix there was encoding="utf-8-sig". The lesson was the same one: the failure
had a display mode that looked like a working system.

The rule I use now

When a check reports success, ask what it examined. If it cannot tell you a number,
it is not evidence.

Concretely, for anything that gates a release, a deploy, or a payment:

Print the denominator, not just the verdict.
Make "examined nothing" a distinct, failing state.
Use separate exit codes for "found a problem" and "could not run".
Write one regression test that feeds the checker an input matching nothing, and assert that it fails.

That last test is the one nobody writes. It is also the only one that would have
caught all three of my versions of this bug.

I packaged this as a tool

The guard above is four lines, but the rules around it (distinct exit codes, literal
paths that do not exist, counting from the check's own output) turned out to be worth
writing down once. It is one file, standard library only:

https://github.com/elwakeupman-shhh/zero-match-guard

# blocks instead of passing when the glob matched nothing
python zero_match_guard.py --paths "dist/**/*.js" -- npm run lint

It ships with the test nobody writes: feed the checker an input matching nothing, and
assert that it fails.

Why I care about this more than most

Almost everything I build is this kind of tool: a small script that watches something
and tells you whether it is fine. Health checks, integrity verification, log analysis,
scheduled monitors. The entire value of that category is that you stop looking, because
the script is looking for you.

Which means a monitor that lies is worse than no monitor. No monitor leaves you
uncertain, and uncertainty makes you check. A green dashboard makes you stop checking —
and then you find out two months later.

So now every one of my checks answers two questions instead of one:

Did you find a problem?
How many things did you look at?

I build single-file Python tools of exactly this kind: service and disk watchdogs,
integrity verification, log analysis, scheduled monitors. Zero third-party dependencies,
one file, meaningful exit codes — small enough that you can read the whole thing before
you trust it with anything. Available for hire on Fiverr.