A backup checker I wrote printed this, on a backup target that no longer existed in any readable sense:
ok | last newest entry 0.0 h old
chmod 000 on that directory. os.listdir raised PermissionError for the user the check runs as. And the report said the newest backup was written just now, with no hedge of any kind, along with eleven other green lines and the summary All backup targets are fresh.
The directory was, at that moment, the only thing the check had actually looked at.
The eight lines responsible
newest = None
for root, dirs, files in os.walk(path, onerror=lambda _e: None):
for name in files:
try:
m = os.stat(os.path.join(root, name)).st_mtime
except OSError:
continue # every file unreadable -> skip
if newest is None or m > newest:
newest = m
if newest is None:
newest = os.stat(path).st_mtime # <- the fallback that lies
return (time.time() - newest) / 3600.0
Two lines carry the bug. The onerror handler is the first: os.walk ignores scandir errors by default, so onerror=lambda _e: None is a no-op wearing the costume of error handling. It looks like somebody thought about failure. Nothing is signaled either way — a walk over an unreadable directory yields nothing, exactly like a walk over an empty one.
The fallback is the second: when the walk found no files, the code asks the directory for its own mtime and treats that as the age of the newest artifact.
That fallback is not a reasonable guess. It answers a different question, and it answers it with a number that is always plausible.
What the permission bits actually gate
I reproduced it on a plain Linux box today, dropping to nobody before measuring, because as root every case below just works and proves nothing:
stat(execonly dir) [0111] ok
stat(execonly/db.dump) [0111] ok
listdir(execonly) [0111] PermissionError
stat(noperm dir) [0000] ok
stat(noperm/db.dump) [0000] PermissionError
listdir(noperm) [0000] PermissionError
os.walk(noperm) files: default=0 onerror=lambda:None=0
Read it in order, because the four lines tell the whole story:
-
os.staton a directory needs execute permission on its parent and nothing else. Mode0000on the directory itself does not stop it. -
os.staton a file inside a directory needs execute on that directory — you can stat a known name through an execute-only parent, and0111is enough. -
os.listdirneeds the read bit. That is the one operation that fails. -
os.walkswallows it either way, so "unreadable" and "empty" arrive at the caller as the same value: zero files.
So the fallback was guaranteed to succeed precisely in the case it was written for — the case where the walk had nothing to report. The failure mode is not "the code crashed on an edge case". The code takes the branch that cannot fail, and produces a fresh timestamp from a directory it could not open.
A directory's mtime is not a backup timestamp
Even with a readable directory, the fallback is answering the wrong question. Measured on the same box:
after create file dir mtime moved
after rewrite content dir mtime unchanged, file mtime moved
after rename dir mtime moved
A directory's mtime tracks changes to its entry list — create, unlink, rename. It does not move when a file's contents are rewritten in place. Which is the normal shape of nightly backups: pg_dump --file /backup/db.dump overwrites the same name every night, rsync into a fixed target, tar czf onto the same path. In that workload the directory mtime is the date the target was created, not the date of the last successful write.
Three of the four cases I measured were wrong, and they were wrong in opposite directions — all from the same branch:
| Case | Reported | Truth |
|---|---|---|
| directory unreadable, mtime recent |
ok, 0.0 h |
unknown — a green nobody earned |
| directory readable but empty |
ok, 0.0 h |
no backup exists |
| directory unreadable, mtime 240 h old |
STALE, 240 h |
unknown — a red that skipped the investigation |
The third row is the one that gets missed. A false red costs an argument with the on-call engineer, who will usually be right, and after the second argument the check gets muted — which is why a checker that lies in both directions ends up trusted in neither.
Three answers, not one
The fix is not a better fallback. It is refusing to compress three states into one:
cannot read directory -> CRIT "cannot read <path>: [Errno 13] Permission denied"
readable but empty -> CRIT "no files found under <path>"
readable with files -> only now, the freshness comparison
"I cannot see it" is never allowed to be translated into a statement about freshness in either direction. Saying "I don't know" out loud is what makes the rest of the report worth reading; borrowing "stale" or "fresh" to fill the gap means the reader trusts a claim no code ever made.
The test that holds it in place is three directories and one assertion per state, run as a non-root user:
mkdir -p /tmp/vc/{fresh,empty,locked}
touch /tmp/vc/fresh/db.dump
chmod 000 /tmp/vc/locked
su nobody -s /bin/sh -c 'your-check /tmp/vc/fresh /tmp/vc/empty /tmp/vc/locked'
-
/tmp/vc/lockedmust exit non-zero with an error naming the errno, not a freshness verdict. -
/tmp/vc/emptymust exit non-zero saying nothing was found. -
/tmp/vc/freshmust beok, and only that one.
If your checker runs as root in a container, run that last command as nobody or your test proves nothing at all — root ignores every permission bit you set. That mistake costs more audits than it should, including one of mine.
What this is not
It is not a story about a typo. Every line was written by someone trying to avoid a crash, and the result was a checker whose green light meant "I have no idea" in three different situations. A monitoring system earns its keep on the day something breaks, and the only thing that breaks it irreparably is a green tick that was never earned — because the tick is what stops the human from walking over to look.
That specific check is what I sell now, and the reason the tool exists is that this exact path exists in a lot of home-grown backup scripts:
Top comments (2)
The 'three answers, not one' framing is the part I'll be stealing for our own checks. A monitor that can't see what it's guarding should say 'unknown', never 'ok' — collapsing unknown into healthy is exactly how a missing backup goes unnoticed until the day you need it. The table row that stuck with me is the third one: a false red costs an argument with on-call, but the false green is the one that costs you a restore.
Do you treat 'cannot read' and 'empty' differently in alerting — for example paging for cannot-read but ticketing for empty — or do both go out at the same severity? I'm curious how you weight 'the check itself is broken' against 'there is no backup'.
Both page. "Cannot read" is your own blindness and "empty" is a missing backup — different fixes, same business claim: no verified copy exists, so ticketing either one is how it becomes a Monday problem.
Keep the distinction in the exit code, not in the message: two distinct non-zero codes (say 2 = cannot read, errno in stderr; 3 = readable but empty) let your router page both while the runbook splits. I rank "the check itself is broken" above "there is no backup" because a blind check also hides the next failure, and because it is cheap to prove: on my box
os.walkwith a swallowingonerrorreturned 0 files for both cases, so the exit code is the only place that difference survives.Reproduce as non-root, since root ignores every bit:
Not measured: how often the two co-occur in the wild, so I page on both rather than tune one of them into noise.