My Backup Exited 0 for Three Weeks and Nothing Paged Me

dafili — Sat, 31 Oct 2026 06:48:02 +0000

A backup script on one of my boxes ran every night for three weeks. It exited 0. It wrote nothing. Nothing alerted, and from the system's perspective, nothing had failed.

I found out because I needed a file from six months ago.

This is not an unusual story. If you have run servers long enough, you have your own version of it. What made it interesting to me was how many different ways the same class of failure can hide, and how little of it is covered by the monitoring people typically install.

The failure that doesn't look like a failure

The mental model most of us carry is: backup → error → alert. But the failures that actually bite are the ones where the exit code is zero.

The target path moved because a volume got remounted somewhere else, so rsync cheerfully copied nothing.
The destination filled up, and the tool wrote a zero-byte file and called it a day.
A cron entry was dropped during an OS upgrade. The job did not fail — it stopped existing.
The script succeeded, but the snapshot retention logic deleted the new snapshot.
systemd put the unit into a failed state weeks ago and nobody looked.

In none of these does the thing you are monitoring — "did the process exit non-zero" — fire.

So the question has to change shape. Instead of "did the job succeed", ask "how old is the newest artifact". That question is answerable without trusting the job at all:

find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

If the newest file is older than your RPO, you are in trouble regardless of what any exit code said. That one inversion is most of the value.

The second thing I got wrong: `df -h`

df -h said 40% free. The application still could not write, because inodes were exhausted — a mail queue had accumulated a few million tiny files.

df -h and df -i answer different questions, and the second one fails harder when it fails:

df -P  -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'

A related trap: percent-used is the wrong metric for alerting. A volume growing at 2 GB/hour is an emergency at 60%; one sitting at 84% for two years is not. What you want is time to full:

time_to_full ≈ (size_bytes − used_bytes) / growth_bytes_per_hour

That requires keeping a little history, which is exactly what people skip because the dashboard already "shows disk".

Clock skew, the silent one

This one surprised me. A box's clock drifted a few minutes. Nothing crashed. But:

TLS validation against external endpoints began failing intermittently
cron fired at surprising times, because it works off wall-clock time
Correlating logs across machines became fiction

timedatectl will tell you whether the clock is synchronised. One line, almost nobody has the check, and when it is wrong it makes every other investigation harder.

`systemctl --failed` is criminally underused

$ systemctl --failed
  UNIT                    LOAD   ACTIVE SUB    DESCRIPTION
● backup@daily.service    loaded failed failed Daily backup

A unit can sit in failed for months. It is not in your logs unless you are looking, and if it is the backup unit, the backup has been dead the whole time while the timer rescheduled itself every night.

What I ended up building

I wanted one command that answered: is anything about to break, and did last night's backup actually run? Ten read-only checks, aggregated:

$ vaultcheck audit

Vaultcheck audit — web-01 — overall: CRIT

CRIT  backup    newest artifact is 22d 4h old (threshold 26h)
WARN  disk      /var at 87% (5.2 GB free)
OK    inodes    max 31% (/)
OK    memory    1.4 GB / 8.0 GB used
OK    load      0.42 / 0.31 / 0.28
CRIT  systemd   backup@daily.service failed
OK    cron      crontab present, 6 jobs
WARN  clock     not synchronised (drift ~184s)
OK    logs      log growth within limits
WARN  updates   14 pending, reboot required

Design constraints, and why

Standard library only. No pip, no virtualenv, no C extension build step. The artifact is a single 38 KB .pyz you can scp. Any dependency becomes a liability when you are installing something on thirty boxes, and a monitoring tool that can itself fail to install is a bad joke.

Read-only, no agent, no daemon. Nothing runs between invocations. There is no state to corrupt, nothing to keep alive, nothing to fight with your init system. You can verify that claim cheaply: run --only disk and watch what happens.

Exit codes designed for automation. --fail-on warn|crit makes it exit non-zero at a chosen severity, so it composes with cron, systemd timers or a heartbeat service instead of asking you to look at something.

No telemetry. No phone-home, no account, no licence check in the audit path. A tool that inspects your servers should be inspectable itself.

Output formats

vaultcheck audit --format text     # humans
vaultcheck audit --format json     # feed into anything
vaultcheck audit --format html -o report.html

JSON is the one I actually use: it lets me diff yesterday against today and only alert on change, which kills the "alert fires every night forever" failure mode that trains people to stop reading alerts.

What it isn't

Worth being blunt about, because I have seen these tools oversold:

Not a replacement for Prometheus, Zabbix or Netdata. No time series, no graphing, no long-term storage.
Linux only.
Closed source, and paid — one-off price, no subscription.

The tradeoff I am making is "one command, no dependencies, works in ten seconds on a box you just SSH'd into" versus "fits into a mature observability stack". If you already run Prometheus with node_exporter and alert rules for backup freshness, you do not need this, and I would genuinely rather you use what you have.

If you do not, the free version of 80% of it is:

# backup freshness (oldest acceptable: 26h)
find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

# disk and inodes
df -P  -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'

# failed units, cron presence, clock
systemctl --failed
systemctl is-enabled cron 2>/dev/null || systemctl is-enabled cronie 2>/dev/null
timedatectl | grep -i synchronized

The one thing I would tell my past self

Assert on the absence of good output, not the presence of errors.

Every failure in this article — the empty backup, the full inodes, the dead cron entry, the failed unit — is invisible if your only signal is "did the process complain". Make the happy path the thing you verify, and monitor for its absence.

Two things I built around the checks above, since a few people asked in the comments:

I build a tool that does the freshness/restore checks described here: vaultcheck.dafili.cn — the audit command is essentially the four levels below, packaged so it runs in one shot and exits non-zero on failure.

Also, I extracted a list of US healthcare providers: npilift.dafili.cn — different project, same idea of going back to a primary source instead of trusting a reseller.

DEV Community: dafili