My Backup Exited 0 for Three Weeks and Nothing Paged Me

A backup script on one of my boxes ran every night for three weeks. It exited 0. It wrote nothing. Nothing alerted, and from the system's perspective, nothing had failed.

I found out because I needed a file from six months ago.

This is not an unusual story. If you have run servers long enough, you have your own version of it. What made it interesting to me was how many different ways the same class of failure can hide, and how little of it is covered by the monitoring people typically install.

The failure that doesn't look like a failure

The mental model most of us carry is: backup → error → alert. But the failures that actually bite are the ones where the exit code is zero.

• The target path moved because a volume got remounted somewhere else, so rsync cheerfully copied nothing.
• The destination filled up, and the tool wrote a zero-byte file and called it a day.
• A cron entry was dropped during an OS upgrade. The job did not fail — it stopped existing.
• The script succeeded, but the snapshot retention logic deleted the new snapshot.
• systemd put the unit into a failed state weeks ago and nobody looked.

In none of these does the thing you are monitoring — "did the process exit non-zero" — fire.

So the question has to change shape. Instead of "did the job succeed", ask "how old is the newest artifact". That question is answerable without trusting the job at all:

find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

If the newest file is older than your RPO, you are in trouble regardless of what any exit code said. That one inversion is most of the value.

The second thing I got wrong: df -h

df -h said 40% free. The application still could not write, because inodes were exhausted — a mail queue had accumulated a few million tiny files.

df -h and df -i answer different questions, and the second one fails harder when it fails:

df -P  -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'

A related trap: percent-used is the wrong metric for alerting. A volume growing at 2 GB/hour is an emergency at 60%; one sitting at 84% for two years is not. What you want is time to full:

time_to_full ≈ (size_bytes − used_bytes) / growth_bytes_per_hour

That requires keeping a little history, which is exactly what people skip because the dashboard already "shows disk".

Clock skew, the silent one

This one surprised me. A box's clock drifted a few minutes. Nothing crashed. But:

• TLS validation against external endpoints began failing intermittently
• cron fired at surprising times, because it works off wall-clock time
• Correlating logs across machines became fiction

timedatectl will tell you whether the clock is synchronised. One line, almost nobody has the check, and when it is wrong it makes every other investigation harder.

systemctl --failed is criminally underused

$ systemctl --failed
  UNIT                    LOAD   ACTIVE SUB    DESCRIPTION
● [e-mail]    loaded failed failed Daily backup

A unit can sit in failed for months. It is not in your logs unless you are looking, and if it is the backup unit, the backup has been dead the whole time while the timer rescheduled itself every night.

What I ended up building

I wanted one command that answered: is anything about to break, and did last night's backup actually run? Ten read-only checks, aggregated:

$ vaultcheck audit

Vaultcheck audit — web-01 — overall: CRIT

CRIT  backup    newest artifact is 22d 4h old (threshold 26h)
WARN  disk      /var at 87% (5.2 GB free)
OK    inodes    max 31% (/)
OK    memory    1.4 GB / 8.0 GB used
OK    load      0.42 / 0.31 / 0.28
CRIT  systemd   [e-mail] failed
OK    cron      crontab present, 6 jobs
WARN  clock     not synchronised (drift ~184s)
OK    logs      log growth within limits
WARN  updates   14 pending, reboot required

Design constraints, and why

Standard library only. No pip, no virtualenv, no C extension build step. The artifact is a single 38 KB .pyz you can scp. Any dependency becomes a liability when you are installing something on thirty boxes, and a monitoring tool that can itself fail to install is a bad joke.

Read-only, no agent, no daemon. Nothing runs between invocations. There is no state to corrupt, nothing to keep alive, nothing to fight with your init system. You can verify that claim cheaply: run --only disk and watch what happens.

Exit codes designed for automation. --fail-on warn|crit makes it exit non-zero at a chosen severity, so it composes with cron, systemd timers or a heartbeat service instead of asking you to look at something.

No telemetry. No phone-home, no account, no licence check in the audit path. A tool that inspects your servers should be inspectable itself.

Output formats

vaultcheck audit --format text     # humans
vaultcheck audit --format json     # feed into anything
vaultcheck audit --format html -o report.html

JSON is the one I actually use: it lets me diff yesterday against today and only alert on change, which kills the "alert fires every night forever" failure mode that trains people to stop reading alerts.

What it isn't

Worth being blunt about, because I have seen these tools oversold:

• Not a replacement for Prometheus, Zabbix or Netdata. No time series, no graphing, no long-term storage.
• Linux only.
• Closed source, and paid — one-off price, no subscription.

The tradeoff I am making is "one command, no dependencies, works in ten seconds on a box you just SSH'd into" versus "fits into a mature observability stack". If you already run Prometheus with node_exporter and alert rules for backup freshness, you do not need this, and I would genuinely rather you use what you have.

If you do not, the free version of 80% of it is:

# backup freshness (oldest acceptable: 26h)
find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1

# disk and inodes
df -P  -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'

# failed units, cron presence, clock
systemctl --failed
systemctl is-enabled cron 2>/dev/null || systemctl is-enabled cronie 2>/dev/null
timedatectl | grep -i synchronized

The one thing I would tell my past self

Assert on the absence of good output, not the presence of errors.

Every failure in this article — the empty backup, the full inodes, the dead cron entry, the failed unit — is invisible if your only signal is "did the process complain". Make the happy path the thing you verify, and monitor for its absence.

Comments

[softverseAdmin]

Your line about asserting on the absence of good output, not the presence of errors, is the part that stuck with me, because a green run can still mean nothing was actually produced. I have the same habit with my own projects, which deploy via GitHub Actions and Vercel on push to main: I do not just trust the pipeline status, I check that the live deployment actually changed. The inode example also resonated, since df showing free space while writes fail is the kind of failure that looks impossible until you check the second metric. Do you also run a periodic restore test, or does the freshness check plus the artifact age give you enough confidence that the backup is usable?

[dafili]

Good question, and the honest answer is: no, freshness plus artifact age is not enough. It is the cheapest useful check, not a sufficient one.

I think of it as four levels of confidence, each costing more than the last:

1. Does an artifact exist and is it recent? That is find -printf '%T@', what we discussed. Catches "the job silently stopped". About a second.

2. Is the artifact structurally intact? tar -tzf, restic check, zstd -t. This catches truncated writes, which is exactly what happens when a disk fills mid-write. A partial file still has a recent mtime, so level 1 will happily pass it.

3. Can it be mounted or read back? restic mount, or extracting a random file to /tmp. Catches corrupt repositories and wrong keys.

4. An actual restore to a scratch machine. The only thing that proves a backup is usable. Everything above only proves it is present and well-formed.

For years I was sitting at level 1 and telling myself it was level 4.

What I do now on my own boxes is a cheap compromise: nightly level 2 (checksums, automated, alerts on failure), and a quarterly level 4 by hand. Restore three random files to a throwaway VM and open them. The calendar reminder matters more than the tooling, and the manual test is the one that never gets skipped precisely because it is rare and annoying.

# level 2, cron-able
restic check --read-data-subset=5%                          # exercises real bytes, not just indexes
tar -tzf /backups/$(ls -t /backups | head -1) >/dev/null    # catches truncation

One trap worth naming: --read-data-subset only checks the subset you pass. If your nightly check says 5%, corruption in the other 95% sits undetected until the day you need it. Rotate the subset, or accept that you are sampling.

Being straight about my own tool: Vaultcheck does not do restore tests. It is deliberately read-only and never writes anything, so verifying a backup by restoring it is outside what it does. What it does is levels 1 and 2, and more importantly it makes the absence of a fresh artifact a hard failure rather than something you happen to notice. That was the actual hole in my setup.

The restore test is still yours to run. If a tool ever claimed to cover level 4 for you, I would want to know exactly what it was doing to my disks.

[softverseAdmin]

That framing of four levels really clears it up for me — especially your admission that for years you were sitting at level 1 and telling yourself it was level 4, because I've done exactly that with a green pipeline and a live site I hadn't actually checked. Your compromise makes a lot of sense too: nightly level 2 checks so the cheap stuff is automated, and then a quarterly level 4 restore by hand, with the calendar reminder doing the heavy lifting. The warning about restic check --read-data-subset=5% only sampling that slice is the part I'll remember, since a passing check can still hide corruption in the other 95%. Do you rotate which subset you check each night, or do you just accept the sampling and let the quarterly restore catch the rest?

[dafili]

I rotate, and I would argue the rotation is the whole point rather than an optimisation.

--read-data-subset=5% samples randomly each run, which means two runs can overlap and the same 5% can be checked twice while another slice goes untouched for a year. Random sampling tells you the corruption rate, not that your data is intact.

The deterministic form is --read-data-subset=n/t, so I pin it to the day of the week and cover the repository completely in seven nights:

# cron: 0 4 * * *
restic check --read-data-subset=$(date +%u)/7

Same total I/O over a week as one full check, but spread out, and the part that matters is that every byte is guaranteed to be verified within seven days rather than probabilistically maybe-never. If a repository is too large for a seventh a night, drop to 1/28 and accept monthly coverage. The guarantee stays; only the window widens.

On whether that replaces the quarterly restore: no, and the reason is that they fail differently.

• restic check catches media and repository corruption: bit rot, truncated packs, bad keys.
• A restore catches logical failure, where the backup contains the wrong thing. The directory that was empty because a mount did not come up. The database dump that succeeded but is 0 bytes. The rsync that copied a bind-mount stub instead of the real volume.

The second class passes every integrity check ever written, because the data is perfectly intact. It is just the wrong data. That is the failure in the original post, and no amount of restic check would have caught it.

So the split I settled on: integrity on a weekly rotation, automated; correctness on a quarterly restore, by hand. The manual one stays manual on purpose, because automating the restore test gives you a green light you did not personally see, which is how I ended up trusting a dead backup for three weeks in the first place.

The uncomfortable part is that the quarterly restore is the only check that catches the class of failure that actually bit me, and it is also the one most likely to get skipped. That is why I put it on a calendar rather than in the tool.

[softverseAdmin]

The n/t pin is the piece I was missing — thank you. I had the percentage form filed as deterministic, and you have it exactly right: 5% re-rolls the dice every night, runs can overlap, and a slice can go untouched indefinitely. Pinning it to $(date +%u)/7 turns that into guaranteed seven-day coverage for the same total I/O. That is strictly better and I have stolen it.

Your integrity-vs-correctness split is also the sharper version of what I was circling. A check proves the data is intact; only a restore proves it is the right data. The failure that bit you was the second kind — perfectly intact, completely wrong — and it sails through every integrity check ever written.

The manual quarterly restore is the part my instincts still want to automate, but your argument is hard to dodge: an automated restore test hands you a green light you never personally saw, and that is precisely how the three weeks happened.

Full disclosure: this thread grew into a full write-up on my blog today, and I have just corrected the rotation section — it now teaches the n/t form, and the correction is credited to this conversation. It is here if you want to check I have it right this time: dev.to/softverseadmin/backup-verif...

[dafili]

Appreciate the disclosure, and the fix is right this time - the n/t form as written will genuinely cover a repository in seven nights.

Since level 4 now carries your whole argument, here is the part I got wrong for years, in case it saves you the same detour.

You cannot automate the declaration, only the comparison. A restore drill that a script decides is "fine" reproduces exactly the failure you described - a green light you never personally saw. But a drill that compares against a fingerprint you wrote down by hand is a different animal: an expected file list, a known file's sha256, a row count from the dump, one config value. The script proves equality to your declaration; only you can declare. That splits the occasional cleanly - automation runs nightly and shouts when the fingerprint drifts, a human writes the fingerprint once a quarter.

And put the drill itself under monitoring. The reason a calendar reminder fails is that nothing notices when it does not fire. Export days_since_last_successful_manual_restore as a gauge and page at, say, 120 days. A stale drill then looks exactly like a stale backup - same alerting path, same on-call, one fewer thing that depends on you remembering.

That is the only structure I have found where the quarterly restore survives contact with a busy quarter.

One note on sourcing, since you were good enough to disclose: the exchange that produced the n/t correction lives in the comments under "My Backup Exited 0 for Three Weeks and Nothing Paged Me" - worth pointing at if anyone ever asks where the rotation fix came from.

[c-wind]

The find on %T@ is the right idea, but rsync -a preserves mtime, so the newest timestamp on the destination is the source file's, not when the copy landed. I've had that exact check report a file as months old an hour after a good sync. Touch a stamp only after the transfer exits 0 and age that instead. maxdepth 1 -type f also misses a restic or borg repo, where the newest file at the root is usually a lock or config.

[dafili]

You are right, and this is the most useful correction on this post. I tested it rather than argue: I set a source file's mtime to 2025-01-15, ran rsync -a, and the command from the article reported the backup as 655 days old while the copy had just completed successfully.

It is slightly worse than that, too. rsync -a implies -t, so the destination directory's mtime gets set to the source directory's mtime as well. So there is no fallback where "at least the directory changed" saves you: if the source tree has not had new files added recently, everything on the destination looks old.

The stamp-file approach is the correct fix, and it is what I should have written:

rsync -a /src/ /backups/ && touch /var/lib/backup-stamp/nightly
find /var/lib/backup-stamp -name nightly -mmin +1560   # older than 26h

Two refinements, both of which bit me while thinking this through:

• Do not put the stamp inside the synced tree unless you exclude it, or the next rsync --delete removes it and you get a permanent false alarm. A path outside the backup target is safer than /backups/.last-success.
• The stamp proves the transport exited 0, not that the data is there. If the source directory is empty because a mount did not come up, rsync exits 0 and the stamp is fresh. Pair it with a cheap object count or byte total, so "success" means "ran and moved roughly what it usually moves".

On the second point, you are right that maxdepth 1 -type f is wrong for a restic or borg repo. There the adapter is not file age at all:

restic snapshots --last --json | jq -r '.[0].time'
borg info --last 1 --json | jq -r '.archives[0].time'

So the honest version of the rule is: age of the newest artifact means "the time the backup system last recorded a successful write", and how you obtain that is backend-specific. A stamp for rsync or tar, the snapshot list for restic or borg, and for a database dump the file mtime is fine because the dump is genuinely rewritten each run.

Thank you for this. I would rather be corrected in public than ship advice that quietly pages people at 3am about a backup that is working fine.

[dafili]

A few things I deliberately left out of the post, since it was already long enough.

On picking the threshold. The backup check is the one people get wrong most often, and it's almost always because the threshold is a round number someone chose once (24h, 7 days) rather than something derived from an actual RPO. If you don't have a defined RPO, a workable default is:

alert_after = 2 x backup_interval + expected_job_runtime

A nightly job that takes 40 minutes should alert at ~26h, not 24h. Otherwise one slow run pages you, you learn the alert is noisy, and you stop reading it — which is the actual failure.

On exit codes. If you're wiring any of this into cron, pick the codes deliberately so they compose. I use 0 ok / 1 warn / 2 crit, and keep a separate code for "the checker itself couldn't run" — that distinction matters, because a monitoring script that fails silently is worse than no monitoring script.

Two checks worth adding that aren't in the list above:

1. Inode growth rate, not inode percentage. Same argument as time-to-full for bytes: a filesystem gaining 50k inodes/hour is an emergency at 30%.
2. Backup target on the same physical device as the source. Cheap to detect via /proc/mounts, and it silently defeats the whole point of a backup. I've seen it twice on VPS setups where /backups was just a directory on the root volume.

On the "diff yesterday against today" idea. This is the bit that actually reduced alert noise for me. Alert on change, not on state: a disk parked at 87% forever shouldn't page anyone, a disk that went 62% -> 87% overnight should. It needs one line of persisted state, which is why most one-shot scripts don't do it.

Happy to go deeper on any of these if useful.