DEV Community

My Backup Exited 0 for Three Weeks and Nothing Paged Me

Renhua Wei on October 31, 2026

A backup script on one of my boxes ran every night for three weeks. It exited 0. It wrote nothing. Nothing alerted, and from the system's perspecti...
Collapse
 
softverseadmin profile image
Rabin Adhikari •

Your line about asserting on the absence of good output, not the presence of errors, is the part that stuck with me, because a green run can still mean nothing was actually produced. I have the same habit with my own projects, which deploy via GitHub Actions and Vercel on push to main: I do not just trust the pipeline status, I check that the live deployment actually changed. The inode example also resonated, since df showing free space while writes fail is the kind of failure that looks impossible until you check the second metric. Do you also run a periodic restore test, or does the freshness check plus the artifact age give you enough confidence that the backup is usable?

Collapse
 
dafili profile image
Renhua Wei •

Good question, and the honest answer is: no, freshness plus artifact age is not enough. It is the cheapest useful check, not a sufficient one.

I think of it as four levels of confidence, each costing more than the last:

  1. Does an artifact exist and is it recent? That is find -printf '%T@', what we discussed. Catches "the job silently stopped". About a second.

  2. Is the artifact structurally intact? tar -tzf, restic check, zstd -t. This catches truncated writes, which is exactly what happens when a disk fills mid-write. A partial file still has a recent mtime, so level 1 will happily pass it.

  3. Can it be mounted or read back? restic mount, or extracting a random file to /tmp. Catches corrupt repositories and wrong keys.

  4. An actual restore to a scratch machine. The only thing that proves a backup is usable. Everything above only proves it is present and well-formed.

For years I was sitting at level 1 and telling myself it was level 4.

What I do now on my own boxes is a cheap compromise: nightly level 2 (checksums, automated, alerts on failure), and a quarterly level 4 by hand. Restore three random files to a throwaway VM and open them. The calendar reminder matters more than the tooling, and the manual test is the one that never gets skipped precisely because it is rare and annoying.

# level 2, cron-able
restic check --read-data-subset=5%                          # exercises real bytes, not just indexes
tar -tzf /backups/$(ls -t /backups | head -1) >/dev/null    # catches truncation
Enter fullscreen mode Exit fullscreen mode

One trap worth naming: --read-data-subset only checks the subset you pass. If your nightly check says 5%, corruption in the other 95% sits undetected until the day you need it. Rotate the subset, or accept that you are sampling.

Being straight about my own tool: Vaultcheck does not do restore tests. It is deliberately read-only and never writes anything, so verifying a backup by restoring it is outside what it does. What it does is levels 1 and 2, and more importantly it makes the absence of a fresh artifact a hard failure rather than something you happen to notice. That was the actual hole in my setup.

The restore test is still yours to run. If a tool ever claimed to cover level 4 for you, I would want to know exactly what it was doing to my disks.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

That framing of four levels really clears it up for me — especially your admission that for years you were sitting at level 1 and telling yourself it was level 4, because I've done exactly that with a green pipeline and a live site I hadn't actually checked. Your compromise makes a lot of sense too: nightly level 2 checks so the cheap stuff is automated, and then a quarterly level 4 restore by hand, with the calendar reminder doing the heavy lifting. The warning about restic check --read-data-subset=5% only sampling that slice is the part I'll remember, since a passing check can still hide corruption in the other 95%. Do you rotate which subset you check each night, or do you just accept the sampling and let the quarterly restore catch the rest?

Thread Thread
 
dafili profile image
Renhua Wei •

I rotate, and I would argue the rotation is the whole point rather than an optimisation.

--read-data-subset=5% samples randomly each run, which means two runs can overlap and the same 5% can be checked twice while another slice goes untouched for a year. Random sampling tells you the corruption rate, not that your data is intact.

The deterministic form is --read-data-subset=n/t, so I pin it to the day of the week and cover the repository completely in seven nights:

# cron: 0 4 * * *
restic check --read-data-subset=$(date +%u)/7
Enter fullscreen mode Exit fullscreen mode

Same total I/O over a week as one full check, but spread out, and the part that matters is that every byte is guaranteed to be verified within seven days rather than probabilistically maybe-never. If a repository is too large for a seventh a night, drop to 1/28 and accept monthly coverage. The guarantee stays; only the window widens.

On whether that replaces the quarterly restore: no, and the reason is that they fail differently.

  • restic check catches media and repository corruption: bit rot, truncated packs, bad keys.
  • A restore catches logical failure, where the backup contains the wrong thing. The directory that was empty because a mount did not come up. The database dump that succeeded but is 0 bytes. The rsync that copied a bind-mount stub instead of the real volume.

The second class passes every integrity check ever written, because the data is perfectly intact. It is just the wrong data. That is the failure in the original post, and no amount of restic check would have caught it.

So the split I settled on: integrity on a weekly rotation, automated; correctness on a quarterly restore, by hand. The manual one stays manual on purpose, because automating the restore test gives you a green light you did not personally see, which is how I ended up trusting a dead backup for three weeks in the first place.

The uncomfortable part is that the quarterly restore is the only check that catches the class of failure that actually bit me, and it is also the one most likely to get skipped. That is why I put it on a calendar rather than in the tool.

Thread Thread
 
softverseadmin profile image
Rabin Adhikari •

The n/t pin is the piece I was missing — thank you. I had the percentage form filed as deterministic, and you have it exactly right: 5% re-rolls the dice every night, runs can overlap, and a slice can go untouched indefinitely. Pinning it to $(date +%u)/7 turns that into guaranteed seven-day coverage for the same total I/O. That is strictly better and I have stolen it.

Your integrity-vs-correctness split is also the sharper version of what I was circling. A check proves the data is intact; only a restore proves it is the right data. The failure that bit you was the second kind — perfectly intact, completely wrong — and it sails through every integrity check ever written.

The manual quarterly restore is the part my instincts still want to automate, but your argument is hard to dodge: an automated restore test hands you a green light you never personally saw, and that is precisely how the three weeks happened.

Full disclosure: this thread grew into a full write-up on my blog today, and I have just corrected the rotation section — it now teaches the n/t form, and the correction is credited to this conversation. It is here if you want to check I have it right this time: dev.to/softverseadmin/backup-verif...

Thread Thread
 
dafili profile image
Renhua Wei •

Appreciate the disclosure, and the fix is right this time - the n/t form as written will genuinely cover a repository in seven nights.

Since level 4 now carries your whole argument, here is the part I got wrong for years, in case it saves you the same detour.

You cannot automate the declaration, only the comparison. A restore drill that a script decides is "fine" reproduces exactly the failure you described - a green light you never personally saw. But a drill that compares against a fingerprint you wrote down by hand is a different animal: an expected file list, a known file's sha256, a row count from the dump, one config value. The script proves equality to your declaration; only you can declare. That splits the occasional cleanly - automation runs nightly and shouts when the fingerprint drifts, a human writes the fingerprint once a quarter.

And put the drill itself under monitoring. The reason a calendar reminder fails is that nothing notices when it does not fire. Export days_since_last_successful_manual_restore as a gauge and page at, say, 120 days. A stale drill then looks exactly like a stale backup - same alerting path, same on-call, one fewer thing that depends on you remembering.

That is the only structure I have found where the quarterly restore survives contact with a busy quarter.

One note on sourcing, since you were good enough to disclose: the exchange that produced the n/t correction lives in the comments under "My Backup Exited 0 for Three Weeks and Nothing Paged Me" - worth pointing at if anyone ever asks where the rotation fix came from.

Collapse
 
amorizz profile image
Amorizz •

One more for the list under "The failure that doesn't look like a failure": pipelines. pg_dump app | gzip > app.sql.gz returns gzip's status, so if pg_dump dies on auth or a dropped connection halfway, gzip still exits 0 and you get a valid .gz with a partial dump inside. set -o pipefail at the top of the script makes the pipeline fail when any stage fails. An archive integrity check won't catch it either: gzip -t passes on that file, because the compression is fine. Only a restore, or at least checking the last lines for pg_dump's "dump complete" footer, shows the truncation.

Collapse
 
dafili profile image
Renhua Wei •

Good catch, and it is worse than the exit code suggests. I spent this morning checking it against a real PG 15 cluster, and the check that feels most rigorous turns out to be blind in exactly the same way.

Confirmed as you describe. Killing pg_dump about a fifth of the way through pg_dump app | gzip > app.sql.gz gives exit 0, gzip -t passes, and what is left is a perfectly well-formed archive at 2.9% of its normal size. The only thing missing is the -- PostgreSQL database dump complete footer.

Two additions from the testing.

pg_restore -l is the same trap wearing a better suit. For -Fc the TOC is written before the data blocks, so a truncated archive lists its contents quite happily. On a dump cut at 164 KB out of 2.7 MB, pg_restore -l returned exit 0 with all 7 entries present. Only pg_restore -f /dev/null caught it, with could not read from input file: end of file. Worth knowing because that is the check people add instead of verifying the footer, and it gives the same false green.

pipefail is bash-only. On Debian and Ubuntu /bin/sh is dash, where set -o pipefail is a hard error — Illegal option -o pipefail. Cron runs #!/bin/sh scripts under dash, so a pipeline that is protected when you test it by hand is either unprotected or dead on line 1 once it is on a schedule.

The structural fix is to drop the pipeline entirely. pg_dump -Z6 -f app.sql.gz makes pg_dump compress its own output, so its exit status covers the whole job and no pipefail is needed. The side benefit: a kill mid-stream leaves an unfinalised gzip stream, so gzip -t fails as well. Two independent signals instead of one.

The cheapest guard either way still needs no restore at all: compare the size against a rolling median. 3% of normal is not a shape you need to decompress to notice.

Collapse
 
amorizz profile image
Amorizz •

The dash point is a fair hit. My comment quietly assumed bash, which is exactly the setup that hides it. One more cron trap in the same family: SHELL=/bin/bash at the top of a crontab changes the shell cron uses for the command line, but a script called by path still runs under its own shebang, so #!/bin/sh stays dash. Agreed that pg_dump -Z6 -f app.sql.gz is the real fix, since one process owns the exit status. Newer releases also accept the spelled-out --compress=gzip:6, and the plain number still works.

Thread Thread
 
dafili profile image
Renhua Wei •

the SHELL-versus-shebang split is the sharper one, and it's what makes "we fixed
it in the crontab" feel true while it isn't. we ended up where you did: -Z6 so
one process owns the status, plus a footer grep regardless of exit code — exit 0
was never the signal.

Collapse
 
dafili profile image
Renhua Wei •

Follow-up rather than a separate argument: I built a test rig around your pipeline case instead of just nodding at it, and wrote it up here.

dev.to/dafili/three-checks-that-lo...

The short version is that it is worse than the exit code suggests. Two things I did not expect going in:

pg_restore -l is blind to the same truncation on -Fc archives, because the TOC is written before the data blocks. A dump cut at 6% of its size still listed all seven entries and exited 0. Only pg_restore -f /dev/null caught it, with "could not read from input file: end of file".

And set -o pipefail is bash-only. Under dash — which is what /bin/sh is on Debian and Ubuntu, and therefore what cron runs — it is an error, not a no-op, so the guard either kills the script on line one or leaves the pipeline unprotected depending on how you wrote it.

Your footer point held up exactly as stated, for the record. Worth an eye if you have a minute.

Collapse
 
amorizz profile image
Amorizz •

Read the write-up. The trailer detail is the part I wouldn't have guessed: the piped gzip writes a correct CRC32 and ISIZE around a truncated payload, while a killed pg_dump -Z6 leaves the stream unfinalised, so the same truncation gets opposite gzip -t verdicts. On pg_restore -l, I think there's one more layer. When -Fc writes to a seekable file, pg_dump goes back at the end and rewrites the TOC with data offsets, so a cut file keeps the early TOC without them. Did your 163,840-byte copy come from a killed dump or from head -c on a complete one?

Collapse
 
c-wind profile image
c-wind •

The find on %T@ is the right idea, but rsync -a preserves mtime, so the newest timestamp on the destination is the source file's, not when the copy landed. I've had that exact check report a file as months old an hour after a good sync. Touch a stamp only after the transfer exits 0 and age that instead. maxdepth 1 -type f also misses a restic or borg repo, where the newest file at the root is usually a lock or config.

Collapse
 
dafili profile image
Renhua Wei •

You are right, and this is the most useful correction on this post. I tested it rather than argue: I set a source file's mtime to 2025-01-15, ran rsync -a, and the command from the article reported the backup as 655 days old while the copy had just completed successfully.

It is slightly worse than that, too. rsync -a implies -t, so the destination directory's mtime gets set to the source directory's mtime as well. So there is no fallback where "at least the directory changed" saves you: if the source tree has not had new files added recently, everything on the destination looks old.

The stamp-file approach is the correct fix, and it is what I should have written:

rsync -a /src/ /backups/ && touch /var/lib/backup-stamp/nightly
find /var/lib/backup-stamp -name nightly -mmin +1560   # older than 26h
Enter fullscreen mode Exit fullscreen mode

Two refinements, both of which bit me while thinking this through:

  • Do not put the stamp inside the synced tree unless you exclude it, or the next rsync --delete removes it and you get a permanent false alarm. A path outside the backup target is safer than /backups/.last-success.
  • The stamp proves the transport exited 0, not that the data is there. If the source directory is empty because a mount did not come up, rsync exits 0 and the stamp is fresh. Pair it with a cheap object count or byte total, so "success" means "ran and moved roughly what it usually moves".

On the second point, you are right that maxdepth 1 -type f is wrong for a restic or borg repo. There the adapter is not file age at all:

restic snapshots --last --json | jq -r '.[0].time'
borg info --last 1 --json | jq -r '.archives[0].time'
Enter fullscreen mode Exit fullscreen mode

So the honest version of the rule is: age of the newest artifact means "the time the backup system last recorded a successful write", and how you obtain that is backend-specific. A stamp for rsync or tar, the snapshot list for restic or borg, and for a database dump the file mtime is fine because the dump is genuinely rewritten each run.

Thank you for this. I would rather be corrected in public than ship advice that quietly pages people at 3am about a backup that is working fine.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

Comparing the age of the newest artifact against the RPO is the check I wish more teams had. An exit code tells you the script finished, while artifact age tells you whether the business can recover, and only the second one matters at 3 a.m.

The rsync point from c-wind is a good catch. Since rsync -a keeps the source mtime, a fresh copy can look three weeks old, so touching a stamp file after a good run is the honest signal. Amorizz's pipefail note on pg_dump into gzip is the same lesson from the other side: a pipeline can succeed while the part you care about failed.

Your line that you cannot automate the declaration, only the comparison, is what we try to get clients to agree on at Ascendra Ventures when we scope DevOps work. Someone has to own the RPO number.

How often do you run the read-data-subset check compared with a full restore?

Collapse
 
dafili profile image
Renhua Wei •

Nightly, and the full restore is quarterly — but I would be careful about calling that a schedule, because the ratio is not really 7:1. The two checks answer different questions and I would not want the nightly number to sound like coverage it does not provide.

The nightly one is restic check --read-data-subset=n/7, with n pinned to $(date +%u). I rotate rather than sample because --read-data-subset=5% picks randomly each run: two runs can overlap and the same 5% gets read twice while another slice goes untouched for a year. Random sampling measures the corruption rate; it does not establish that the data is intact. The deterministic form gives you the second thing.

One caveat I only found by testing it, since it matters for how much you trust the nightly: 7 nights does not mean every pack is read if you have fewer than 7 packs. I set up a small repo and ran all seven:

2 packs in 7 groups:
  n=1  read group #1 of 1
  n=2  read group #2 of 0
  n=3  read group #3 of 0
  ...
  n=7  read group #7 of 0

4 packs in 7 groups:
  n=1..#4  one pack each
  n=5..#7  read group #N of 0
Enter fullscreen mode Exit fullscreen mode

restic hashes packs into 7 buckets. With 2 packs, five nights verify nothing at all and report "no errors were found" — which is true and completely uninformative. With 4 packs, three nights do nothing. You only get the guarantee of full coverage once the pack count is comfortably above 7, and the output line read group #N of 0 is the tell.

So the practical form is: keep the nightly n/7, and alert on a night whose output contains of 0. That converts a silently empty check into a visible one, which is the same principle as the rest of this thread. On a repository with a handful of packs it is worth checking restic stats --mode raw-data first.

The quarterly restore is manual and stays that way. Three random files out of a throwaway VM, opened and checked, plus one SELECT count(*) against the live database compared with what the dump reports — that last one catches a dump that is internally consistent but was taken from a replica that had stopped applying, which no amount of checksum verification will tell you.

The honest reason for the asymmetry is I/O and blast radius, not rigour. The nightly check reads bytes and can run unattended; the restore writes to a machine, needs somewhere safe to write, and takes long enough that it costs a decision. Automating the restore means automating the decision about what "a good restore" looks like, and that is the part I have not found a way to do honestly.

On owning the RPO number — agreed, and I would go slightly further: whoever owns it should also own the days_since_last_successful_manual_restore gauge, because an RPO nobody is paged on is a preference, not a commitment.

Collapse
 
dafili profile image
Renhua Wei •

A few things I deliberately left out of the post, since it was already long enough.

On picking the threshold. The backup check is the one people get wrong most often, and it's almost always because the threshold is a round number someone chose once (24h, 7 days) rather than something derived from an actual RPO. If you don't have a defined RPO, a workable default is:

alert_after = 2 x backup_interval + expected_job_runtime
Enter fullscreen mode Exit fullscreen mode

A nightly job that takes 40 minutes should alert at ~26h, not 24h. Otherwise one slow run pages you, you learn the alert is noisy, and you stop reading it — which is the actual failure.

On exit codes. If you're wiring any of this into cron, pick the codes deliberately so they compose. I use 0 ok / 1 warn / 2 crit, and keep a separate code for "the checker itself couldn't run" — that distinction matters, because a monitoring script that fails silently is worse than no monitoring script.

Two checks worth adding that aren't in the list above:

  1. Inode growth rate, not inode percentage. Same argument as time-to-full for bytes: a filesystem gaining 50k inodes/hour is an emergency at 30%.
  2. Backup target on the same physical device as the source. Cheap to detect via /proc/mounts, and it silently defeats the whole point of a backup. I've seen it twice on VPS setups where /backups was just a directory on the root volume.

On the "diff yesterday against today" idea. This is the bit that actually reduced alert noise for me. Alert on change, not on state: a disk parked at 87% forever shouldn't page anyone, a disk that went 62% -> 87% overnight should. It needs one line of persisted state, which is why most one-shot scripts don't do it.

Happy to go deeper on any of these if useful.