DEV Community

Hive80-lab
Hive80-lab

Posted on

Your backups are lying to you: the 5-minute restore test most small teams skip

#sysadmin #devops #backup #smallbusiness

The backup that didn't exist

A client once told me, "We're fine, we back up every night." When their file server died, we opened the backup share: the job had been failing for four months. The cron entry was there. The script was there. The email from the script went to a mailbox nobody read. Every night it silently wrote rsync: connection refused into a void.

A backup you have never restored is a rumor. That's the whole post, but let's make it practical.

The 5-minute restore test

Once a month, on the first Monday, do exactly this:

  1. Pick one thing — the most boring-but-critical item (the accounting folder, the mail store, the customer DB).
  2. Restore it to a scratch location — not over the live copy. restore/2026-01/ is fine.
  3. Open one file from it — actually open it. Checksums can lie; a file that opens in Excel doesn't.
  4. Time it. Write the minutes on the wall. "Restore took 22 minutes" is an SLA you can promise a client. "We have backups" is a prayer.
  5. Delete the scratch copy — restore space, disk full, everyone forgets, next month's test fails for the dumbest reason.

That's it. Five minutes, once a month. If step 2 or 3 fails, you just prevented the worst day of your year, for free.

The failure modes I see over and over

  • The job runs, the check doesn't. A green dot next to "backup" should mean verified, not attempted.
  • Email reports to nobody. If the failure email lands in a dead mailbox, the job is decorative.
  • One copy, one disk, one office. Same building, same fire, same thief, same ransomware.
  • The restore path was never tested with the person who'd actually do it. If only one admin knows it and they're on holiday, you don't have a restore procedure; you have a lottery.

Make it a script, not a resolution

The teams that actually do this don't have more discipline — they have a checklist file and a cron entry. The test script runs, restores to scratch, opens a canary file, and prints a one-line verdict: RESTORE TEST PASS (22m, 4.1GB, canary ok) or it fails loudly.

I packaged exactly this pattern — the monthly restore test, the backup-log scanner that catches silent failures, and the one-page morning report — into a starter kit so I stop rebuilding it per client:

Even if you never buy anything: run the 5-minute test this month. The first one finds something. It always finds something.

I run ops for small teams and write about the boring automations that keep them out of trouble. More at https://hive80lab.gumroad.com

Top comments (0)