DEV Community

Hive80-lab
Hive80-lab

Posted on

We tested a restore for the first time in 3 years. Here's what broke.

#backup #devops #sre #productivity

The backup was fine. The restore was not.

Every small team I've met has the same setup: a backup that runs every night, a green checkmark in the logs, and zero proof it has ever been restored. Backups are graded on whether they run. Restores are graded on whether you get your business back. Those are different exams — and most teams only find out during the fire.

Last quarter we ran a 15-minute restore drill on a "perfectly healthy" setup. Three things broke.

1. The backup restore depended on a machine that no longer exists

The restore script had a hardcoded internal hostname of a jump box that was decommissioned 18 months ago. Backups kept running (nothing in the pipeline touched that hostname). Restores would have failed on day one. Fix: restore scripts must be self-contained — no hostnames, no paths to living infrastructure, just the archive and a fresh machine.

2. Nobody knew the decryption passphrase workflow

Encrypted backups, passphrase in one person's head, that person on vacation. A backup you cannot open is a sculpture. Fix: passphrase goes in the team vault with a documented handover, tested quarterly as part of the drill.

3. The archive was complete but the database inside was corrupt

tar succeeded nightly on a corrupt dump — the dump step failed after the compression step in the chain, but the old file from last week was still being packed. The green checkmark checked the wrong thing. Fix: verify the dump is fresh (mtime within 26h) and loadable (sqlite3 file 'PRAGMA integrity_check;' or a pg_restore --list) before it enters the archive.

The 15-minute drill itself

Once a quarter, on a spare machine or a $2 cloud box:

  1. Pull the newest backup archive.
  2. Restore it end-to-end with only the runbook and the vault passphrase.
  3. Open the app / load the data. Check one real record.
  4. Write down every step that needed a human memory instead of a document.

If step 4 has more than one line, your backup is a liability wearing a green checkmark.

Make it boring on purpose

The whole point is that the drill is scheduled, small, and repeats — so the failure modes surface in a calm quarter instead of a catastrophic week. If you'd rather not build the drill kit from scratch, I package the restore-drill checklist, the backup-verification guards, and the failure playbooks:

Top comments (0)