We had backups. Nightly, automated, three regions, a green checkmark in the dashboard every morning for years. We were proud of our backups right up until the afternoon we actually needed to restore one and discovered it didn't work, had never worked, and had been silently producing unusable files the entire time.
The backup job ran fine. It connected, dumped, uploaded, exited zero. Every step succeeded. What no one had ever done was take one of those files and restore it into a real database. When we finally tried, under pressure, with data on the line, the restore failed on a version mismatch and a truncated dump that the backup job had never noticed because writing a broken file is still "success" as far as the job is concerned. We had spent years backing up nothing, confidently.
The lesson is blunt and I now repeat it to everyone: a backup you have never restored is not a backup. It's a hope. It's a file you pay to store and pray you never open. The only thing that proves a backup works is a restore, and if the first restore you ever attempt is during a real disaster, you are running your recovery process for the first time at the worst possible moment.
We changed it completely. A scheduled job now restores the latest backup into a throwaway database and runs real checks against it — row counts in a sane range, key tables present, a few known records readable. If the restore fails or the data looks wrong, that pages someone, the same as any production alert. A backup that can't be restored is treated as an outage, because that's exactly what it is, just a delayed one.
The deeper principle covers more than backups. Any recovery mechanism you haven't exercised is a guess. Failover you've never triggered, a standby you've never promoted, a runbook you've never followed — all the same lie in different costumes. If it only runs during a disaster, then disaster day is your test day, and you do not want to discover your assumptions are wrong with customers watching.
Restore your backups on a schedule. An untested recovery isn't insurance. It's a story you tell yourself until reality asks for the file.
– Sergey Shinder
Top comments (0)