A backup job can report success for years while restoring nothing. The check everyone skips is the only one that matters: a real restore, against a real deadline, with a stopwatch running.
Here is the 45-minute restore drill I run on every estate I inherit. Schedule it monthly. It costs an hour and has saved clients from six-figure outages.
The drill
- Pick yesterday's backup of one live system (a database, a file share, one VM). Not a test copy — the actual artifact your job produced.
- Restore it to an isolated host. Never overwrite production. Note every credential, network path, and permission problem you hit — each one is a failure that would have been an outage at 3am.
- Time everything. The two numbers that matter: time-to-first-byte restored, and time-to-full-consistency. Compare both against your actual RTO. Most teams discover their real RTO is 8x what the runbook claims.
- Verify the data, not just the files. Open the app against the restored copy. Run its integrity checks. A restored file that the application rejects is a backup that does not exist.
- Write the gaps into the runbook the same day. Missing credential, expired service account, DNS not documented — each finding gets an owner and a deadline.
What the drill usually finds
- Backups that silently stopped including the thing you think they back up (new volume, new namespace, new table)
- Restore credentials owned by someone who left the company
- Encryption keys stored on the same system as the backups they unlock
- Restore bandwidth so slow your "4-hour RTO" is actually 3 days
That last one is the killer. Nobody notices restore speed until the day it is the only number that matters. The fix is usually boring: a cache tier, a local staging copy, a retention policy that keeps the most recent 24 hours on fast storage.
Make it routine
Automate the boring parts: a monthly calendar entry, a script that picks a random day's artifact and stages the isolated host, a one-page report (artifact, start time, restored bytes, app-check result, total minutes). The report is what turns "we have backups" into "we can recover, here is the evidence."
If your answer to "when did you last restore?" is a shrug, run the drill this week. It is the cheapest insurance audit you will ever do.
I package the drill and the reporting into kits you can run without hiring anyone:
- The First 30 Minutes — free incident quick-start checklist
- Ops Starter Kit — incident response for small teams — $14
- Automation Starter Pack — scripts that do the checking for you — $19
- Ops Mega Bundle — all kits in one download — $49
Code HIVE-LAUNCH30 takes 30% off any kit at checkout.
Top comments (0)