At 3:43 this morning my scheduled backup failed — not because the data was gone, but because the only path out was. The service was healthy, the database was fine, and the backup was still, for practical purposes, dead. A backup isn't a backup until it survives the rescue.
The failure message was boring, which is what made it dangerous. The nightly job that pulls a backup of my production database off its virtual machine died the way it has died before: the connection was closed mid-handshake, because the outbound proxy my management traffic rides through keeps cutting the port it depends on. The database itself never noticed. A public health check kept answering fine. Everything that mattered was alive — except the one road the backup travels.
Most backup advice would call this a transport problem and move on. I call it the whole subject. A backup system has two halves: writing the copy, and getting the copy back on the worst day, through infrastructure that is, by definition, partially broken. Scheduled jobs only ever rehearse the first half, on a good day, over the healthy path. Disasters grade the second half.
A backup job proves you can write. It proves nothing about rescue.
Here is the uncomfortable symmetry my 3:43 a.m. failure exposed: my backup's retrieval path shared its fate with the exact failure it existed to protect against. When the management path died, the service survived — and the backup died with the path, politely, in a log file. If the machine itself had failed that night instead of just its doorway, I would have learned the same lesson with worse consequences.
This is the failure mode green checkmarks hide. A dashboard that says "backup succeeded" 29 nights in a row has tested one thing: that writing works when nothing is wrong. It has said nothing — nothing — about the morning something is.
The rescue is a chain, and every link lies to you once
So this morning we ran the rescue for real, down a management channel that does not use the dead path at all, and retrieved the database through temporary cloud object storage. The rescue worked. It also failed, informatively, at almost every link first — and those failures are the actual curriculum:
A dispatch command created in the wrong account compartment was accepted and then sat there, unclaimed, expiring in silence. Accepted is not executed; a rescue step that acknowledges you and does nothing is worse than an honest error. The dispatch API, it turned out, wanted a plain-text script where I had assumed an encoded one — an assumption that produced a perfectly delivered command that could not run. The execution account on the machine could not read the deployment directory, so the dump had to go through the application container itself, using the database's own backup API: a consistent dump of a live database, taken from inside, rather than a file copy of a database that is still writing. And a file that size was never coming back through the broken path, so it left through a temporary storage bucket with a single-object, write-only, time-limited access grant — created for this one rescue, and deleted the moment it was used.
None of those lessons are in the backup tool's documentation. All of them were waiting inside the rescue I had never run.
Retrieved is not verified, and rescued is not finished
The dump that arrived was 35,307,039 bytes compressed, 83,116,032 bytes decompressed — matching the source database exactly. Its checksum matched on both sides of the journey. The archive passed its integrity test, and the database inside it passed its own: integrity check clean, all 20 tables present. Until those checks ran, I did not have a backup. I had a file with a promising name.
Then the unglamorous ending that most rescue stories skip: the temporary bucket, the access grant, the uploaded object, and the staging files on the machine were all deleted. A rescue path that leaves standing access behind has not restored your safety; it has traded one risk for another. And one honest footnote, recorded the way I require everything in my operation to be recorded: the backup job's own failure counter will not clear until tomorrow's scheduled run succeeds on its own. That is a lagging counter, not a data gap — and confusing the two, in either direction, is how teams talk themselves into both panic and complacency.
The rescue also told me something the job never had: the database has grown to 83 megabytes, most of it from 2,267 memory records imported yesterday, on a disk that is now 85 percent full. Backups rehearse size. Rescues reveal it.
Backup is a verb you practice on the bad path
Here is the claim I will defend: a backup is not an artifact, a schedule, or a green checkmark. It is a retrieval you have actually performed, under failure conditions, with verification at the end and cleanup after it. Everything else is a hope with a cron schedule.
So run the rescue before the disaster does it for you. Break the primary path on purpose, pull the data back down the ugly road, check the hashes, and then tear down every temporary door you opened. Your backup job will keep saying success. Make sure the rescue agrees.
— Jiahui Miao, founder of vertciti, building a personal AI operating system for one.
Top comments (1)
Back in my tech support days, just after covid hit, I actually had a nasty wakeup call for backups...
My dad had a bookkeeping firm, he closed his office in 1 town, moved the server, etc. to the other town, so everything can be combined. In transit, 1 of the hdd's in the raid 5 setup got a cracked pin, so raid borked. What's worse, is that the raid controller was without power long enough that it forgot the raid config. So use the external hdd backup? A WD NAS, which uses a custom board, the enclosure died in transit... So I have a broken raid and an unusable external hdd... Had to experiment with various software raids to find the right configuration (which takes a while on 12tb of data), eventually got it, reprogrammed the raid controller, detected the raid and recovered it, then moved all the data over to the other drive, so the data can be compacted into the existing 2nd office's data, so I can dedup the data. Cuz half the data co-existed across both offices due to how they had handled work distribution.
That's why from there on, I set up a 3 phase backup plan. You have your file history for immediate recovery of draft data, then you have the external drive that's always live and connected, for on-site hot-recovery, then you have an external drive for cold backups weekly. It's customer financial data and regulatory bodies prevent cloud-based solutions for backing it up, so everything had to stay local. Since then, there's been a few drive failures, a few corrupted files, accidental deletes, etc. And I've been able to recover every single incident. So I'd add, a single backup isnt a backup, it's just unproven redundancy, to be secure, it needs multiple backup locations, methods and media types, otherwise 1 kind of issue can affect everything.