DEV Community

Cover image for The 3-2-1 rule, explained by someone who has actually restored at 3am
Mario Fang
Mario Fang

Posted on Originally published at mariosfong.hashnode.dev

The 3-2-1 rule, explained by someone who has actually restored at 3am

The pager went off just after 3am, and half an hour later I was sitting on a cold floor with a laptop, a failing server, and a decision to make about which copy of the data I trusted. That night taught me more about 3-2-1 than any diagram ever did — mostly because two of my three copies turned out to be less useful than the label suggested.

Three copies, two media, one offsite. Everyone quotes it. Almost nobody has tested it under pressure, which is where the rule stops being a rule and becomes a series of uncomfortable surprises.

The copy that saved the night was the boring one

It was a plain external drive, disconnected, sitting in a cabinet, last written to the previous evening. No agent, no console, no dashboard. It was also the only copy that was both complete and untouched.

Meanwhile, the "live" replica was corrupted within minutes of the failure — whatever took the primary took it too, because it was reachable from the same place with the same credentials. And the clever snapshot scheme was on the same storage array, so the snapshots went with the array. Copy count said three. Usable copies said one.

The lesson isn't "replicas are bad". It's that a copy which can be reached by the same mistake isn't a second copy. It's the same copy with a delay.

An external drive in a cabinet

An offsite copy is a promise your uplink has to keep

My offsite copy was real. It was also much larger than the link it had to come down, and the part I actually needed that night was not the small part.

Restoring over the wire would have taken until well past the point where anyone else was awake and waiting. So the offsite copy did what it does best in a real emergency: it confirmed the data existed, and then it was too slow to use. We ended up restoring from the local disconnected drive and using the offsite copy only to verify that the files we pulled were the right ones.

If your offsite copy can only be restored over the internet, measure the restore time before you need it. Nobody has ever been rescued by a copy that takes three days to arrive.

Two copies in one building is one copy with extra steps

Same room, same power, same switch, same domain, usually the same admin account. Fire, flood, a bad firmware push, a stolen credential — pick any of them and both copies take the hit together.

The "two media" part of 3-2-1 is often satisfied on paper by a NAS and a USB drive that both live in the same rack. That's media diversity, not resilience. It buys you a disk failure. It doesn't buy you a bad night.

A backup you have never restored is a hypothesis

This is the one that hurt. My nightly job reported success for months. What it didn't report was that a whole branch of the directory tree had been silently skipped since a permissions change — the job exited clean, the log said "completed", and the files simply weren't in the archive.

Nobody runs a restore to a scratch directory on a schedule because it's boring and it eats time. So the failure sits there quietly until the night you actually need the files, and then you find out that your backup software has been lying to you in good faith.

A restore test doesn't have to be elaborate. Pick a random folder, restore it somewhere harmless, open three files, delete the test. Fifteen minutes a month would have caught mine.

If I'd known that at the start, I'd have spent those fifteen minutes a month from day one instead of discovering the gap at the worst possible moment. That's the whole "I wish someone had told me" part of this: the backup job is not the proof. The restore is the proof, and it's the only part nobody schedules.

What I do differently now

The short version: I stopped counting copies and started counting what survives a single bad decision. Three things, in the order I'd do them again:

  1. Give at least one copy a real gap. Unplug it, or make the destination unreachable once the job finishes. Distance beats redundancy — one copy nobody can reach is worth more than three that share a credential.
  2. Write down the restore time, not just the backup time. If the number is in days, you don't have a recovery plan, you have an archive. Restore something real and time it.
  3. Put the restore test on a calendar. Not on instinct, not "when there's time". The test is the backup; the job status is just a hopeful log line.

The 3-2-1 rule is still the right shape, and I still use it. But it's a floor, not a finish line — and it only means anything if you know which of the three copies actually gets you through the night. That night, mine was the one in the cabinet.

— Mario

I write about backups, self-hosting and keeping systems alive. I also build RansomShield (getransomshield.com), the air-gapped backup tool described above.

Top comments (0)