Your backup is not a backup until you have restored it

This is an English write-up of a post from my Japanese dev diary. Original: https://saas-diary.com/tech-log/backup-restore-drill-automation/

For over a year, my backup job has reported success every single night. Green check, every day, no exceptions.

Then I asked myself one question and went cold:

"How many times have I actually restored from it?"

Zero. Not once.

"It was backed up" and "it can be restored" are different states

My setup has two paths. One mirrors all source to a private repo. The other packs the things I can never recreate — notes, config, and Android signing keys — into an encrypted bundle and ships it to a private channel every night.

Both were green every day. But green only proved the upload finished. It never proved the contents were right, or that the archive could even be opened.

Within one month, I had two failures that stayed green the whole time.

Failure 1. The collector for signing keys used three hardcoded paths. I kept shipping new apps, so the number of keys kept growing — but the collector didn't. By the time I noticed, 7 of 10 keys were missing from the backup. Five of those apps were live on the store. If my machine had died, I could never have shipped an update for them again. The backup reported success every night through all of it.

Failure 2. The mirror push failed 7 days in a row (a large binary hit the host's file-size limit). But the script printed "✅ done" and returned exit code 0 even when one half failed. A failure that isn't visible isn't a failure — it's a time bomb.

So I automated a restore drill

Once a month, a job now does this:

1. Rebuild the encrypted bundle (without shipping it)
2. Actually decrypt it with the stored passphrase
3. Extract it and count what's inside
4. Check the mirror is not stalled (latest commit timestamp via API)
5. Delete the scratch folder and the generated bundle

The encryption is openssl-compatible AES-256-CBC with PBKDF2 (SHA-256, 100k iterations). I deliberately avoided depending on the openssl binary, because the moment you need a restore is the moment you're on a fresh machine with nothing installed. Node's stdlib is enough:

import crypto from 'node:crypto';

// openssl format: "Salted__" + salt(8) + ciphertext
function opensslDecrypt(buf, password) {
  if (buf.slice(0, 8).toString('ascii') !== 'Salted__') {
    throw new Error('not an openssl-compatible file');
  }
  const salt = buf.slice(8, 16);
  const keyiv = crypto.pbkdf2Sync(Buffer.from(password, 'utf-8'), salt, 100000, 48, 'sha256');
  const d = crypto.createDecipheriv('aes-256-cbc', keyiv.slice(0, 32), keyiv.slice(32, 48));
  return Buffer.concat([d.update(buf.slice(16)), d.final()]); // wrong key -> throws here
}

Decrypting isn't enough — define what "complete" means

An empty zip decrypts just fine. So I wrote the pass criteria as numbers:

export function judgeBundle(inv) {
  const ng = [];
  if (inv.noteFiles < 50) ng.push('too few notes (collector probably missed a folder)');
  if (!inv.hasIndex) ng.push('index file missing');
  if (!inv.hasSettings) ng.push('settings missing');
  if (inv.hasSettings && !inv.settingsValid) ng.push('settings is not valid JSON');
  if (!inv.hasRecovery) ng.push('recovery instructions missing');
  // regression check for the incident above:
  if (inv.keystores < inv.publishedApps) ng.push('fewer signing keys than published apps');
  return ng;
}

That last line is failure #1, encoded. Turn every incident into a condition that goes red. A written post-mortem doesn't stop a repeat; a failing check does.

Then test the test

This is the part I'd skip if I were in a hurry, and it's the part that mattered most. A check that always returns green is worse than no check — it hands out confidence while looking at nothing.

So I mutated the judge function on purpose and confirmed each mutation turned it red: remove a threshold, invert the ordering, make it always return an empty list. I also added a control pair for the decryption itself:

let ok1 = false, ok2 = false;
try { ok1 = opensslDecrypt(enc, 'right-pass').equals(sample); } catch {}
try { opensslDecrypt(enc, 'wrong-pass'); } catch { ok2 = true; }  // failing is the pass condition

Without the second one, "it decrypted" carries no information — the code might open with anything.

First run, and a Windows trap

The first drill reported: 495 note files, 18 signing keys, settings parses as JSON, recovery doc present, 519 files total at 2.43 MB, mirror committed the same day. That's the first moment I could honestly say I had a backup.

One trap worth sharing. Calling process.exit() right after fetch on Windows can abort in libuv while sockets are still closing:

Assertion failed: !(handle->flags & UV_HANDLE_CLOSING)

The logic had completed successfully, but the process returned exit code 127, so the scheduler recorded a failure. For a monitoring job, "reports failure when it actually succeeded" is about the worst behaviour possible. Fix: stop calling process.exit(); set process.exitCode and let the runtime drain.

main: {
  if (nothingToDo) break main;        // no process.exit()
  // ...
  process.exitCode = failed ? 1 : 0;  // let it end naturally
}

Takeaways

• Exit code 0 doesn't prove the work happened. It proves the process started and ended.
• A backup you have never restored is a file you hope is a backup.
• Encode incidents as conditions. Reflection doesn't survive; a red check does.
• Break your checks on purpose and watch them fail. A check that has never gone red has never told you anything.

I also keep the recovery instructions inside the encrypted bundle. Obvious in hindsight: a runbook stored only on the machine that died is not a runbook.