DEV Community

Syed Masood Shah
Syed Masood Shah

Posted on

I stopped trusting automatic backups and started watching them like the operations problem they are

About two years ago I lost a week of client data because a backup job had been silently failing for six days. The log lived on a server nobody checked. When I finally dug in, the disk was full, retention had nuked the latest snapshots, and "automatic backup" turned out to mean "backup that runs, fails, and keeps it to itself."

That's the thing nobody tells you about backups. They're not a set-it-and-forget-it thing. They're a monitoring problem with extra steps. A job that doesn't run, one that fails halfway, one that quietly eats its own retention — from the outside they all look identical, which is to say: totally fine, right up until they're not.

So this year I rebuilt mine properly. The change that mattered most wasn't the backup engine at all. It was the dashboard.

I run restic for the snapshots — boring, reliable, encrypted, cheap. It backs up the handful of machines and databases that actually matter to the work I do, to a couple of local drives plus one offsite target. The usual routine. But the part I'd been ignoring was the "did it actually happen" question.

Now every restic run posts a tiny result — repo, snapshot count, bytes, last successful run, whether the integrity check passed — to a small dashboard I can glance at without digging through logs. One screen. Green block means it ran. Red block means my morning starts with a mystery. Because it's all self-hosted Docker, the whole thing sits on the same box as everything else. No cloud account, no monthly charge, no third party holding the keys to my restore point.

The rules I landed on after a year of running it, in plain terms:

  • Run a restore drill once a month, not just a backup. A snapshot that won't restore is a folder of dead weight. I restore a test directory to a scratch volume and check it opens. Takes five minutes.
  • Alert on "didn't run" as loudly as "failed." A job that never started is the silent failure that costs you.
  • Check your retention math at least once. The disk-full cascade that bit me two years ago was a retention config, not a disk problem.
  • Keep the recovery path human-readable. If things burn down and I'm the one holding the tape, the dashboard should tell the next person in two glances what's healthy and where the restore goes.

Full disclosure: this is exactly the kind of thing small businesses skip because "the cloud backs up our email" and nobody wants to argue with that. It costs almost nothing to do properly, and it's the one piece of "security" that actually matters when everything else goes sideways.

If you want the whole thing without re-architecting it yourself — restic plus the audit dashboard as a self-contained Docker stack you can stand up in an afternoon — I put together a pack for exactly this. You can grab it here: https://symshah.gumroad.com/l/smb-security-pack

Top comments (0)