My Raspberry Pi's SD Card Died at 2 AM. My "Automatic" Backups Turned Out to Be a 47-Day-Old Lie.

The alert came in at 2:14 AM: three of my automation agents had missed their heartbeat window. By the time I SSHed in from my phone, I already knew — the Pi that runs my entire one-person business stack had stopped responding mid-write. When I finally pulled the power and booted from a fresh card, fsck gave me the verdict: the filesystem was corrupt beyond repair.

No problem, I thought. I have nightly backups. I've had them since day one.

The backup ran every night at 3 AM. It logged success every single night. And when I opened the backup directory, the newest snapshot was 47 days old.

This is the post-mortem of how a backup system that "worked" for a month and a half was actually broken from roughly week one — and the handful of cheap, boring fixes that made sure it can never lie to me again.

What the stack was

For context, this Pi runs the unglamorous core of my setup:

• Two AI agents (support triage and a content scheduler) running as systemd services
• A small Python job runner with cron-triggered tasks
• An SQLite database that holds customer state, agent memory, and job history
• A reverse proxy in front of two internal web apps

Nothing exotic. Total cost: a $60 Pi, one SD card, and the assumption that "I set up backups" means "I have backups."

The backup that wasn't

My backup was a shell script in /etc/cron.daily. Roughly this:

#!/bin/bash
rsync -a --delete /home/sean/agents/ /mnt/usb/backup/agents/
sqlite3 /home/sean/agents/state.db ".backup /mnt/usb/backup/state.db"
echo "$(date): backup OK" >> /var/log/backup.log

It looked correct. It even logged success. Three separate bugs were stacked on top of each other:

Bug 1: The USB drive had silently unmounted. About 47 days before the crash, the drive dropped off the bus after a brief power dip. The mount point /mnt/usb still existed — as an empty directory on the SD card. rsync happily wrote into it, consuming SD card space instead of backing anything up.

Bug 2: The log line always said OK. The echo ran unconditionally. I never checked $? on the rsync or the sqlite backup. "backup OK" was printed whether or not anything had been copied. I built a system whose only job was to tell me the truth, and I wrote it so it could only tell me one thing.

Bug 3: Nobody read the log anyway. Even a correct log line goes nowhere if the only failure mode is "human remembers to check." I had alerting on agent heartbeats but zero alerting on the thing whose entire purpose was disaster recovery.

The irony: I monitor my AI agents obsessively. I got paged when an agent missed a heartbeat by six minutes. But the backup script — the thing standing between me and total data loss — could fail silently for seven straight weeks and I'd never know.

The fix: backups must prove themselves

I rebuilt the backup around one principle: a backup that doesn't verify itself is a rumor, not a backup. Here's the current version, trimmed:

#!/bin/bash
set -euo pipefail

DEST=/mnt/usb/backup
STAMP=$(date +%F)
FAIL=0

# 1. Prove the destination is real storage, not an empty mountpoint
if ! mountpoint -q /mnt/usb; then
  notify "BACKUP FAIL: /mnt/usb not mounted"
  exit 1
fi

# 2. Prove there's space
AVAIL=$(df --output=avail -BM /mnt/usb | tail -1 | tr -dc '0-9')
if [ "$AVAIL" -lt 2000 ]; then
  notify "BACKUP FAIL: only ${AVAIL}MB free"
  exit 1
fi

# 3. Copy, and capture the real exit status
rsync -a --delete /home/sean/agents/ "$DEST/agents/$STAMP/" || FAIL=1
sqlite3 /home/sean/agents/state.db ".backup $DEST/db/$STAMP.db" || FAIL=1

# 4. Verify the DB is a readable SQLite file with plausible content
if ! sqlite3 "$DEST/db/$STAMP.db" "PRAGMA integrity_check;" | grep -q "^ok$"; then
  notify "BACKUP FAIL: integrity check on $STAMP.db"
  exit 1
fi

# 5. Only NOW claim success — and post the proof
if [ "$FAIL" -eq 0 ]; then
  SIZE=$(du -sh "$DEST/agents/$STAMP" | cut -f1)
  CHECKSUM=$(sha256sum "$DEST/db/$STAMP.db" | cut -c1-12)
  notify "backup OK $STAMP size=$SIZE db_sha=$CHECKSUM"
else
  notify "BACKUP PARTIAL FAIL on $STAMP"
  exit 1
fi

Key changes:

1. set -euo pipefail — the script dies on the first real error instead of strolling to the success line.
2. mountpoint -q — explicitly proves the destination is mounted storage. This one check would have caught all 47 days of failure.
3. Integrity check on the copied database — PRAGMA integrity_check catches truncated or corrupt copies, not just missing ones.
4. A positive heartbeat, not just negative alerts. Every successful run pushes a message (I use a tiny webhook to my phone) containing the date, size, and a checksum prefix. If I don't see that message for 36 hours, something is wrong even if no error ever fired. Absence of bad news is not news; I made the backup generate actual news.

The part I skipped the first time: restore drills

Here's the second failure, and it's more embarrassing than the first.

Two weeks after rebuilding the script, I decided to test a restore. The backup was fresh, verified, checksummed, beautiful. And the restore failed — because the backup included the running SQLite database files copied with plain rsync in an older snapshot, and more importantly, my agents' config referenced absolute paths and an environment file that lived outside the backed-up directory. I had a complete copy of data and no copy of the thing that makes the data usable.

The fix was a RESTORE.md in the repo — a literal runbook:

1. Flash fresh Raspberry Pi OS Lite
2. apt install the pinned package list (I now generate packages.txt nightly into the backup)
3. Copy .env from the encrypted secrets folder (which itself was in neither backup — it's now backed up separately with age encryption)
4. Restore the DB from the newest integrity_check-passing snapshot
5. Run systemctl start on the units, verify heartbeats within 10 minutes

Then I did the thing I should have done from day one: I ran the drill. Booted a spare Pi from scratch, followed my own document, hit two outdated steps, fixed them, and got to a fully verified restore in 22 minutes. Now the drill runs the first Sunday of every month, and the timer is in cron like everything else.

A backup's real metric isn't "did the copy succeed." It's time-to-restored-service. Mine went from "unknown/infinite" to a tested 22 minutes.

What I'd tell myself six months ago

• Silence is a failure mode. Any safety system that only speaks when asked is a system you'll forget to ask. Make it send a heartbeat with content — size, checksum, timestamp — so a missing or malformed heartbeat is detectable.
• Verify the destination before writing to it. One mountpoint -q line would have saved 47 days of false confidence.
• Untested restores don't exist. The restore drill found two gaps the backup script structurally could not find. Run the drill before you need it, not after.
• The unmonitored thing will be the thing that kills you. I had dashboards for agents, alerts for API spend, and heartbeats for cron jobs. The backup ran outside all of it, because it "just worked." That phrase should be a lint error in every ops setup.
• Back up the environment, not just the data. DBs and files are half a system. Pinned package lists, env/secrets (encrypted), and a written restore runbook are the other half.

Total cost of the rebuild: one $12 USB drive kept in a different room, about 90 minutes of work, and one monthly 20-minute drill. The SD card that died cost me an evening — instead of the business.

The deeper lesson, and the one that applies way beyond Raspberry Pis: every automation you run — agents included — needs the same treatment. Prove it ran, prove the output is valid, and rehearse the recovery while nothing is on fire.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.