Incident report, 02:17.
Service status: unavailable.
Backup status: somebody remembers setting something up.
Recovery estimate: depends how quickl...
For further actions, you may consider blocking this person and/or reporting abuse
The incident report framing at the top is painfully relatable — "somebody remembers setting something up" is how most small teams actually experience backups until the day they need one. Starting there made the rest of the design land harder.
Two details stood out for me. First, treating completed snapshots as read-only when hard-linking is enabled — that is the kind of footnote that saves someone a genuinely awful afternoon. Second, the distinction between the scheduler knowing "assigned" vs "in progress" vs "finished," plus a failed run not advancing the last-success timestamp, is the real difference between monitoring and a checklist.
The closing line about the manifest is the thesis, really: checksums tell you the file is intact, only a restore tells you it works. Do you run scheduled restore drills against these snapshots, or is the restore still a manual "close the ticket" ritual?
Thanks for the thoughtful read! Scheduled restore drills aren’t implemented in RsyncBoy yet; restore verification is still an operator step. The scheduler tracks backup completion, but that doesn’t establish recoverability.
A useful extension would be restoring into an isolated directory or disposable database, checking the recovered data, and recording a separate restore result and last-success timestamp. A successful backup shouldn’t conceal a failed restore drill.
You’ve identified the next gap to close. The manifest helps establish what was captured; the restore demonstrates what we can recover.
Restore drills would honestly be the killer feature — a backup tool that actually verifies its own restores is the one I'd trust.
Give me about a week for a first iteration. I’m starting with scheduled filesystem restore drills: restore into an isolated directory, verify the restored files, and report the drill’s own last-success timestamp.
Keep an eye out—and if you haven’t seen an update in a week, pester me. Hold me to it!
Following up as promised: the first iteration of RsyncBoy restore drills is now implemented!
It supports scheduled filesystem drills that restore into an isolated scratch directory, verify against a SHA-256 inventory, and save a separate restore-success record.
Our first live drill verified 1,500 files, then cleaned up its scratch directory. The report explicitly states that application recovery remains unverified.
Your suggestion helped move this forward. Thanks for pushing for it—and you won’t need to pester me for this first iteration!
Shipping the first iteration of restore drills in a week — restoring into an isolated scratch directory, and having the first live drill verify 1,500 files before cleaning up, is seriously impressive follow-through.
The line that "I thought we had backups" is not a recovery plan is the right framing. Starting a process and finishing a recoverable snapshot are different states, and many small teams only track the first.
What I like in the RsyncBoy design is the insistence on dated destinations, checksums or manifests, and a last-success record someone else can read on the next shift. That is the difference between folklore and an operable system.
When we hand off production work, the acceptance check is a restore, not a green cron line. Can a different engineer bring yesterday's data back without calling the person who set it up? If the answer is no, the backup job is still a draft. Making that question part of the original problem statement prevents the 02:17 incident report that opens this post.
That’s the handoff test I want RsyncBoy to pass: can someone who didn’t configure it restore the data and verify the result without calling me?
I’m aiming to have a first version of scheduled filesystem restore drills next week, with an isolated restore destination, file verification, and a separate last-success record for the drill.
Clear restore instructions belong alongside that evidence. Thanks for highlighting the next engineer’s experience—that makes the acceptance criteria much more concrete.
Câu chuyện này nghe quen quá. Năm ngoái team mình cũng từng "có backup" — cho đến khi cần restore thì phát hiện file dump bị corrupt từ 3 tháng trước vì cron job fail lặng lẽ mà không ai check log.
Bài học đắt giá: backup chưa test restore = không có backup. Mình giờ bắt buộc chạy fire drill hàng quý: restore vào staging, diff schema, chạy smoke test. Còn thêm alert nếu backup size giảm bất thường >10% so với trung bình 7 ngày (dấu hiệu dump rỗng hoặc partial).
Đặc biệt với DB lớn, cần lưu ý
pg_dump/mysqldumpsingle-thread có thể chạy giờ đồng hồ — đã từng downtime 6h chỉ để dump xong. Chuyển sang snapshot ở filesystem level (ZFS/LVM) + WAL archiving thì RTO giảm từ giờ xuống phút.Ai đang dùng managed DB (RDS, Cloud SQL) cũng đừng chủ quan: point-in-time recovery có hạn chế retention, và cross-region replica không tự động promote khi region chính chết. Phải document runbook rõ ràng: ai bấm nút, sequence nào, contact tree ra sao (site: labagent .tech)