Your Backups Have Been Silently Corrupt Longer Than Your Retention Window

The problem

Corruption gets discovered on read, almost always. A report pulls a file, the numbers look wrong, someone finally opens the raw bytes and finds the sector is garbage. Fine — that's what checksums are for. You confirm the file is bad, you go pull it from backup, and you breathe out.

Then the restored copy has the exact same corruption.

So does the one from three days before that. So does the one from three weeks before that. You don't have a "restore from backup" problem anymore. You have a "how far back do I have to go to find a good copy, and do I still have one" problem — and the honest answer, more often than teams expect, is no.

Why it happens

Almost every system checksums on read: verify the bytes when something asks for them, so you at least know not to trust a bad file. Far fewer systems checksum on write: verify the bytes the moment they're stored, so corruption gets caught within seconds of happening instead of whenever someone next happens to need that exact object.

That gap between "corruption occurs" and "corruption is detected" is the dwell time. Nothing about a backup process shortens it. A backup job doesn't know what "correct" looks like — it just faithfully, efficiently copies whatever bytes are sitting there, intact or not. If the source was already corrupted when last night's backup ran, you now have a corrupted backup, indistinguishable from a good one, sitting in your retention window right next to a dozen other backups with the identical problem.

The failure only becomes visible once dwell time exceeds retention window. A firmware bug flips bits in a file on day one. Nobody reads that specific file again until day forty-five, for a quarterly report. If your retention is thirty days, every single backup you're holding — all thirty of them — postdates the corruption. There is no "go back further" option, because further back doesn't exist anymore. The backup succeeded every night. The system was never lying to you about that. It just was never checking the thing you actually needed it to check.

What to do about it

1. Checksum at write time, not just read time. Compute and store a checksum (or use a filesystem/object store that does this natively — ZFS, Btrfs, S3's built-in integrity checks) the moment data lands. Verify on every read after that, and alert immediately on a mismatch — don't wait for someone to notice the number looks off.

2. Run scheduled integrity scans, not just on-demand checks. A read-triggered checksum only catches corruption in data someone actually reads. Cold data — old logs, archives, rarely-touched records — can sit corrupted indefinitely with nothing ever tripping the check. A periodic full-corpus scrub (ZFS scrub, or a scheduled job that reads and re-verifies every object) forces detection on a schedule you control, instead of waiting on read patterns you don't.

3. Size your scrub interval against your retention window, deliberately, as a stated relationship — not two settings that happen to coexist. If backups are kept 30 days, your worst-case detection time — the longest possible gap before a scrub or a read would catch a given piece of corruption — needs to be meaningfully shorter than 30 days. If it isn't, you don't actually have 30 days of real recovery coverage; you have some smaller, undefined number that depends on when corruption happens to occur relative to your scan cycle.

4. Keep at least one backup generation older than your detection SLA, treated as untouchable. Even with fast detection, you want a fallback that predates your worst plausible dwell time — a monthly or quarterly snapshot held outside the normal rotation, specifically so "go back further" is still a real option the day you need it.

Key takeaways

• Checksums that only run on read tell you a file is bad; they don't limit how long it's been bad, and that gap is what actually determines whether backups can save you.
• A backup process has no concept of "correct" — it copies whatever bytes exist, so corruption propagates through every generation made after it occurred.
• The real risk metric isn't "do we have backups," it's "is our worst-case corruption dwell time shorter than our retention window" — most teams have never measured this.
• Write-time checksums plus scheduled full-corpus scrubbing are what actually bound dwell time; read-triggered checks alone leave cold data unprotected indefinitely.
• Keep one backup generation deliberately older than your detection SLA as an untouchable fallback.

I wrote about the mechanics of catching corruption in the first place — checksum algorithms, where to place the check, what it costs — on Medium: https://medium.com/@speed_enginner/checksum-everything-corruption-caught-before-catastrophe-5cace12122fa