DEV Community

speed engineer
speed engineer

Posted on

RAID Doesn't Protect You From the Corruption That Actually Gets You

#storage #performance #backend #architecture

The problem

A client's RAID 6 array did exactly what it was designed to do for eighteen months straight: survive drive failures without losing data. Meanwhile, bit rot quietly ate through their primary research database the entire time. By the time the application started throwing errors, 23% of a 2.3TB genomics dataset was corrupted — and RAID had faithfully mirrored every corrupted byte across every drive, because that's all RAID actually promises to do.

Recovery took 72 hours and cost six figures in emergency consulting and reconstruction. The failure wasn't a bad disk. It was a wrong mental model.

Why it happens

RAID protects against hardware failure: a drive dies, redundancy rebuilds it. It says nothing about data corruption that happens before a write ever reaches the disk — a bit flipped by a cosmic ray in RAM, an L2 cache glitch during computation, a firmware bug in the storage controller, a kernel bug during I/O. None of that trips RAID's failure detection, because from RAID's point of view nothing failed. The corrupted bytes get written, mirrored, and backed up with the same fidelity as good data.

This is exactly where traditional filesystems make it worse. EXT4, NTFS, and XFS checksum their own metadata — the bookkeeping that tracks where files live — but not file contents. So cp, rsync, and tar will silently propagate corrupted data through your entire backup chain, and every copy will look identical to a checksum that was never actually checking the payload.

What to do about it

The fix is checksumming at the layer where corruption can't hide: end-to-end, verified on every read, not just on write.

Filesystems built for this — ZFS, Btrfs — store a hash alongside every data block and verify it on access, not just during scheduled scrubs:

$ zfs set checksum=sha256 tank
$ zpool status -v tank
  ada0  ONLINE  0 0 3   # 3 checksum errors detected and corrected
  ada3  ONLINE  0 0 1   # 1 checksum error detected and corrected
Enter fullscreen mode Exit fullscreen mode

If you can't move the whole stack to ZFS/Btrfs, application-level checksums are portable and let you choose the right trade-off for your access pattern. The trade-off is bigger than people assume — on a 1GB file:

Algorithm Time Notes
SHA-256 3.7s Cryptographically secure, slowest
SHA-1 2.3s Faster, weaker guarantees
CRC32 (hardware) 0.8s Hardware-accelerated via dedicated CPU instructions
xxHash64 0.2s ~18x faster than SHA-256, non-cryptographic

That's not a rounding error at scale — it's the difference between checksumming being "too expensive to do everywhere" and "basically free." Modern CPUs expose a CRC32C instruction directly:

uint32_t hardware_crc32c(const void* data, size_t length) {
    const uint8_t* buffer = (const uint8_t*)data;
    uint32_t crc = 0xFFFFFFFF;
    while (length >= 8) {
        crc = _mm_crc32_u64(crc, *(const uint64_t*)buffer);
        buffer += 8; length -= 8;
    }
    return crc ^ 0xFFFFFFFF;
}
Enter fullscreen mode Exit fullscreen mode

Pick your algorithm by what you're protecting: xxHash64 or hardware CRC32C for hot-path integrity checks where speed matters more than cryptographic guarantees; SHA-256 for anything where an attacker (not just cosmic rays) might want to forge a matching checksum — financial records, medical data, anything audited.

The part people skip: verify on read, not just on write. A checksum you only calculate once, at write time, protects against nothing — corruption happens after the write, sitting on disk or in transit. The value is in comparing stored-vs-current at every access.

Key takeaways

  • RAID protects against drive failure, not data corruption — they are different problems with different defenses.
  • Traditional filesystems (EXT4/NTFS/XFS) checksum metadata, not your actual data. Corruption in file contents propagates silently through every backup.
  • Checksum algorithm choice is a real trade-off, not a formality — hardware-accelerated CRC32C and xxHash64 are ~5-18x faster than SHA-256 on the same data.
  • A checksum only protects you if you verify it on every read, not just calculate it once on write.

A longer version of this piece, with the full financial-impact breakdown, first ran on Medium.

Top comments (0)