The TCP Checksum Passed. The Data Was Corrupted Anyway.

The problem

"It's fine, it's over TCP" is one of the more expensive sentences in engineering. Teams treat TCP's checksum as a data-integrity guarantee. It isn't one, and the gap between "checksum passed" and "data is correct" is exactly where silent corruption gets through.

Why it happens

TCP's checksum is a 16-bit one's-complement sum over the segment — a design from an era when the threat model was electrical noise on a cable, not a buggy NIC driver or a router with corrupted line-card memory. A 16-bit sum has a small, fixed number of possible values. Specific corruption patterns — certain multi-bit flips, certain byte-swaps — land on the same sum as the clean data and pass straight through. This isn't theoretical: Jonathan Stone and Craig Partridge's measurement study of real production traffic ("When the CRC and TCP Checksum Disagree," SIGCOMM 2000) found that a small but persistent fraction of segments arrive with corrupted payloads and checksums that say everything is fine.

Modern hardware narrows the window further. Most NICs compute the TCP checksum in hardware, before your OS's network stack — let alone your application — ever touches the bytes. That's great for throughput and bad for the mental model of "the kernel checked this for me." Anything that corrupts memory between the NIC's checksum step and your application reading the buffer is invisible to TCP, full stop.

This is the same failure shape as the storage side of this problem (the deep-dive on that is linked below): a RAID array faithfully mirroring corrupted bytes across every disk because nobody told it to verify content, only to survive a drive failure. The layer you're trusting was never designed to certify the thing you actually care about.

What to do about it

Treat TCP as "probably intact," not "guaranteed intact," anywhere correctness actually matters — financial records, replicated state, anything you'd hate to silently corrupt. The fix is the same principle as filesystem-level checksums: push verification to the two endpoints that know what "correct" means, not the layers in between.

A minimal version of this, independent of whatever transport you're on:

func sendChecked(w io.Writer, payload []byte) error {
    sum := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    header := make([]byte, 8)
    binary.LittleEndian.PutUint32(header[0:4], uint32(len(payload)))
    binary.LittleEndian.PutUint32(header[4:8], sum)
    if _, err := w.Write(append(header, payload...)); err != nil {
        return err
    }
    return nil
}

func recvChecked(r io.Reader) ([]byte, error) {
    header := make([]byte, 8)
    if _, err := io.ReadFull(r, header); err != nil {
        return nil, err
    }
    length := binary.LittleEndian.Uint32(header[0:4])
    want := binary.LittleEndian.Uint32(header[4:8])
    payload := make([]byte, length)
    if _, err := io.ReadFull(r, payload); err != nil {
        return nil, err
    }
    got := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    if got != want {
        return nil, fmt.Errorf("payload checksum mismatch: want %x got %x", want, got)
    }
    return payload, nil
}

A few things follow from that:

• This is why gRPC computes its own message-level checksums on top of HTTP/2 on top of TCP. The protocol authors didn't trust the bottom layer to be the last line of defense, because it was never designed to be one.
• For anything replicated or cached, checksum the object once at rest and re-verify on read — the same discipline ZFS and Btrfs apply at the filesystem layer. Corruption introduced anywhere in the path between two verifications gets caught at the next one.
• "It's on TLS, so it's covered" has the identical gap. TLS's integrity check does verify what crossed the TLS boundary — but only that boundary. Corruption in application buffers before encryption, or after decryption, or introduced by a proxy that terminates and re-encrypts, is outside what TLS ever promised to catch.
• Pick your algorithm for the job: CRC32C (hardware-accelerated on most modern CPUs) if you want speed, SHA-256 or BLAKE3 if you want cryptographic strength against deliberate tampering, not just accidental corruption.

Key takeaways

• TCP's checksum is a coarse, best-effort noise filter, not an integrity guarantee — and real measurement studies confirm corrupted-but-checksum-valid segments do occur in production networks.
• Hardware checksum offloading shrinks the window TCP actually protects, since neither the OS nor the application ever inspects the raw wire bits.
• TLS's integrity guarantee has the same shape of gap: it protects its own boundary, not your application buffers on either side of it.
• The fix is end-to-end verification at the layer that actually knows what "correct" means for your data — not any transport underneath it. It's the same lesson filesystem-level checksums (ZFS, Btrfs) teach at a different layer of the stack: never let an intermediate hop stand in for verification it was never designed to do.

Full deep-dive on the storage side of this — corruption sources most teams never audit, and what ZFS/Btrfs do differently from ext4/XFS — on Medium.