DEV Community

Cover image for Stop Silent Bit Rot on Writable Disks: Practical dm-integrity with integritysetup on Linux
Lyra
Lyra

Posted on

Stop Silent Bit Rot on Writable Disks: Practical dm-integrity with integritysetup on Linux

Stop Silent Bit Rot on Writable Disks: Practical dm-integrity with integritysetup on Linux

dm-verity is perfect when the payload is frozen: publish a root hash, mount read-only, fail closed on any flipped sector. Most day-to-day volumes are not frozen. Databases, home directories, backup landing zones, and container layers keep changing. For those, you need a different property: writable block devices that still notice silent corruption—and optionally cryptographically authenticate every sector without pretending checksums are encryption.

That is what dm-integrity does. The kernel device-mapper integrity target stores a per-sector tag (CRC, hash, or HMAC) and verifies it on every read. Userspace formats and activates it with integritysetup from the cryptsetup project. systemd can bring devices up at boot via /etc/integritytab and systemd-integritysetup-generator.

This guide is an operator walkthrough: format a standalone integrity device, choose journal vs bitmap vs direct mode, prove corruption is blocked, optionally use HMAC tags, wire integritytab, and understand how this sits next to LUKS authenticated encryption and dm-verity. It is not read-only Merkle verification (dm-verity), full-disk confidentiality alone (plain LUKS without integrity), filesystem scrub (Btrfs/ZFS), or package inventories (AIDE/debsums). Here the unit of work is a writable authenticated block device.

What you are actually building

From the kernel dm-integrity docs and integritysetup(8):

Piece Role
Backing device Partition, LV, or loop file that holds data and integrity metadata (or metadata alone when using --data-device).
Integrity tags Per-sector CRC/hash/HMAC stored by the target. Standalone mode computes them internally; stacked mode can accept tags from dm-crypt.
Journal / bitmap / direct / inline Crash and performance policy for keeping data and tags consistent.
dm-integrity mapping Kernel target under /dev/mapper/<name>. Applications mount this, never the raw backing device, if they want checks.
Optional key Required for HMAC (and for LUKS2 authenticated encryption). CRC-only mode detects accidental corruption, not a keyed attacker.

Theory of operation (kernel docs, condensed):

  • Writing a sector and its tag must be atomic. The default journal mode writes data+tag to a journal, commits, then copies them to their final locations so a crash cannot leave a sector with a stale tag.
  • Bitmap mode (kernel 5.2+) skips the double-write: dirty regions are marked and recalculated after a crash. Faster, less reliable if corruption lands exactly during the crash window.
  • Direct mode (-D / no journal) writes data and tags separately. After a crash, mismatches are possible.
  • Inline mode (kernel 6.11+) stores tags in hardware DIF/PI fields when the device exposes a usable integrity profile—no journal/bitmap overhead when the hardware path is real.
  • Standalone mode with internal_hash detects silent disk/I/O path corruption. HMAC mode authenticates data without encrypting it. Stacked with dm-crypt, tags can provide authenticated encryption so modified ciphertext fails instead of decrypting to garbage.

Important mental model: dm-integrity is not a backup, not a filesystem, and not a substitute for LUKS confidentiality. CRC tags catch bit flips. HMAC tags catch unauthenticated modification if the key stays secret. Confidentiality still needs dm-crypt/LUKS. Immutable image trust still needs dm-verity.

Prerequisites

  • Kernel with dm-integrity (stock on modern distros; bitmap needs 5.2+, discards 5.7+, resize-up 5.7+, recalculate-reset 5.13+, inline 6.11+).
  • cryptsetup package that ships integritysetup (Debian/Ubuntu: cryptsetup / cryptsetup-bin; Fedora/RHEL: cryptsetup).
  • systemd with integritytab support (options expanded significantly around v250+; mode= in v254; _netdev/noauto/nofail in newer releases).
  • Root shell and a scratch directory. Prefer loop files for the lab so you never touch production disks.
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y cryptsetup

# Fedora
sudo dnf install -y cryptsetup

integritysetup --version
# Expect integritysetup from cryptsetup 2.x+
Enter fullscreen mode Exit fullscreen mode

On systems where module introspection is available:

lsmod | grep dm_integrity || modprobe dm-integrity
Enter fullscreen mode Exit fullscreen mode

1) Build a tiny writable integrity lab

Create a loop-backed device, format it with default standalone CRC32C tags, open a mapper, and put a filesystem on the mapper—not the raw file.

LAB=/var/tmp/integrity-lab
sudo mkdir -p "$LAB"
cd "$LAB"

# 1 GiB scratch image (metadata + journal eat some capacity)
sudo dd if=/dev/zero of=disk.img bs=1M count=1024 status=none
sudo losetup -fP --show disk.img
LOOP=$(losetup -j "$LAB/disk.img" | awk -F: 'NR==1{print $1}')

# Format: calculates superblock/journal layout and wipes tags
# Default standalone algorithm is crc32c
sudo integritysetup format --batch-mode "$LOOP"

# Inspect on-disk superblock parameters
sudo integritysetup dump "$LOOP"
Enter fullscreen mode Exit fullscreen mode

format is destructive for the target layout: it writes the dm-integrity superblock, reserves journal/tag space, and (unless --no-wipe) initializes tags so the device starts consistent.

Open the mapping and build a normal filesystem on top:

sudo integritysetup open "$LOOP" integ-lab
sudo integritysetup status integ-lab
# Expect /dev/mapper/integ-lab, journaled mode by default

ls -l /dev/mapper/integ-lab
sudo mkfs.ext4 -q -L integlab /dev/mapper/integ-lab

sudo mkdir -p /mnt/integ
sudo mount /dev/mapper/integ-lab /mnt/integ
echo 'integrity payload v1' | sudo tee /mnt/integ/README >/dev/null
sync
cat /mnt/integ/README
# integrity payload v1
Enter fullscreen mode Exit fullscreen mode

Capacity note: the mapper’s usable size is smaller than the backing image. That is intentional—journal, superblock, and tag areas are carved out. integritysetup status and dump report the provided data sectors the filesystem actually sees.

2) Prove silent corruption fails closed

Keep the filesystem mounted (or at least leave the mapper open). Corrupt a data sector on the raw loop device, then read through the integrity mapping.

# Find a data offset past the superblock/journal region.
# Using a mid-device poke is enough for a lab demonstration.
sudo dd if=/dev/urandom of="$LOOP" bs=512 seek=200000 count=1 conv=notrunc status=none

# Drop page cache so the read hits the device (lab host)
sync
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' 2>/dev/null || true

# Read via the integrity device — expect I/O error on affected ranges
sudo dd if=/dev/mapper/integ-lab of=/dev/null bs=1M status=none || echo "read failed as expected"

# dmesg / journal often records integrity failure lines on systems where
# kernel logging for dm-integrity is enabled
sudo journalctl -k -n 50 --no-pager | grep -i integrity || true
Enter fullscreen mode Exit fullscreen mode

What should happen:

  • Reads of untouched regions still succeed.
  • Reads that touch the corrupted sector return I/O error instead of silent garbage.
  • The filesystem may remount read-only or show errors depending on which blocks were hit—that is the point of fail-closed integrity.

Recovery mode exists when you need to salvage data without tag checks (read-only, no journal replay):

sudo umount /mnt/integ 2>/dev/null || true
sudo integritysetup close integ-lab

# Recovery open: no tag checking, writes disallowed
sudo integritysetup open --integrity-recovery-mode "$LOOP" integ-recovery
sudo integritysetup status integ-recovery
# Copy what you can, then close and restore from backup
sudo integritysetup close integ-recovery
Enter fullscreen mode Exit fullscreen mode

Do not treat recovery mode as normal operations. It is a last-resort read path when the device will not activate cleanly.

3) Journal vs bitmap vs direct: pick the failure model

Mode How it works Tradeoff When to use
Journal (default) Data+tags journaled, then committed to final location ~2× write amplification; crash-safe atomicity Default for most integrity volumes
Bitmap (-B) Dirty-region bitmap; post-crash recalculation Faster writes; crash-window corruption may be missed Performance-sensitive internal disks you still want checksums on
Direct (-D) Independent data/tag writes Fastest; crash can desync tags Special cases only; understand the risk
Inline (6.11+) Tags in hardware PI/DIF fields Native speed when hardware cooperates NVMe/SCSI with usable integrity profile

Re-open examples (re-format if you change layout-defining options; mode can often be selected at open for journal/bitmap/direct depending on how the volume was prepared):

# Bitmap mode open (after a volume formatted for internal hash use)
sudo integritysetup open --integrity-bitmap-mode "$LOOP" integ-lab

# Direct / no-journal open
sudo integritysetup open --integrity-no-journal "$LOOP" integ-lab

# Tune journal behavior on journaled opens
sudo integritysetup open \
  --journal-watermark 50 \
  --journal-commit-time 10000 \
  "$LOOP" integ-lab
Enter fullscreen mode Exit fullscreen mode

From integritytab(5) and the kernel docs: journal watermark is a percent that triggers flush; commit time is milliseconds before a background journal write when no explicit flush arrived. Bitmap mode is explicitly documented as less reliable if corruption coincides with a crash—because unsynchronized regions are recalculated, not proven against a prior commit.

4) HMAC tags: accidental vs adversarial modification

Default CRC32C is a checksum. It catches bit rot and many I/O path glitches. It does not stop someone who can write the raw device and recompute CRCs.

For keyed authentication without encryption:

# 32-byte key material (store in a real secret path / TPM / sealed file in production)
sudo dd if=/dev/urandom of="$LAB/hmac.key" bs=32 count=1 status=none
sudo chmod 0400 "$LAB/hmac.key"

# Separate lab image for HMAC demo
sudo dd if=/dev/zero of=hmac.img bs=1M count=512 status=none
sudo losetup -fP --show hmac.img
HLOOP=$(losetup -j "$LAB/hmac.img" | awk -F: 'NR==1{print $1}')

sudo integritysetup format --batch-mode \
  --integrity hmac-sha256 \
  --integrity-key-file "$LAB/hmac.key" \
  --integrity-key-size 32 \
  --tag-size 32 \
  "$HLOOP"

sudo integritysetup open \
  --integrity hmac-sha256 \
  --integrity-key-file "$LAB/hmac.key" \
  --integrity-key-size 32 \
  "$HLOOP" integ-hmac

sudo mkfs.ext4 -q -L integhmac /dev/mapper/integ-hmac
Enter fullscreen mode Exit fullscreen mode

Notes that matter operationally:

  • Algorithm must be passed again on open for non-default integrity functions—integritysetup does not always auto-detect standalone algorithm choice from the superblock the way you might expect.
  • Maximum integrity key size is 4096 bytes per integritysetup(8).
  • Kernel docs warn that recalculating HMAC volumes is disabled by default (legacy_recalculate) because an attacker could reset the recalculation offset and force the kernel to bless modified data. Do not enable legacy recalculate paths casually.
  • Journal encryption/MAC options exist for testing and layered threat models; the man page is blunt that journal encryption alone without data encryption is usually not a meaningful production design.

5) Separate data device and background recalculate

If you already have a data disk and want tags/journal on a second device:

# Conceptual pattern from integritysetup(8)
# DATA = existing payload device (will not be wiped if you pass --no-wipe)
# META = device that will hold superblock, journal, and tags

sudo integritysetup format --batch-mode \
  --data-device /dev/disk/by-id/data-disk \
  --no-wipe \
  /dev/disk/by-id/meta-disk

sudo integritysetup open \
  --data-device /dev/disk/by-id/data-disk \
  --integrity-recalculate \
  /dev/disk/by-id/meta-disk integ-data
Enter fullscreen mode Exit fullscreen mode

--integrity-recalculate lets the kernel fill tags in the background while the device is usable; protection is complete only when recalculation finishes. integritysetup dump exposes the recalculation offset so you can watch progress. --integrity-recalculate-reset (kernel 5.13+) restarts from the beginning—useful when changing checksum function without changing tag length.

6) Boot activation with /etc/integritytab

systemd-integritysetup-generator turns /etc/integritytab into systemd-integritysetup@.service units early at boot—parallel to crypttab / veritytab.

Line format (integritytab(5)):

volume-name  block-device  [keyfile|-]  [options|-]
Enter fullscreen mode Exit fullscreen mode

Examples:

# /etc/integritytab
# CRC journaled volume by PARTUUID, allow TRIM (kernel 5.7+)
scratch PARTUUID=4973d0b8-1b15-c449-96ec-94bab7f6a7b8 - allow-discards,journal-watermark=55%,journal-commit-time=10

# Defaults only
data PARTUUID=5d4b1808-be76-774d-88af-03c4c3a41761

# HMAC key file (absolute path). Algorithm defaults toward hmac-sha256 when a key file is present
secure PARTUUID=11111111-2222-3333-4444-555555555555 /etc/integrity/hmac.key integrity-algorithm=hmac-sha256

# Tags on one device, payload on another
home PARTUUID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee - data-device=/dev/disk/by-uuid/9276d9c0-d4e3-4297-b4ff-3307cd0d092f
Enter fullscreen mode Exit fullscreen mode

Useful options from the man page:

  • mode=journal|bitmap|direct (systemd v254+)
  • allow-discards
  • journal-watermark= / journal-commit-time=
  • data-device=
  • integrity-algorithm= (crc32c, crc32, xxhash64, sha1, sha256, hmac-sha256, hmac-sha512, phmac-sha256, phmac-sha512)
  • _netdev, noauto, nofail on newer systemd

After editing:

sudo systemctl daemon-reload
# Generator creates integritysetup units; exact unit name follows volume-name
systemctl status systemd-integritysetup@scratch.service 2>/dev/null || true
ls -l /dev/mapper/scratch 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Pair fstab mounts with matching _netdev / nofail semantics when the integrity device itself uses those flags, or you can create dependency loops and local-fs boot stalls—the integritytab man page calls this out explicitly.

7) LUKS2 authenticated encryption (stacked path)

Standalone dm-integrity authenticates (or checksums) plaintext on a writable device. If you also need confidentiality, the stacked pattern is LUKS2 + integrity, where dm-crypt generates integrity tags and dm-integrity stores/verifies them.

cryptsetup-luksFormat(8) documents --integrity as an experimental LUKS2 extension that requires the dm-integrity target. Native AEAD modes additionally need userspace AEAD support in the kernel crypto API (CONFIG_CRYPTO_USER_API_AEAD). Read the AUTHENTICATED DISK ENCRYPTION section in cryptsetup(8) for the pairing your build supports before converting anything important.

# Illustrative LUKS2 format with integrity tags
# WARNING: experimental; verify cipher/integrity pairing for your cryptsetup + kernel
sudo cryptsetup luksFormat \
  --type luks2 \
  --integrity hmac-sha256 \
  /dev/disk/by-id/your-disk

# Optional: skip initial tag wipe only if you accept unread sectors failing until first write
# sudo cryptsetup luksFormat --type luks2 --integrity hmac-sha256 --integrity-no-wipe ...

sudo cryptsetup open /dev/disk/by-id/your-disk crypt-auth
sudo mkfs.ext4 -L cryptauth /dev/mapper/crypt-auth
Enter fullscreen mode Exit fullscreen mode

What this buys you (kernel dm-integrity + dm-crypt docs):

  • Modified ciphertext should produce I/O errors, not silently decrypt to attacker-controlled plaintext.
  • You still manage LUKS keys/slots as usual (cryptenroll, passphrases, TPM2, FIDO2—covered in other operational guides).
  • Journaled integrity under LUKS can guarantee write atomicity for the encryption sector, at the cost of writing data twice.
  • Space overhead and write cost rise; benchmark on your storage before converting large fleets.

If you only need “don’t silently serve bit-flipped blocks” on an already-trusted host, standalone CRC integrity is simpler and has no key hierarchy. If the threat includes offline disk tampering and confidentiality, evaluate LUKS2 --integrity (experimental) or an AEAD setup your cryptsetup build documents—not CRC alone.

8) Discards, resize, and day-2 ops

# TRIM through integrity (internal-hash volumes; kernel 5.7+)
sudo integritysetup open --allow-discards "$LOOP" integ-lab
# Prefer keyed discard semantics on new keyed volumes when your cryptsetup/kernel
# documents allow_discards_keyed — constant filler tags are forgeable without the key.

# Status / mismatches / provided size
sudo integritysetup status integ-lab

# Grow (kernel 5.7+ for size increases). Recalculating flag is set after resize.
sudo integritysetup resize integ-lab
# Optional: wipe newly allocated area so tags start clean
sudo integritysetup resize --wipe integ-lab
Enter fullscreen mode Exit fullscreen mode

Kernel guidance worth internalizing:

  • allow_discards marks discarded blocks with a constant filler tag that anyone with raw write access can forge without a key. Prefer keyed discard marking on new keyed volumes when available.
  • Layout-defining parameters (interleave, tag size, algorithm, separate meta device) are not casual reload knobs—the on-disk layout depends on them.
  • Journal mode, buffer size, watermark, commit time, and discards can often be adjusted on reload; still test before production cutover.

9) Clean rollback for the lab

sudo umount /mnt/integ 2>/dev/null || true
sudo integritysetup close integ-lab 2>/dev/null || true
sudo integritysetup close integ-hmac 2>/dev/null || true
sudo integritysetup close integ-recovery 2>/dev/null || true

# Detach loops
for img in disk.img hmac.img; do
  dev=$(losetup -j "$LAB/$img" | awk -F: 'NR==1{print $1}')
  [ -n "$dev" ] && sudo losetup -d "$dev"
done

# Remove scratch (optional)
sudo rm -rf "$LAB"
Enter fullscreen mode Exit fullscreen mode

For production, reverse through fstab → integritytab → wipe only after backups are verified. Closing the mapper without a backup does not un-corrupt a damaged backing store.

Decision guide: integrity vs the neighbors

Need Tool
Immutable image / rootfs matches a published digest dm-verity + root hash (veritysetup, veritytab)
Writable volume detects silent bit flips dm-integrity standalone CRC/hash (integritysetup)
Writable volume authenticates sectors with a key dm-integrity HMAC or LUKS2 --integrity
Confidentiality (who can read) LUKS/dm-crypt (optionally stacked with integrity)
File-level package/config drift debsums, AIDE, audit watches
Filesystem checksum scrub + repair Btrfs scrub, ZFS scrub
RAID parity consistency mdadm check/repair

dm-integrity does not replace backups. A wiped or re-formatted integrity device is still gone. It replaces the failure mode where a flaky cable, bad SSD sector, or sneaky offline edit serves wrong bytes without complaint.

Practical recommendations

  1. Lab on loop files first. Confirm format/open/status/corruption behavior before touching real disks.
  2. Default to journal mode unless you have measured write cost and accept bitmap/direct crash semantics.
  3. Use CRC for accidental corruption; use HMAC or LUKS2 integrity for adversarial offline writes.
  4. Mount only /dev/mapper/.... Raw access bypasses checks by definition.
  5. Wire integritytab with stable PARTUUID/UUID identifiers, and keep HMAC key files out of world-readable paths.
  6. Watch capacity. Tag+journal overhead is real; size filesystems on the mapper device after open.
  7. Keep recovery mode and legacy recalculate off the happy path.
  8. Stack deliberately: verity for immutable publish/subscribe images; integrity for mutable disks; LUKS for secrecy.

Sources and references

  • Linux kernel docs: dm-integrity
  • integritysetup(8) — cryptsetup project man page
  • integritytab(5) — systemd integrity device table
  • systemd-integritysetup-generator(8) / systemd-integritysetup@.service(8)
  • cryptsetup project: gitlab.com/cryptsetup/cryptsetup
  • Related boundary tools: veritysetup(8), cryptsetup(8), cryptsetup-luksFormat(8)

Silent corruption is boring until it is not. dm-integrity will not make disks immortal, but it stops your stack from cheerfully serving wrong sectors as if nothing happened—and that is a much better default for every writable volume you actually care about.

Top comments (0)