Mustafa ERBAY

Posted on Sep 18 Originally published at mustafaerbay.com.tr

dm-integrity Finds the Bad Block; To Fix It, It Wants a Mirror

#dmintegrity #linux #kernel #devicemapper

[673528.865410] device-mapper: integrity: dm-0: Checksum failed at sector 0x44000

A few days ago I flipped a single bit under dm-verity on a read-only image and watched the kernel stop at exactly that block. At the end of that piece I left myself an honest note: most disks on a server are not read-only. Databases, logs, backups, container layers; all of them get written. dm-verity's hash tree is invalid after the first write. For those disks, what is the answer to "is what the disk hands me what I wrote"?

Linux's answer is dm-integrity. It arrived in 4.12 in 2017, it sits under LUKS2's authenticated encryption and inside LVM RAID's --raidintegrity option, but it can also be set up on its own. I once again started with a 1 GiB file on the server; I flipped a byte, moved a block out of place, closed and reopened the journal, and finally built a two-leg LVM mirror and broke one leg. The sentence in the title is the summary of that experiment: on its own, dm-integrity finds but does not fix. To fix, it wants a good copy next to it.

Overhead at 1.56 percent

integritysetup format with defaults took 6.1 seconds; all of that time goes into wiping the device with zeros, and the loop device's write counter showed about 1 GiB. The table from the superblock:

superblock_version 5
log2_interleave_sectors 15
integrity_tag_size 4
journal_sections 93
provided_data_sectors 2064392
sector_size 512
flags fix_padding fix_hmac

1 GiB is 2,097,152 sectors; I was left with 2,064,392. The 32,760 sectors in between, 16 MiB, come from three items: 64 tag areas holding a 4-byte CRC32C tag for every 512-byte sector (64 × 128 KiB = 8 MiB; the last one nearly empty), the journal, whose default size is 1/128 of the device (8,380,416 bytes; capped at 64 MiB, rounded to the section size), and the 4 KiB superblock. 1.56 percent. The overhead is reasonable, but change the tag size and the bill changes. Formatting the same file with --integrity hmac-sha256 --tag-size 32 left 1,957,896 sectors; the loss rose to 6.64 percent. The other way, with --sector-size 4096 the CRC32C tag drops to one per 4 KiB block and the loss fell to 0.86 percent. If your filesystem already works in 4 KiB blocks, the only thing you get from 512-byte sectors is double overhead.

I also noted a line that landed in dmesg during formatting, because the first time I saw it I thought I had broken something:

workqueue: integrity_metadata [dm_integrity] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND

Nothing is broken; the kernel is saying that the workqueue computing tags during the wipe held a core for more than 10 milliseconds. That queue is CPU-bound: the tag computation runs on whichever core the writing process is on. With a single writer, an 18-core machine is still one core.

What's on disk: superblock, journal, then tags and data interleaved

I opened the device, put ext4 on it and wrote a 64 MiB file whose first 4 KiB start with KOPRU-INTEGRITY-MARKER-2026. Then I searched for that marker with grep -boa both on /dev/mapper/ilab and on the raw file underneath. On the mapper it sat at byte 142,606,336 (sector 278,528, the 0x44000 in the log); in the raw file at byte 152,170,496. The 9,564,160-byte difference matches the layout in the kernel document exactly: a 4 KiB superblock, an 8,380,416-byte journal, then a 128 KiB tag area in front of every 32,768-sector data area. Sector 278,528 is in the ninth area; nine tag areas are 1,179,648 bytes. The sum works out with no remainder.

Rather than guess what the tag is, I computed it. The integrity_sector_checksum function in the source hashes first the 16-byte salt from the superblock (only when the fix_hmac flag is set), then the sector number (8 bytes, little-endian), then the data. I wrote a pure-Python CRC32C and tried sector 278,529: the tag on disk is 683dde2c, my calculation is 683dde2c. Skip the salt or hash only the data and it does not match. The consequences of that small detail come shortly.

One byte: 84

In the raw file I turned the A 100 bytes past the marker into a B, unmounted and remounted the filesystem (to empty the page cache) and read. A plain read() gave "Input/output error". Reading with O_DIRECT returned something else:

O_DIRECT: 84 Invalid or incomplete multibyte or wide character
buffered: 5 Input/output error

84 is EILSEQ. The block layer maps the BLK_STS_PROTECTION status to -EILSEQ (the "protection" row in the table in blk-core.c), and dm-integrity returns exactly that status for a mismatched tag. In a read that goes through the page cache the information is lost, because when ext4 cannot fill the page it only tells you EIO. So if you want to tell "did my disk fail or did integrity fail" apart at the application level, you have to accept glibc's "invalid multibyte character" message. The message is absurd, the code is right.

The rest of the file read fine; only that 4 KiB block was gone. The tag belongs to a 512-byte sector, but since ext4 reads 4 KiB pages the error is page-sized. The first number in the dmsetup status output is a mismatch counter that goes up once per read attempt:

0 2064392 integrity 6 2064392 -

I had tried reading six times; six. This counter lives in memory; it reset to zero when I closed and reopened the device, it is not written to the superblock. The Checksum failed line is rate-limited too (DMERR_LIMIT), so counting dmesg undercounts. If you are setting up monitoring, aggregating dmsetup status across restarts is on you; if you want a durable record, look at the integrity-checksum event that lands in the audit subsystem on kernels built with CONFIG_DM_AUDIT.

There is also a detail that stops the kernel from fooling you. Patch c88f5e55, which went into 6.8, reads the block once more when the tag does not match, this time into a kernel buffer userspace cannot touch, and recomputes. The reason was a real case: a process doing repeated O_DIRECT reads into the same buffer modified the buffer while the kernel was computing the hash, and a RAID leg got kicked out of the array over a false integrity error. The patch description says "RAID leg being kicked out of the array"; you will see why that line is there at the end of the article.

What was written to the journal may not have reached the data area

To fix the bad block I rewrote the same 4 KiB through the filesystem, said sync, read: clean. Then out of curiosity I checked the byte in the raw file. Still B.

byte+100 on disk now:  4241 4141  BAAA
after close:           4141 4141  AAAA

It became A when I closed the device with integritysetup close. This is the design itself. In dm-integrity's default J mode every write first goes to the journal together with its data and tag; sync commits the journal, but copying to the data area is a separate worker's job. According to the source, after a commit that worker is queued only if the number of free sectors in the journal has dropped below the threshold (the default 50 percent watermark); and when the device is suspended. In the meantime, someone looking at the raw disk would think "the fix was never made". For a forensic examination or a tool that takes block-level backups the meaning is clear: the raw blocks under a dm-integrity device may lag behind what the device above says.

The relocated block

To see what including the sector number in the tag buys, I tried this: sectors 278,529 and 278,530 have identical content (both are A from end to end), but their tags differ: 683dde2c and bde4bd3e. I copied 278,529's data and tag over 278,530's slots in the raw file; both are "valid" pieces, they only changed place.

device-mapper: integrity: dm-0: Checksum failed at sector 0x44002

The kernel caught it. Anyone moving a valid copy of a block to another location hits a wall even with CRC32C; the sector number is inside the hash. What is not caught is a rollback in place: put the old data back into the same sector together with its old tag and the tag stays valid, because there is no counter or version in the tag; HMAC does not solve this either. And one more thing: a CRC is not a signature. Someone with raw access to the disk can write new data and compute and place the right CRC next to it; the salt sits in the open in the superblock. If you want protection against a deliberate attacker you move to HMAC (--integrity hmac-sha256 and a key file); since the key is not on disk, the tag cannot be produced. But then the overhead is 6.64 percent and the number in dmsetup status may no longer mean "the disk rotted" but "someone touched it".

There is also --allow-discards. A TRIMmed block is marked with a constant filler tag (0xf6), and the kernel document now says explicitly that this can be forged without a key; it recommends allow_discards_keyed on new volumes; that one needs an HMAC volume and cannot be turned off once enabled. According to the integritysetup manual the option comes with kernel 7.3; today's Ubuntu 24.04 kernel does not have it.

The unit of repair is the filesystem block

To fix the bad sector I wanted to say "just write those 512 bytes". I tried writing 512 bytes at byte 1,024 of the file; dd gave "error writing: Input/output error". I cannot write, because ext4 has to read the rest of the 4 KiB page first, and that read comes back with the integrity error. Writing the whole 4 KiB went through. I then corrupted the next sector (278,531) as well and this time tried a 512-byte O_DIRECT write straight onto /dev/mapper/ilab; that fixed it on its own. The rule: dm-integrity's unit is its own sector, but everything above it thinks in its own block; the repair has to be at least that block. If the database page is 8 KiB, that is two ext4 blocks, sixteen sectors at once.

Writing twice, but not always

The document says write throughput "degrades twice" in J mode. I measured it, using the loop device's /sys/block counter to count how many sectors dm-integrity really wrote underneath. Two workloads: sequential 256 MiB with O_DIRECT (1 MiB pieces) and 4,096 random 4 KiB writes within the first 512 MiB (16 MiB).

                 sequential 256 MiB            random 4096 x 4 KiB
raw loop         1.41 s   524,288 sectors       2.10 s   32,768 sectors
J (journal)      2.89 s 1,076,472 sectors x2.05 0.52 s   90,424 sectors x2.76
D (no journal)   1.54 s   528,384 sectors x1.01 1.91 s   48,144 sectors x1.47
B (bitmap)       1.20 s   528,512 sectors x1.01 1.68 s   48,272 sectors x1.47

The times are only relative, since the loop device writes into the page cache; the sector counts are exact. For sequential I/O the document is right: J mode writes 2.05 times as much and the time approaches double. For random I/O the table turns around. J mode finished in 0.52 seconds despite writing 2.76 times the sectors; D mode took 1.91, the raw loop 2.10. Two things overlap here. The journal gathers scattered small writes and writes them as sequential blocks; lvmraid(7) describes this as "scattered writes packed into a single journal write", and the sector counts show that packing. What really shortens the time is this: according to the source, a write without FUA counts as complete as soon as it is copied into the in-memory journal; reaching the disk is left to the commit. The 0.52 seconds is largely a measure of that acceptance. In a database that fsyncs after every write, each fsync is a commit; most of the gain melts there. The 47 percent surplus in D and B modes is writes to the tag area; for every 4 KiB of data, part of a tag buffer also goes to disk.

Bitmap mode is fast, true; but its price is one sentence in the document: data corrupted at the moment of a crash may go undetected, because dirty regions are recalculated. So in B mode a "data and tag do not match" state after a crash is closed by "fix the tag to match the data". The integrity layer closing its eyes at exactly the moment you want it to protect you. I would not choose B on a write-heavy machine that can lose power; if the server has no UPS, never.

Adding it to an existing disk

Formatting a new disk is easy; the real question is whether it can be added to a full one. It can: I gave a full 512 MiB ext4 image as --data-device and a separate 64 MiB file as the metadata device, formatted with --no-wipe, and opened with --integrity-recalculate. The third field of dmsetup status shows the progress:

t=1: 0 1048576 integrity 0 1048576 425984
t=2: 0 1048576 integrity 0 1048576 786432
t=3: 0 1048576 integrity 0 1048576 1048576

With half-second samples it finished in a second and a half; the SHA-256 of the 200 MiB file is the same as before. For blocks read before the calculation is done, the kernel skips the tag check (in the source, reads beyond the recalc_sector boundary go to skip_check); so the protection is only as far as the progress bar. One oddity: after the job is done, flags recalculating stays in the superblock and recalc_sector equals the device size. There is not a single line in the 6.8 source that clears that flag; the way to say "done" is the two numbers being equal.

The mirror: LVM RAID1 + raidintegrity

Now the second half of the title. I made two 1 GiB loop devices PVs and built a mirror with lvcreate --type raid1 -m 1 --raidintegrity y -L 512M. LVM puts a dm-integrity layer in front of each leg and allocates a 12 MiB _imeta sub-LV for the tags:

mirror                  512.00m raid1     93.09   mirror_rimage_0(0),mirror_rimage_1(0)
[mirror_rimage_0]       512.00m integrity 100.00  mirror_rimage_0_iorig(0)
[mirror_rimage_0_imeta]  12.00m linear            /dev/loop9(129)
[mirror_rimage_0_iorig] 512.00m linear            /dev/loop9(1)

I wrote the same marked file, found it at byte 147,849,216 in both PV files; I corrupted the one in the first leg and read the file:

md/raid1:mdX: dm-3: rescheduling sector 278528
device-mapper: integrity: dm-3: Checksum failed at sector 0x44000
md/raid1:mdX: read error corrected (8 sectors at 278528 on dm-3)
md/raid1:mdX: redirecting sector 278528 to other mirror: dm-3

The lines are as dmesg printed them. The read succeeded, the SHA-256 is the same, lvs -o integritymismatches shows 3 on the first leg (md reads the bad leg more than once while correcting, and every attempt is counted), and the byte in the raw file has turned back into A by itself. What dm-integrity alone could not give is here: RAID1 sees the integrity failure as a read error, reads from the other leg and writes the correct data back onto the bad leg. The same mechanism is why the "recheck" patch in 6.8 matters; a false tag error could get a healthy leg thrown out of the array.

Second question: does corruption sit there until it is read? It does; I tried. This time I corrupted the second leg and, without reading the file at all, ran lvchange --syncaction check. It reached 100 percent in two seconds, raid_mismatch_count 128, the second leg's integritymismatches count 1. I looked at the raw file: the byte is still corrupt. I tried --syncaction repair, and LVM refused:

Use syncaction check to detect and correct integrity checksum mismatches.

According to LVM, check both finds and corrects. So why is the byte still bad? The answer is two sections back: the correction was sitting in that leg's dm-integrity journal. When I deactivated the LV with lvchange -an, the A appeared in the raw file; reopening and reading gave zero errors. So a regular --syncaction check is part of this setup; otherwise the error shows up six months later in front of whoever reads that file first, and by then the second leg may be sick too.

lvmraid(7) also lists the limits of this arrangement: with integrity on there is no lvreduce, pvmove, --splitmirrors or --rebuild; before lvconvert --repair or --replace you turn it off with --raidintegrity n and turn it back on afterwards, which means recalculating all the tags. There is also the 4 MB of metadata per leg for every 500 MB of data, and the requirement that the area be on the same PV as the data; if the PV is full during lvextend, it gets stuck.

Before you set it up

The integrity layer finds, it does not fix. For fixing, either RAID1/4/5/6/10 with --raidintegrity y, or a restore from backup. On a single-disk server dm-integrity only gives you the "it broke" news earlier; that is something too.
Match the sector size to your filesystem's. A 512-byte tag on top of 4 KiB ext4 doubles the overhead for nothing.
J mode is the default and the right default. Twice the writes for sequential; a packing gain for scattered, but one commit per fsync. Consider B mode only where a power cut is truly impossible.
The dmsetup status counter lives in memory; monitoring has to aggregate it across restarts. lvs -o integritymismatches reads the same number.
Do not say "it's fixed" by looking at the raw disk; the write in the journal can wait until the watermark.
HMAC only makes sense against deliberate tampering; the key must not sit on the disk and has to be supplied at every open. CRC32C is enough against rot, not against an attacker.
If the tag area is damaged and the device will not open, R mode (--integrity-recovery-mode: no tag checks, no writes) recovers the data read-only; if known-good data was written back from a backup, --integrity-recalculate-reset (5.13) regenerates the tags from scratch.
For automatic setup at boot, /etc/integritytab (mode=journal|bitmap|direct); LUKS2's --integrity option is still stamped EXPERIMENTAL in the manual.
If your NVMe disk offers a PI (DIF) field, --integrity-inline (kernel 6.11 plus cryptsetup 2.8; not in the 2.7.0 shipped with 24.04) works without a journal and without extra writes; but the disk has to be low-level formatted to that LBA format.

Would I put it on this server

This machine has a single disk, with a hypervisor's own storage underneath. Putting dm-integrity on a single disk would only buy me an earlier error message; I already get that message from the filesystem and application layers. If it were a setup with legs, a backup server with two physical disks say, I would build it with --raidintegrity y and put a weekly --syncaction check on a timer. Because RAID1 sees the difference between two legs but does not know which one is right; the tag gives it that knowledge. That is exactly what the experiment taught me: the mirror itself cannot answer "which one is right", and the integrity layer cannot answer "what is the right one". Put them side by side and you get a disk that heals itself; on their own, two guards, one blind and the other without arms.

DEV Community