In the Secure Boot post I followed the chain up to the kernel: firmware verifies shim, shim verifies GRUB, GRUB verifies the kernel. Then the kernel reads this line, and the chain ends there:
BOOT_IMAGE=/vmlinuz-6.8.0-139-generic root=UUID=6d085073-3c5a-49aa-9133-d0c30d62315b ro
That is my server's /proc/cmdline. The kernel finds the disk with that UUID, mounts it as ext4 and trusts whatever is on it. The ro flag verifies nothing; it only stops the kernel from writing, not someone who touches the disk from elsewhere. Signed kernel, unsigned root file system: the longest link in the verification chain is empty.
On Linux the mechanism that fills that link is dm-verity. Many have heard the name, few have held it: it is what Android phones use on every boot, and it is almost never set up on servers. I had never set it up either. For this post I prepared a 1 GiB image on the server, built the hash tree, then flipped a single bit in the image and watched what the kernel did. It was stricter than I expected, and looser than I expected in two places.
The tree: 262,144 blocks, 2,065 hashes
Setup is short. data.img is a 1 GiB file filled with random content; I created ext4 inside it and put a few files in (/etc/passwd, three binaries and 200 MiB of random data called usr/big.bin). Then:
# veritysetup format data.img hash.img
VERITY header information for hash.img
UUID: c3563772-266c-4781-89a0-9aa199734a74
Hash type: 1
Data blocks: 262144
Data block size: 4096
Hash blocks: 2065
Hash block size: 4096
Hash algorithm: sha256
Salt: 8d3e33011c1e8896ad7fcb7056942c84aec49c416b8deb3f9d26508404f81f59
Root hash: a6439beae82753ae3be8a619fd3279fa6058eca0f6a740e772f25c183840078c
It took 4.08 seconds. The arithmetic works out by hand: a 4096-byte hash block holds 128 SHA-256 outputs of 32 bytes each. 262,144 data blocks need 2,048 leaf hash blocks; the hashes of those 2,048 blocks fit in 16 blocks, and those 16 fit in a single root block. 2,048 + 16 + 1 = 2,065. The hash of the root block (together with the salt) is the 32 bytes we call the root hash. hash.img is 8,462,336 bytes: one superblock plus 2,065 hash blocks, 0.79% of the data.
The thing to notice: when the kernel reads a data block it computes that block's hash and compares it with the entry in the leaf block; the leaf block itself is compared against an entry in the block above it, and that one against the root. Verified hash blocks are kept in memory with a hash_verified mark, so not every read re-hashes every level, but that is the logic, and the single point of trust at the end of the chain is the root hash.
Opening is not verifying
My first surprise was here. I opened the device with a wrong root hash — I changed the last hex digit of the correct value from c to d:
# veritysetup open data.img vroot hash.img a643...078d
Verity device detected corruption after activation.
# echo $?
0
Exit code zero. The device is sitting there as /dev/mapper/vroot. The kernel reads no block while it builds the table; it finds out whether the root hash is right during the first read. Even the warning line does not come from the kernel; it comes from veritysetup checking dmsetup status once more after activation (cryptsetup lib/verity/verity.c, dm_status_verity_ok). So why does that status say "corrupted" before I have read anything? Because as soon as the device was created, udev's blkid scan tried to read the first block; ten lines of metadata block 1 is corrupted landed in dmesg. Without udev it would have said status: verified; indeed when I open with the correct hash and only run veritysetup status, it also says "verified", because nothing has been read yet.
If you want to check everything end to end without opening the device or touching the kernel, that is a separate tool:
# veritysetup verify data.img hash.img a643...078d
Verification of root hash failed.
# veritysetup verify data.img hash.img a643...078c && echo OK
OK
verify recomputes the whole tree in user space; 4.53 seconds for 1 GiB, the same order as format. open takes 0.15 seconds; in those 0.15 seconds no block is read, so nothing is verified.
One bit
With filefrag I found where big.bin lives inside ext4: the file starts at block 34,816. I went to the file's block 1000, which is the device's block 35,816, picked a byte in the middle of the block and flipped its lowest bit (f6 → f7). Closed the device, reopened it, mounted it:
# sha256sum mnt/etc/passwd
e787b373a74594b33b77892a903647d7a6cf519d427a0b0790e11be4482696e0 mnt/etc/passwd
# sha256sum mnt/usr/big.bin
sha256sum: mnt/usr/big.bin: Input/output error
# dd if=mnt/usr/big.bin bs=4096 count=1000 of=/dev/null # first 1000 blocks
(succeeded)
# dd if=mnt/usr/big.bin bs=4096 skip=1000 count=1 of=/dev/null
dd: error reading 'mnt/usr/big.bin': Input/output error
# dd if=mnt/usr/big.bin bs=4096 skip=1001 of=/dev/null # 1001 to the end
(succeeded)
The kernel side:
device-mapper: verity: 7:7: data block 35816 is corrupted
Three lines landed in dmesg: two from the reads I made on purpose, the third most likely from the first dd's readahead; even readahead goes through verification. The remaining 51,199 blocks of the file read fine; only the 4 KiB block containing the bit I flipped returns EIO. At the same moment the device also emits a udev event; I caught it with udevadm monitor -p:
KERNEL[415628.654463] change /devices/virtual/block/dm-0 (block)
DM_VERITY_ERR_BLOCK_NR=0,35816
0 is the block type (data), 35816 the block number. User space can listen for this; on a server you can attach a udev rule to that event and raise an alert. veritysetup status now says corrupted, and that flag does not go back until the device is closed. There is one more limit: the kernel logs at most 100 corruptions per device (DM_VERITY_MAX_CORRUPTED_ERRS), then writes a single "reached maximum errors" line and goes quiet. Past a hundred blocks you cannot count corrupted blocks from the log.
What the page cache hides
Second surprise. I flipped the bit back, opened the device clean and read big.bin once from start to end; the hash was correct. With the device open and mounted, I flipped the same bit in the underlying data.img again and read the file once more:
1f82c51030a858cbd668ac0234481d11ff86bad84a0e55890d9ee5e78e3a23c9 mnt/usr/big.bin
exit=0, no new line in dmesg, status: verified
Correct hash, no error. Because the second read never went to disk; it came from ext4's page cache, from the pages of big.bin that were verified during the first read and stayed in memory. dm-verity works at the block layer and sees a block only as it passes through from disk. Drop the cache and the picture changes:
# echo 3 > /proc/sys/vm/drop_caches
# sha256sum mnt/usr/big.bin
sha256sum: mnt/usr/big.bin: Input/output error
Do not read this as a weakness; in the intended model the disk does not change while it is running, because writing underneath a read-only, verified image while it is live is outside the threat model anyway. But the sentence "dm-verity is on, therefore every byte I read was verified just now" is wrong. The byte you read was verified when it passed through from disk; that moment may have been an hour ago.
There is an option that makes the same trade-off deliberately and permanently: check_at_most_once. The kernel keeps a one-bit map per data block (32 KiB for 262,144 blocks), and once a block has been verified it never hashes it again. I tried it: opened clean, read the file once, flipped the bit, dropped the cache, read again:
309954a3d1bb33e182af5e403d51a2575f096c91225394a346a182dd558ddea9 mnt/usr/big.bin
exit=0, dmesg empty, status: verified
This time the disk really was read, the corrupt block was handed to the user, the hash changed, and there is no trace anywhere. The veritysetup man page puts a bold warning on this flag: it catches only offline tampering, not online tampering. It exists to reduce overhead; I cannot see a reason to switch verification off this way on a server.
One bit in the hash tree: 512 KiB gone
This time I flipped a bit not in the data but in hash.img. My target was hash block 297: the leaf level starts at block 18, the hash of data block 35,816 is in leaf block 35816 / 128 = 279, and 18 + 279 = 297. In that block I touched a byte of the entry at index 5, the sixth counting from zero: the hash of data block 35,717. Result:
data block 35711 OK
data block 35712 EIO
data block 35717 EIO
data block 35816 EIO
data block 35839 EIO
data block 35840 OK
device-mapper: verity: 7:7: metadata block 297 is corrupted
I corrupted a single hash entry, but because the kernel cannot verify the whole leaf block against the level above, all 128 data blocks under that block, 35,712 through 35,839, 512 KiB, became unreadable. One bit in the data takes one block; one bit in the hash tree takes a hundred and twenty-eight. The higher up the tree you go, the more: one of the 16 upper-level blocks closes 16,384 blocks, and the root block closes everything. That is why the hash partition's neighbourhood on disk is worth more than the data's.
What should happen on corruption: three modes
The default behaviour is the EIO you saw. In the kernel source verity_handle_err knows three modes, and they map one to one onto veritysetup open flags:
-
--ignore-corruption: log, send the uevent, hand over the corrupt block. I tried it;sha256sumhappily printed the wrong hash, a line landed indmesg, and status becamecorrupted. This mode is for monitoring; it does not protect the data. -
--restart-on-corruption:kernel_restart("dm-verity device corrupted"). The machine reboots. The kernel documentation says it "requires user space support", meaning that to avoid reading the same block on the next boot and rebooting again, someone (on Android, the bootloader) has to see the state and switch to the backup slot. I did not try this on a live server, for obvious reasons. -
--panic-on-corruption:panic(...). For those who want a crash dump rather than a reboot.
Since 6.12 there is also a pair that treats I/O errors like corruption, restart_on_error/panic_on_error; the veritysetup counterpart is --error-as-corruption, not yet in 2.7.0. It means reboot even if the disk is physically unreadable; my 6.8 has neither. On a server I would stay with the default: EIO crashes the application, the crashed application raises an alert, the alert reaches a human. Rebooting only makes sense if you have a spare root image and bootloader logic that switches to it; otherwise a single bad block puts the machine into an endless boot loop.
FEC: the part Ubuntu did not build
Since Android 7, forward error correction data (Reed-Solomon) can travel alongside dm-verity; the aim is to repair a corrupt block instead of declaring it unreadable. I generated it:
# veritysetup format --fec-device=fec.img --fec-roots=2 data.img hash-fec.img
FEC device size: 8560640 [bytes]
format+fec: 35.19 s
The hash tree took 4 seconds; with parity data, 35 seconds. The FEC file is 8.2 MiB (8,560,640 bytes), 0.8% of the data (the kernel documentation gives exactly that ratio for fec_roots=2). Then I tried to open it:
# veritysetup open --fec-device=fec.img data.img vfec hash-fec.img 1edc...1315
device-mapper: reload ioctl on vfec (252:0) failed: Invalid argument
device-mapper: table: 252:0: verity: Invalid number of feature args (-EINVAL)
I looked at the source for what the kernel is saying. The upper bound on table options is DM_VERITY_OPTS_MAX; in 6.8 it is 4 + DM_VERITY_OPTS_FEC + DM_VERITY_ROOT_HASH_VERIFICATION_OPTS (the constant became 5 in master when restart_on_error/panic_on_error were added). When CONFIG_DM_VERITY_FEC is off, the middle term is defined as zero in dm-verity-fec.h; the last one is 2 when signature support is on. veritysetup sends eight options for FEC, the limit on my kernel is six, the answer is "invalid number of feature args". The kernel cannot even say it does not recognise the word FEC; it trips on the count.
# grep DM_VERITY_FEC /boot/config-6.8.0-139-generic
# CONFIG_DM_VERITY_FEC is not set
Off in Ubuntu 24.04's 6.8 kernel. To see whether a newer kernel turned it on, I downloaded the linux-modules-6.17.0-42-generic package from noble-updates and looked at the config file inside: is not set there too. Debian's trixie kernel, 6.12.107-1, ships with CONFIG_DM_VERITY_FEC=y. Same veritysetup, same image, same command; on Debian it repairs, on Ubuntu it does not open. I had not thought of FEC as a distribution decision; the Kconfig help text says "If unsure, say N", and Ubuntu's choice was N. This is also why I could not measure the repair itself; in this post I can show only that FEC data was generated and that Ubuntu rejected it, not that it works.
Who holds the root hash
Everything so far rested on the assumption that the root hash is correct. Who protects it? The answer starts with the command line.
The first is the kernel command line. When systemd-veritysetup-generator sees the roothash= parameter, it does not even need you to name the data and hash partitions: it looks for the data partition under the GPT partition UUID derived from the first 128 bits of the root hash, and for the hash partition under the one derived from the last 128 bits. systemd-repart writes the UUIDs of Verity=data and Verity=hash partitions by the same rule when it builds the image; so give the right hash and the right partitions are found, give the wrong hash and the partitions are not even found. The root hash is protected exactly as well as the command line itself: not at all if it sits in GRUB's editable menu, as well as Secure Boot if it is embedded in a signed UKI.
The second is a signature. If the kernel is built with CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG (Ubuntu's is), veritysetup open --root-hash-signature accepts a PKCS#7 signature; the kernel verifies it against the builtin trusted keyring and, if CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG_SECONDARY_KEYRING is on (it is on Ubuntu), the secondary keyring (dm-verity-verify-sig.c, verify_pkcs7_signature; master additionally has its own .dm-verity keyring, which 6.8 does not). You can also make it reject unsigned tables outright; the module has a parameter for that:
# modprobe dm_verity require_signatures=1
# veritysetup open data.img vroot hash.img a643...078c
device-mapper: reload ioctl on vroot (252:0) failed: Required key not available
device-mapper: table: 252:0: verity: Root hash verification failed (-ENOKEY)
The parameter is read-only (0444), so it cannot be changed afterwards; it is given at module load, preferably from modprobe.d. I ran this test; I could not do the signed activation, because this virtual machine has no UEFI and no shim; enrolling a certificate through MOK, or building my own kernel with CONFIG_SYSTEM_TRUSTED_KEYS, was not something to do here.
The last is carrying the hash not on the command line but in a signed partition verified against a certificate: systemd-repart's Verity=signature partition (it holds the root hash's signature as JSON) and DDI images go in that direction, but that is beyond this post.
Cost
On an 18-vCPU server that has other work to do, I read 1 GiB three times with the cache dropped: 3.2 to 4.4 seconds from the loop device, 4.4 to 6.8 seconds from the dm-verity device. The numbers are noisy and the machine is shared; I cannot give a clean "X percent". But the upper bound is easy to compute: the kernel logs which implementation it picked (verity: sha256 using implementation "sha256-ni"), and on the same CPU openssl speed -evp sha256 gives 1.0 to 1.2 GB per second per core on 1 to 8 KiB chunks. So a 1 GiB read adds at most one core-second of hashing, and that work runs in parallel on the kverityd worker threads. If the disk delivers 300 MB/s the bottleneck is the disk; on NVMe the arithmetic changes. Second and later reads come from the page cache; dm-verity does not run on that path.
Before you set it up
The first question is whether the root file system is really immutable. If not, dm-verity is not for you; the tree becomes invalid on the first write. A server that updates packages has to move to image-based deployment first; that is the on-disk counterpart of the model in the immutable infrastructure post. When the image changes, the root hash changes, so an update means a second pair of partitions (A/B slots) and a bootloader that switches slots; the data partition cannot be grown afterwards, format generates a random salt every time, and if you want a reproducible image you need --salt or systemd-repart.
The second question is where the root hash lives and who can change it. If the answer is "the cmdline", the next question is "who signs the cmdline". The location of the hash tree also needs a decision: a separate partition, or the tail of the data partition via --hash-offset; the latter makes A/B easier with a single partition.
The third is what happens on corruption: EIO or a reboot? A reboot without a spare slot is a loop. If you want repair, check /boot/config-* on your kernel for CONFIG_DM_VERITY_FEC; Ubuntu does not have it.
And one small detail that bites: when you mount ext4 read-only on top of verity, the file system must be clean; replaying a dirty journal needs writes, the device is not writable, so the mount stops there (the noload option skips the journal, but that means accepting inconsistencies); directories that get written, like /etc and /var, need an overlay or tmpfs. The last check is the dmsetup table output: if you see a number after the salt followed by check_at_most_once or ignore_corruption, ask who added it and why.
Would I put it on this server
Not on VPS3, and the reason is not technical but operational: this machine lives on apt upgrade, its root file system changes every week. dm-verity belongs not on machines that live by patching, like the ones in the "hotpatch" post, but on machines whose image is replaced as a whole: kiosks, edge devices, CI runner images, Kubernetes nodes. There, the setup is one format, one roothash= and a signed UKI. I would carry what I learned today into that setup: opening does not verify, the cache keeps the verified copy, and the hash partition is worth more than the data.
The lab stayed on the server under /tmp/verity-lab; mappings closed, require_signatures back to zero.
Versions used: Ubuntu 24.04, kernel 6.8.0-139-generic (verity target v1.10.0), cryptsetup/veritysetup 2.7.0. For Android's dm-verity page see source.android.com; the Kconfig help text is from the kernel master branch; the option limit and verity_handle_err were also checked against the v6.8 tag. The page-cache experiment was done over loop devices; on a real partition you would write to the underlying block with dd on the raw disk, the mechanism is the same.
Official Sources
- dm-verity — kernel admin guide: table format, options, tree structure
- dm-verity-target.c — verity_handle_err, DM_VERITY_MAX_CORRUPTED_ERRS, DM_VERITY_OPTS_MAX
- dm-verity-fec.h — DM_VERITY_OPTS_FEC is 0 when CONFIG_DM_VERITY_FEC is off
- dm-verity-verify-sig.c — require_signatures and root hash signature verification
- drivers/md/Kconfig — DM_VERITY_FEC: "If unsure, say N"
- veritysetup(8) — format/open/verify, the --check-at-most-once warning, corruption modes
- cryptsetup lib/verity/verity.c — status check after activation
- systemd-veritysetup-generator — roothash= and GPT UUID derivation
- repart.d(5) — Verity=data|hash|signature, VerityMatchKey=
- Debian linux 6.12.107-1 kernel config — CONFIG_DM_VERITY_FEC=y
- Ubuntu noble-updates linux-modules-6.17.0-42-generic
Top comments (0)