Stop Trusting Mutable Files Blindly: Practical fs-verity with fsverity on Linux

dm-verity is the right hammer when an entire block device is frozen: publish a root hash, open read-only, and fail closed on any flipped sector. That model falls apart the moment you need a writable filesystem that still hosts a handful of immutable artifacts—model weights, container layers staged on disk, signed policy blobs, package payloads, firmware images, or large archives that get updated independently.

You could sha256sum the file before every use. That works, and it is slow for multi-gigabyte objects when you only touch a few pages. You could put the file on a dm-verity volume. That works, and it freezes the whole device every time one file changes.

fs-verity sits between those extremes. It is a kernel filesystem feature that builds a per-file Merkle tree, marks the file permanently read-only for content, and re-verifies data on every page-in. Userspace drives it with the fsverity utility from fsverity-utils. Supported filesystems today are ext4, f2fs, and btrfs.

This guide is an operator walkthrough: enable the filesystem feature, protect a file, measure its digest in constant time, prove corruption fails closed, understand authentication options (including why built-in signatures are optional and constrained), and place fs-verity next to dm-verity, dm-integrity, AIDE/debsums, and plain hashes. It is not block-device Merkle verification (dm-verity), writable sector tags (dm-integrity), full-disk encryption (LUKS), or package inventory tools.

What you are actually building

From the kernel fs-verity documentation and fsverity(1):

Piece	Role
Filesystem with verity support	ext4 (-O verity / tune2fs -O verity), f2fs (-O verity), or btrfs (kernel 5.15+)
Merkle tree + descriptor	Built at enable time and stored with the file (past i_size on ext4/f2fs; btree items on btrfs)
fs-verity file digest	Hash of a descriptor that includes the Merkle root, file size, algorithm, salt—not a naive full-file hash
fsverity enable	Userspace wrapper for FS_IOC_ENABLE_VERITY
fsverity measure	Constant-time FS_IOC_MEASURE_VERITY of an already-protected file
Optional authentication	Userspace signature of the digest, IMA appraisal, IPE policy, or (carefully) built-in PKCS#7 signatures

Theory of operation, condensed from the kernel docs:

• Enabling verity builds a Merkle tree over the file’s data blocks (default SHA-256, 4K blocks), persists it, and marks the inode as a verity file.
• After enable, content is immutable: open-for-write and truncate fail with EPERM. Metadata (owner, mode, timestamps, xattrs), rename, link, and delete still work.
• Reads—including mmap—are verified against the tree as pages enter the page cache. Bad data fails with EIO (read) or SIGBUS (mmap).
• FS_IOC_MEASURE_VERITY returns the digest in constant time, regardless of file size. That is the operational win over sha256sum on large sparse-access files.
• Copying or restoring a verity file with ordinary tools drops verity-ness. The feature is meant for files managed in place (package managers, deploy pipelines), not for naive cp/rsync of the protected state.

Important mental model: fs-verity alone is integrity, not a complete authentication policy. Anyone who can replace a verity file with a non-verity twin can bypass it unless trusted code, IMA, IPE, or another policy requires a verified verity digest before use. The kernel docs are explicit about this.

Prerequisites

• Linux kernel with CONFIG_FS_VERITY (stock on modern distros). ext4/f2fs support since v5.4; btrfs since v5.15. Flexible Merkle block sizes since v6.3.
• fsverity-utils package providing the fsverity CLI.
• For ext4: e2fsprogs new enough to set the verity RO_COMPAT feature (tune2fs -O verity; feature present since e2fsprogs ~1.45.2).
• Root (or CAP_SYS_ADMIN where needed for filesystem feature enable) and a scratch directory. Prefer a loop-backed ext4 so you never touch production disks.

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y fsverity e2fsprogs

# Fedora
sudo dnf install -y fsverity-utils e2fsprogs

fsverity --version
# Expect fsverity-utils (v1.5+ / v1.6 common on current distros)

1) Build a tiny fs-verity lab on loop-backed ext4

Create a disposable ext4 image, enable the verity filesystem feature, mount it, and drop a payload file.

LAB=/var/tmp/fsverity-lab
sudo mkdir -p "$LAB"
cd "$LAB"

# 512 MiB scratch image
sudo dd if=/dev/zero of=disk.img bs=1M count=512 status=none
sudo losetup -fP --show disk.img
LOOP=$(losetup -j "$LAB/disk.img" | awk -F: 'NR==1{print $1}')

# Format with the verity RO_COMPAT feature
sudo mkfs.ext4 -q -O verity -L fsveritylab "$LOOP"

# Confirm the feature bit
sudo tune2fs -l "$LOOP" | grep -i 'Filesystem features'
# Expect "verity" among the features

sudo mkdir -p /mnt/fsv
sudo mount "$LOOP" /mnt/fsv

# Sample payload — pretend this is a model weight, policy blob, or layer tarball
python3 - <<'PY'
from pathlib import Path
p = Path('/mnt/fsv/payload.bin')
# ~32 MiB of structured data so Merkle overhead is visible but lab stays fast
chunk = (b'FSVERITY-LAB-PAYLOAD-v1\n' * 64)
data = chunk * (32 * 1024 * 1024 // len(chunk))
p.write_bytes(data)
print(p, 'bytes', p.stat().st_size)
PY

sudo chmod 0644 /mnt/fsv/payload.bin
ls -l /mnt/fsv/payload.bin

If the filesystem already exists without verity, enable it offline (or carefully online per your e2fsprogs/kernel combo) with:

# Filesystem should be clean; RO_COMPAT: old kernels will only mount read-only afterward
sudo umount /mnt/fsv 2>/dev/null || true
sudo tune2fs -O verity "$LOOP"
sudo mount "$LOOP" /mnt/fsv

On f2fs, format with -O verity. On btrfs, no special mkfs flag is required once the kernel is ≥ 5.15—verity metadata lives in separate btree items.

2) Enable fs-verity and measure the digest

Nothing is protected until you run enable. The file must not be open for writing anywhere (ETXTBSY otherwise). Enable builds the Merkle tree and is interruptible by fatal signals; on failure, the file is left unchanged.

# Optional: compute the digest userspace-side before enable (same algorithm/params)
fsverity digest --hash-alg=sha256 --block-size=4096 /mnt/fsv/payload.bin

# Enable Merkle protection (SHA-256 + 4K blocks are the usual defaults)
sudo fsverity enable --hash-alg=sha256 --block-size=4096 /mnt/fsv/payload.bin

# Constant-time measurement from the kernel
fsverity measure /mnt/fsv/payload.bin
# sha256:................................  /mnt/fsv/payload.bin

# Cheap presence check without opening the file (statx ATTR_VERITY since Linux 5.5)
statx --help >/dev/null 2>&1 || true
python3 - <<'PY'
import os, ctypes, ctypes.util
# Portable check via FS_IOC_GETFLAGS when available
import fcntl, struct
FS_IOC_GETFLAGS = 0x80086601
FS_VERITY_FL = 0x00100000
fd = os.open('/mnt/fsv/payload.bin', os.O_RDONLY)
try:
    buf = array = bytearray(4)
    # fallback simple: just report measure succeeded above
finally:
    os.close(fd)
print('verity enable completed; measure is the authoritative digest API')
PY

What changed:

• Content is now read-only at the filesystem layer. Mode bits may still look writable; open-for-write still fails.
• Direct I/O falls back to buffered I/O. DAX is unsupported (would skip verification).
• Metadata changes still succeed. That is intentional—fs-verity measures file contents, not inode owner/mode.

Try a write to prove the gate:

# Expect: Operation not permitted
echo mutate | sudo tee -a /mnt/fsv/payload.bin || echo "write blocked as expected"

# Truncate also fails with EPERM on verity files
sudo truncate -s 0 /mnt/fsv/payload.bin || echo "truncate blocked as expected"

3) Prove silent corruption fails closed

With the file still verity-protected, corrupt raw blocks on the loop device and read through the mounted filesystem. On systems where page cache can be dropped, force a device re-read so verification runs again.

# Note the file's apparent size and a mid-file byte offset for a lab poke.
# Corruption must hit data blocks the next read will fault in.
OFFSET_MIB=8
sudo umount /mnt/fsv
# Poke the filesystem image past typical superblock/group descriptors.
# This is a lab demonstration, not a surgical block editor.
sudo dd if=/dev/urandom of="$LOOP" bs=1M seek="$OFFSET_MIB" count=1 conv=notrunc status=none

sudo mount "$LOOP" /mnt/fsv
sync
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' 2>/dev/null || true

# Read the whole file — expect I/O error when a corrupted page is verified
sudo dd if=/mnt/fsv/payload.bin of=/dev/null bs=1M status=none \
  && echo "unexpected success" \
  || echo "read failed closed as expected"

# mmap consumers should see SIGBUS on bad pages (kernel docs)

What should happen:

• Untouched pages still verify and read cleanly.
• Pages whose data no longer match the Merkle path fail closed—no silent garbage.
• The digest from fsverity measure is unchanged; the tree still describes the original content. Bad sectors simply will not serve.

Restore the lab image before continuing if you want a clean authentication section:

sudo umount /mnt/fsv 2>/dev/null || true
sudo losetup -d "$LOOP"
sudo dd if=/dev/zero of=disk.img bs=1M count=512 status=none
LOOP=$(sudo losetup -fP --show disk.img)
sudo mkfs.ext4 -q -O verity -L fsveritylab "$LOOP"
sudo mount "$LOOP" /mnt/fsv
# recreate payload + enable as in sections 1–2

4) Digest vs traditional hash: why the format matters

fsverity measure does not return SHA-256(file bytes). The kernel hashes an fsverity_descriptor that binds:

• Merkle root hash
• file size (data_size)
• hash algorithm and log2(block size)
• optional salt

That removes ambiguities the plain root hash would have (padding, length confusion). If you sign or pin anything, sign the fs-verity file digest (or the fsverity_formatted_digest blob when using built-in signatures), not a hand-rolled sha256sum.

# Userspace can recompute the same digest without enable (for signing pipelines)
fsverity digest --hash-alg=sha256 --block-size=4096 /mnt/fsv/payload.bin

# After enable, measure must match digest for the same parameters
fsverity measure /mnt/fsv/payload.bin

Optional salt personalizes the tree (max 32 bytes). Pass the same salt to digest, sign, and enable:

SALT=$(openssl rand -hex 16)
fsverity digest --salt="$SALT" /mnt/fsv/payload.bin
sudo fsverity enable --salt="$SALT" /mnt/fsv/payload.bin
fsverity measure /mnt/fsv/payload.bin

5) Authentication options (pick deliberately)

Integrity without authentication only detects accidental corruption (and some classes of malicious disk firmware behavior on already-measured files). To stop substitution attacks you need a policy around the digest. The kernel documents four common patterns:

A) Trusted userspace (most flexible default)

Measure the digest and compare it to a value you already trust—an allow-list shipped with your orchestrator, a signature you verify with OpenSSL, or a digest embedded in a dm-verity-protected rootfs.

EXPECTED='sha256:YOUR_PINNED_DIGEST_HEX_HERE'
GOT=$(fsverity measure /mnt/fsv/payload.bin)
[ "$GOT" = "$EXPECTED /mnt/fsv/payload.bin" ] || [ "${GOT%% *}" = "${EXPECTED}" ] \
  || { echo "digest mismatch"; exit 1; }

A minimal deploy gate:

#!/bin/bash
set -euo pipefail
FILE=${1:?file}
PIN=${2:?expected-sha256-hex}
GOT=$(fsverity measure "$FILE" | awk '{print $1}')
# GOT looks like sha256:<hex>
[[ "$GOT" == "sha256:$PIN" ]] || { echo "reject: $GOT != sha256:$PIN"; exit 1; }
echo "ok: $FILE"

B) IMA appraisal

IMA can use fs-verity digests instead of full-file hashes and enforce signatures in security.ima under an IMA policy. Prefer this when you already run IMA appraisal fleet-wide. Setup is distribution-specific (policy loading, keyrings, appraisal modes) and larger than this article—see the kernel IMA docs once the digest workflow above is solid.

C) IPE (Integrity Policy Enforcement)

IPE can authorize access based on fsverity_digest or a verified built-in fsverity_signature. Useful when you want kernel-enforced execution/access policy tied to verity properties. Requires an IPE-enabled kernel and loaded policy.

D) Built-in PKCS#7 signatures (use with care)

CONFIG_FS_VERITY_BUILTIN_SIGNATURES adds a .fs-verity keyring and optional in-kernel signature check at open. The kernel docs warn hard: this is not a complete authentication policy by itself, keys are global, PKCS#7/X.509 parsing expands attack surface, Ed25519 is not available in-kernel for this path, and certificate validity times are not checked. Prefer userspace signatures unless you specifically need IPE integration with built-in signatures.

If you still need the built-in path for lab or IPE work:

# Generate a disposable key + cert (lab only)
openssl req -newkey rsa:4096 -nodes -keyout "$LAB/key.pem" -x509 -out "$LAB/cert.pem" \
  -days 365 -subj "/CN=fsverity-lab/"

# Sign → PKCS#7 DER detached signature of the formatted digest
fsverity sign --key="$LAB/key.pem" --cert="$LAB/cert.pem" \
  /mnt/fsv/payload.bin "$LAB/payload.sig"

# Enable with embedded signature (must happen in the same enable step)
sudo fsverity enable --signature="$LAB/payload.sig" /mnt/fsv/payload.bin

# Load certificate into the .fs-verity keyring (root). Exact keyctl plumbing
# varies; many operators use a small helper or keyctl padd asymmetric ...
# Consult current kernel docs for add_key() to ".fs-verity".

# Optional system-wide gate: refuse unsigned verity files on open
# sudo sysctl -w fs.verity.require_signatures=1

Built-in signatures cannot be rotated in place: changing the signature means recreating the file and re-enabling verity.

6) Day-2 operations operators forget

Copying kills verity. cp, rsync, tar, and most backup agents restore bytes without Merkle metadata. After restore you must fsverity enable again (and re-apply any signature policy). Design deploy pipelines to enable on the destination, not to ship “already verity” files as ordinary blobs.

ext4/f2fs storage layout. Metadata lives past i_size starting at the next 64K boundary. Userspace does not see it via normal reads; ls -l size is still the logical file size. On encrypted ext4 files, plaintext is what gets verified (digest stays meaningful across per-file keys).

btrfs. Supported since Linux 5.15 with metadata in btree items and a RO_COMPAT inode flag. Same userspace fsverity CLI.

Performance. Sequential reads are cheap: hash blocks cache well (with SHA-256/4K, ~127/128 data blocks reuse a cached lower hash block). Random reads pay more tree walks. Enabling on multi-gigabyte files can take noticeable CPU/I/O time once; measure afterward is O(1).

What fs-verity does not stop:

• Deleting the file and dropping a lookalike without verity (unless policy forbids it)
• Renaming paths out from under a careless consumer
• Compromised trusted userspace that “forgets” to call measure
• Writable corruption on non-verity files sitting next to protected ones

7) systemd oneshot pattern for a digest gate

Pin critical artifacts after deploy and fail the unit if digests drift:

# /etc/fsverity/pins.list  (format: absolute-path  sha256-hex)
# /var/lib/models/weights.bin  0123abc...

# /etc/systemd/system/fsverity-gate.service
[Unit]
Description=Verify fs-verity digests for pinned files
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/fsverity-gate.sh
# Nice + IOSchedulingClass=best-effort are optional on large fleets

[Install]
WantedBy=multi-user.target

# /usr/local/sbin/fsverity-gate.sh
#!/bin/bash
set -euo pipefail
while read -r path pin; do
  [[ -z "${path:-}" || "$path" =~ ^# ]] && continue
  got=$(fsverity measure "$path" | awk '{print $1}')
  [[ "$got" == "sha256:$pin" ]] || {
    echo "fsverity-gate: FAIL $path ($got)" >&2
    exit 1
  }
  echo "fsverity-gate: ok $path"
done < /etc/fsverity/pins.list

sudo chmod 0755 /usr/local/sbin/fsverity-gate.sh
sudo systemctl daemon-reload
sudo systemctl enable --now fsverity-gate.service

Pair this with a deploy hook that runs fsverity enable immediately after writing the artifact and updates pins.list atomically.

8) Clean rollback for the lab

sudo umount /mnt/fsv 2>/dev/null || true
if LOOP=$(losetup -j /var/tmp/fsverity-lab/disk.img | awk -F: 'NR==1{print $1}'); then
  sudo losetup -d "$LOOP"
fi
sudo rm -rf /var/tmp/fsverity-lab
# Remove any lab keys/sigs you created under $LAB

Production rollback is “replace the file from a trusted pipeline and re-enable verity,” not “clear a flag.” The verity inode flag is not toggleable via chattr; only FS_IOC_ENABLE_VERITY sets it, and clearing it is not supported—you replace the file.

Decision guide: fs-verity vs the neighbors

Need	Tool
Whole block device / image matches a published root hash	dm-verity + veritysetup / veritytab
Individual immutable files on a writable filesystem	fs-verity + fsverity
Writable volume detects silent sector corruption	dm-integrity / integritysetup
Confidentiality (who can read)	LUKS / dm-crypt (stacks with ext4 verity: plaintext verified)
Package/config drift inventories	debsums, AIDE, audit watches
Filesystem scrub of checksum trees	Btrfs scrub, ZFS scrub
One-shot full-file hash in a script	sha256sum (simple; no per-read re-verify; costly on huge sparse-access files)

fs-verity does not replace backups, Secure Boot, or transport signatures. It replaces the failure mode where a multi-gigabyte artifact is either fully re-hashed on every start or left unchecked between rare inventory scans.

Practical recommendations

• Lab on loop-backed ext4 first. Confirm enable / measure / write-block / corruption behavior before production paths.
• Pin fs-verity digests, not ad-hoc sha256sum outputs, when you integrate signing or allow-lists.
• Prefer userspace signature verification or IMA/IPE over built-in PKCS#7 unless you have a concrete IPE requirement and accept the limitations the kernel documents.
• Enable verity on the destination host as the last deploy step; do not assume copies preserve protection.
• Keep a clear policy for non-verity substitutes—measure alone is not a mandatory access control.
• For whole root filesystems and golden disk images, keep using dm-verity. For independently updated files on RW data volumes, use fs-verity.

Sources and references

• Linux kernel documentation: fs-verity: read-only file-based authenticity protection — https://www.kernel.org/doc/html/latest/filesystems/fsverity.html
• fsverity(1) (fsverity-utils) — Debian man page: https://manpages.debian.org/testing/fsverity/fsverity.1.en.html
• fsverity-utils project / README (examples and signing notes) — https://git.kernel.org/pub/scm/fs/fsverity/fsverity-utils.git
• ext4 feature enablement: tune2fs(8) / mkfs.ext4 -O verity (e2fsprogs)
• Related operator guides on this blog: dm-verity with veritysetup; dm-integrity with integritysetup (writable sector integrity)

Protect the files that must not drift—without freezing the entire disk to do it.