DEV Community

Cover image for Stop Overprovisioning Blindly: Practical LVM Thin Pools and Thin Snapshots on Linux
Lyra
Lyra

Posted on

Stop Overprovisioning Blindly: Practical LVM Thin Pools and Thin Snapshots on Linux

#linux #storage #devops #opensource

Stop Overprovisioning Blindly: Practical LVM Thin Pools and Thin Snapshots on Linux

Thick LVs are honest: if you ask for 500 GiB, LVM carves 500 GiB of physical extents now. Thin LVs are different. You declare a virtual size, and blocks only get allocated when something actually writes. That is powerful for VM images, container volumes, homelab scratch space, and dense snapshot trees—and dangerous if you treat “virtual free space” like real free space.

This guide is an operator walkthrough of LVM thin provisioning on modern Linux: create a thin pool, cut thin LVs, take space-efficient thin snapshots, wire dmeventd autoextend, choose full-pool behavior, monitor Data% / Meta%, and recover metadata with thin_check / thin_repair when the kernel sets needs_check.

It is not about Btrfs send/receive, mdadm scrubbing, multipath path failover, or filesystem-level snapshot tools. Those solve adjacent problems. Here the unit of work is the device-mapper thin-pool.

What you are actually building

From lvmthin(7) and the kernel thin-provisioning docs, the stack looks like this:

Piece Role
Thin pool LV Special LV you name but do not mount. Holds physical data + metadata.
_tdata Hidden data LV — physical storage for all thin volumes in the pool.
_tmeta Hidden metadata LV — mapping tables for dm-thin-pool.
Thin LV Virtual block device (-V / --virtualsize). Space comes from the pool on write.
Thin snapshot Another thin LV that initially shares blocks with its origin. No classic COW chunk LV.

Key operational facts from the kernel docs:

  • Data and metadata live on separate devices under the hood (LVM builds that for you).
  • Newly provisioned blocks are tracked in pool metadata; commits happen on FLUSH/FUA (or about once a second).
  • If data space runs out, the pool either queues or errors I/O (configurable).
  • If metadata space is exhausted or a metadata op fails, the pool can enter a bad state requiring offline check/repair (needs_check).
  • Recursive thin snapshots do not degrade like stacked classic COW snapshots; depth is fine, but fragmentation and pool fullness still matter.

Prerequisites

  • A volume group with free physical extents (lab example uses vg0).
  • Packages: lvm2 (provides lvcreate, dmeventd / lvm2-monitor), and thin provisioning tools:
    • Debian/Ubuntu: thin-provisioning-tools
    • Fedora/RHEL family: often device-mapper-persistent-data
  • Root shell, and a non-production VG if you are learning. 
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y lvm2 thin-provisioning-tools

# Fedora
sudo dnf install -y lvm2 device-mapper-persistent-data
Enter fullscreen mode Exit fullscreen mode

Confirm the monitor path that drives autoextend:

systemctl status lvm2-monitor.service
# or, depending on distro packaging:
systemctl status dm-event.socket dm-event.service
Enter fullscreen mode Exit fullscreen mode

1) Create a thin pool

Simplest path — let LVM size metadata automatically:

# 100 GiB physical data capacity in pool "tp0" on volume group vg0
sudo lvcreate --type thin-pool -n tp0 -L 100G vg0
Enter fullscreen mode Exit fullscreen mode

Inspect the hidden components:

sudo lvs -a vg0
# Expect something like:
#   tp0            twi-a-tz--  100.00g
#   [tp0_tdata]    Twi-ao----  100.00g
#   [tp0_tmeta]    ewi-ao----    N.NNm
#   [lvol0_pmspare] ...   # spare metadata copy for repair workflows
Enter fullscreen mode Exit fullscreen mode

Custom metadata size (recommended for dense snapshot hosts)

If you expect many thin volumes or heavy snapshot churn, do not rely on the tiny default metadata LV forever. Create data + metadata yourself, then convert:

sudo lvcreate -n tp0_data -L 100G vg0
sudo lvcreate -n tp0_meta -L 1G vg0   # size to taste; larger = more headroom
sudo lvconvert --type thin-pool --poolmetadata tp0_meta vg0/tp0_data
# Resulting pool keeps the data LV's name: vg0/tp0_data → rename if you want
sudo lvrename vg0/tp0_data tp0
Enter fullscreen mode Exit fullscreen mode

Kernel guidance for raw dm-thin (useful mental model): metadata size scales roughly with

48 * data_size / data_block_size, with a practical upper bound around 16 GiB of metadata. LVM’s --poolmetadatasize is the knob you want instead of hand-rolling dmsetup tables.

Chunk size and discards (set at pool creation)

From lvcreate(8):

  • --chunksize for thin pools: 64 KiB–1 GiB, multiple of 64 KiB. Default starts at 64 KiB and may scale with pool/metadata sizing.
  • Smaller chunks → better snapshot granularity, more metadata pressure.
  • Larger chunks → less metadata, coarser allocation (often fine for bulk thin provisioning). 
sudo lvcreate --type thin-pool -n tp0 -L 100G \
  --chunksize 256K \
  --discards passdown \
  vg0
Enter fullscreen mode Exit fullscreen mode

Discard modes (--discards):

  • passdown — reclaim in-pool and pass TRIM/UNMAP to the underlying device (good on SSDs that support it).
  • nopassdown — reclaim in-pool only.
  • ignore — ignore discards (pool never frees via TRIM).

Full-pool I/O policy 

sudo lvcreate --type thin-pool -n tp0 -L 100G --errorwhenfull y vg0
# or change later:
sudo lvchange --errorwhenfull y vg0/tp0
Enter fullscreen mode Exit fullscreen mode
  • --errorwhenfull y → fail I/O immediately when the pool cannot allocate (fail fast; apps see ENOSPC-style failures).
  • --errorwhenfull n (common default behavior path) → queue I/O for a while so autoextend can catch up. The kernel module parameter no_space_timeout (default 60s in upstream docs) bounds how long queuing lasts before errors.

For automated homelabs with autoextend and spare VG free space, queuing can be convenient. For multi-tenant or database hosts, error-when-full is often safer than silent stalls.

2) Create thin logical volumes

Virtual size is independent of pool size:

# 500 GiB *virtual* thin LV living in a 100 GiB physical pool
sudo lvcreate --type thin -n vm-disk0 -V 500G --thinpool tp0 vg0

# equivalent short form used a lot in docs:
sudo lvcreate -T vg0/tp0 -V 500G -n vm-disk0
Enter fullscreen mode Exit fullscreen mode

Format and mount like any block device:

sudo mkfs.ext4 -L vm-disk0 /dev/vg0/vm-disk0
sudo mkdir -p /mnt/vm-disk0
sudo mount /dev/vg0/vm-disk0 /mnt/vm-disk0
df -h /mnt/vm-disk0
Enter fullscreen mode Exit fullscreen mode

df shows the virtual size. Pool consumption is a different question — always check the pool:

sudo lvs -o name,lv_size,data_percent,metadata_percent,pool_lv,lv_attr vg0
Enter fullscreen mode Exit fullscreen mode

Combined create (pool + first thin LV in one shot):

sudo lvcreate --type thin -n app-data -V 200G \
  --thinpool tp0 -L 50G vg0
# Creates pool tp0 at 50G physical, then thin LV app-data at 200G virtual
Enter fullscreen mode Exit fullscreen mode

3) Thin snapshots (the feature people actually want)

Classic LVM COW snapshots need a fixed chunk store and get painful as they fill. Thin snapshots are thin LVs that share unchanged blocks:

# IMPORTANT: do not pass -L/--size here or you get a classic COW snapshot
sudo lvcreate --snapshot -n vm-disk0-snap1 vg0/vm-disk0

# Snapshot of a snapshot is fine
sudo lvcreate --snapshot -n vm-disk0-snap2 vg0/vm-disk0-snap1
Enter fullscreen mode Exit fullscreen mode

Activation skip on new thin snapshots

New thin snapshots often get the skip activation property (k in lvs attr, or lvs -o skip_activation). That means plain vgchange -ay / lvchange -ay will not activate them unless you force it:

sudo lvchange -ay -K vg0/vm-disk0-snap1
# clear the skip flag if this snapshot should activate normally:
sudo lvchange --setactivationskip n vg0/vm-disk0-snap1
Enter fullscreen mode Exit fullscreen mode

lvm.conf knob: activation/auto_set_activation_skip controls whether freshly created snapshots get the skip flag by default.

External origin (golden image pattern)

You can snapshot a read-only external LV (thick or thin-in-another-pool) into a thin pool: unwritten regions read through to the origin; writes land in the pool. Useful for many VMs sharing one golden disk image:

sudo lvchange --permission r vg0/golden-image
sudo lvcreate --snapshot -n guest1-disk --thinpool tp0 vg0/golden-image
Enter fullscreen mode Exit fullscreen mode

Do not write to the external origin while it is serving as an origin.

Merge a thin snapshot back 

# Must not be open/mounted if you want immediate merge; otherwise merge defers
sudo lvconvert --merge vg0/vm-disk0-snap1
Enter fullscreen mode Exit fullscreen mode

After merge, the origin takes the snapshot’s content and the snapshot LV is removed.

4) Monitor pool health like a hawk 

sudo lvs -o name,lv_size,data_percent,metadata_percent,seg_monitor vg0/tp0
sudo lvs -a vg0   # see _tdata / _tmeta
Enter fullscreen mode Exit fullscreen mode

Rules of thumb:

  • Extend before Data% or Meta% approaches 100%.
  • Metadata full is worse than data full: metadata exhaustion can force repair before the pool is fully usable again.
  • Removing thin LVs does not always free as much as you hope (shared snapshot blocks + fragmentation), per lvmthin(7).

Optional live dm status (advanced):

sudo dmsetup status vg0-tp0-tpool 2>/dev/null || sudo dmsetup status | grep thin
Enter fullscreen mode Exit fullscreen mode

Kernel status fields include used/total data and metadata blocks, ro|rw|out_of_data_space, error_if_no_space|queue_if_no_space, and needs_check|-.

5) Extend the pool (manual and automatic)

Manual 

# grow data (metadata may grow automatically relative to new size)
sudo lvextend -L +50G vg0/tp0

# grow metadata explicitly
sudo lvextend --poolmetadatasize +512M vg0/tp0

# or target hidden LVs directly
sudo lvextend -L +50G vg0/tp0_tdata
sudo lvextend -L +512M vg0/tp0_tmeta
Enter fullscreen mode Exit fullscreen mode

You need free extents in the VG. Thin does not invent disks.

Automatic via dmeventd

lvmthin(7) + dmeventd(8):

  1. Ensure the thin pool is monitored: 
   sudo lvs -o+seg_monitor vg0/tp0
   sudo lvchange --monitor y vg0/tp0
Enter fullscreen mode Exit fullscreen mode
  1. Configure thresholds in /etc/lvm/lvm.conf (or a drop-in profile): 
activation {
    # Extend when usage reaches this percent (minimum 50; 100 disables)
    thin_pool_autoextend_threshold = 70
    # Grow by this percent of current size
    thin_pool_autoextend_percent = 20
}
Enter fullscreen mode Exit fullscreen mode
  1. Keep free space in the VG for growth, and keep lvm2-monitor / dmeventd running.

What dmeventd does (thin plugin):

  • Warns in syslog as the pool crosses ~80/85/90/95% fullness.
  • Above ~50%, periodically runs the configured thin command (default: internal lvextend --use-policies).
  • Retries with backoff if extend fails (up to ~42 minutes per the man page).
  • Child environment includes DMEVENTD_THIN_POOL_DATA, DMEVENTD_THIN_POOL_METADATA, and LVM_RUN_BY_DMEVENTD=1.

Verify monitoring after activation:

sudo lvs -o name,data_percent,metadata_percent,seg_monitor vg0
journalctl -u lvm2-monitor -u dm-event --since "1 hour ago"
Enter fullscreen mode Exit fullscreen mode

6) Discard / TRIM so deleted data can return to the pool

Without discards, deleting files inside a guest FS often does not free thin-pool blocks.

# mount with discard if your workload tolerates inline TRIM, or run periodic:
sudo fstrim -v /mnt/vm-disk0
Enter fullscreen mode Exit fullscreen mode

Pair this with pool --discards passdown or nopassdown so the thin target actually processes discards. Periodic fstrim.timer on hosts that store thin-backed filesystems is a good habit (same idea as SSD maintenance, different layer).

7) When things go wrong: needs_check, thin_check, thin_repair

From the kernel thin-provisioning guide:

  • Metadata failures can set needs_check on the pool.
  • While flagged, expect impaired operation; repair before relying on the pool again.
  • After serious metadata problems, run filesystem consistency checks on upper layers too — completions may have been acknowledged before the failure.

Check metadata (offline)

Deactivate thin LVs and the pool first (all users of the pool must be down):

sudo umount /mnt/vm-disk0 || true
sudo lvchange -an vg0/vm-disk0
sudo lvchange -an vg0/tp0

# Check the metadata LV (name may be tp0_tmeta)
sudo thin_check /dev/mapper/vg0-tp0_tmeta
Enter fullscreen mode Exit fullscreen mode

Useful thin_check options (thin_check(8) from thin-provisioning-tools):

Option Meaning
--super-block-only Quick superblock-only pass
--skip-mappings Skip bulk mapping checks
--clear-needs-check-flag Clear kernel needs_check only if check succeeded
--metadata-snapshot / -m Check a held metadata snapshot (can be used on live metadata with limits)
--auto-repair Fix trivial issues (e.g. metadata leaks) 
sudo thin_check --clear-needs-check-flag /dev/mapper/vg0-tp0_tmeta
Enter fullscreen mode Exit fullscreen mode

If check fails, use thin_repair (same package family), then thin_check again. Do not clear needs_check on a failed check and hope.

Live metadata inspection

The thin-pool target supports reserving a metadata snapshot for userspace. LVM/thin tools integrate with that path; thin_check -m is the documented way to examine a metadata snap without taking the pool fully offline. Prefer scheduled offline checks on spare capacity when you can.

8) Loop-device lab (safe practice)

No spare disk? Use sparse files:

truncate -s 32G /var/tmp/lvm-thin-lab.img
sudo losetup -f --show /var/tmp/lvm-thin-lab.img
# suppose it printed /dev/loop5
sudo pvcreate /dev/loop5
sudo vgcreate thinlab /dev/loop5

sudo lvcreate --type thin-pool -n tp0 -L 8G thinlab
sudo lvcreate -T thinlab/tp0 -V 20G -n vol0
sudo mkfs.xfs /dev/thinlab/vol0
sudo mkdir -p /mnt/thinlab
sudo mount /dev/thinlab/vol0 /mnt/thinlab

# consume some real pool space
dd if=/dev/urandom of=/mnt/thinlab/blob bs=1M count=1024 status=progress
sudo lvs -o name,lv_size,data_percent,metadata_percent thinlab

sudo lvcreate --snapshot -n vol0-s1 thinlab/vol0
sudo lvs -a thinlab
Enter fullscreen mode Exit fullscreen mode

Cleanup:

sudo umount /mnt/thinlab
sudo lvremove -y thinlab/vol0-s1 thinlab/vol0 thinlab/tp0
sudo vgremove -y thinlab
sudo pvremove /dev/loop5
sudo losetup -d /dev/loop5
rm -f /var/tmp/lvm-thin-lab.img
Enter fullscreen mode Exit fullscreen mode

9) Operational checklist

  1. Pool size is the real capacity. Sum of thin virtual sizes can exceed it; plan for actual written bytes + snapshot delta.
  2. Watch both Data% and Meta%. Metadata surprises hurt more.
  3. Leave VG free space if you enable autoextend.
  4. Run lvm2-monitor / dmeventd and confirm seg_monitor on the pool.
  5. Pick --errorwhenfull deliberately for each host class.
  6. TRIM/fstrim so deletes can return blocks to the pool.
  7. Thin snapshots: no -L, learn activation skip (-K), prune old snaps before the pool fills with unique blocks.
  8. Backups still matter. Thin snapshots are rollback/clone tools on the same pool/disk failure domain—not off-box backups (contrast with btrfs send/receive or real backup software).
  9. Raid the components if the data matters. lvmthin(7) shows building RAID LVs for data/metadata, then lvconvert --type thin-pool.
  10. After metadata repair, fsck/xfs_repair the filesystems that lived on affected thin LVs.

10) Rollback / removal 

# remove snaps first, then thin LVs, then pool
sudo lvremove -y vg0/vm-disk0-snap2 vg0/vm-disk0-snap1
sudo umount /mnt/vm-disk0
sudo lvremove -y vg0/vm-disk0
sudo lvremove -y vg0/tp0
Enter fullscreen mode Exit fullscreen mode

Disable autoextend policy if you are decommissioning monitoring expectations:

# threshold 100 = disabled per lvmthin(7)
sudo sed -n 's/.*thin_pool_autoextend.*/&/p' /etc/lvm/lvm.conf
Enter fullscreen mode Exit fullscreen mode

Boundaries (what this post is not)

  • Not Btrfs snapshots or btrfs send/receive off-box replication.
  • Not mdadm array monitoring/scrub.
  • Not DM-Multipath path redundancy.
  • Not filesystem quotas or project quotas.
  • Not a license to oversubscribe pools without monitoring — thin is an allocation strategy, not free capacity.

References

Thin provisioning pays off when virtual sprawl is real but simultaneous dirty data is not—and when you treat pool Data% / Meta% as first-class capacity signals. Build the pool deliberately, monitor it, autoextend into spare VG space, and keep thin_check in your recovery muscle memory. That is the difference between “clever storage” and a very quiet, very full disk at 3 a.m.

Top comments (0)