BtrFS, ZFS, and More

Butter Filesystem. Hold the toast.

I've started experimenting with BtrFS which aims to provide an "advanced and modern filesystem" (heavily compared to ZFS) on Linux. With my new workstation I've started using BtrFS for my home directories (/home) and my build directories (/mnt/slackbuilds) to gain exposure to the filesystem and compare it to ZFS and EXT4 on LVM (all of my other data, including my root disk is on EXT4 on LVM).

I have used ZFS heavily in the past, and using BtrFS is significantly different as many of the fundamental concepts are different. BtrFS has no concept of "pools" or "volume groups" -- instead there are "volumes." BtrFS has no concept of "datasets" or "logical volumes" -- instead there are "subvolumes".

Here's a comparison between ZFS, BtrFS, and EXT4 on LVM:

Comparison

	ZFS	BtrFS	EXT4 and LVM
Commands Involved	zpool, zfs	mkfs.btrfs, btrfs	pvcreate, vgcreate, lvcreate, mkfs.ext4
Pool of disks	"zpool"	"volume"	"volume group"
Mountable unit	"dataset"	"volume" and "subvolume"	"logical volume"
License	CDDL	GPL	GPL
Can be Boot filesystem	Yes	Yes (grub 2.00)	No
Can be Root filesystem	Yes	Yes	Yes
Can provide swapspace	Yes (zvols)	No	Yes (lvm)
OSes with Implementations	Solaris, OpenSolaris, Nexenta, FreeBSD, Mac OS X, Linux	Linux	Linux
Stability	Stable	Stable	Stable
CLI-System Integration [1]	Strong	Weak	Mild
Grow Online	Yes	Yes	Only when there are no snapshots
Shrink Pool	No	Online	Online
Shrink Filesystem	Online (reduce quota)	Online	Offline
Replace Disk (without parity)	Yes (must be compatible size disk)	Yes	Yes (copies space allocated on disk)
Filesystem Level Storage Pooling	Yes	Yes	No
Re-balance	No	Yes	Can be done manually (pvmove)
Checksumming	Yes	Yes	No
Autocorrect Checksum Errors	Yes	???	No
Compression	Yes	Yes	No
De-duplication	Yes (only synchronous)	Yes (only asynchronous)	No
Ditto Blocks	Yes	???	No
Tiered Caching	Yes	No	No
Writable Snapshots	Yes (clone)	Yes	Yes
Copy-on-Write	Fast, space-efficient	Fast, space-efficient	Slow, requires pre-allocating an LV
Redundancy	Mirroring and Parity (x1, x2, x3)	Mirroring	Mirroring, though the PVs can be RAID devices
Maximum Volume Size	16 Exabytes	16 Exabytes	1 Exabyte
Maximum File Size	16 Exabytes	16 Exabytes	16 Terabytes
Maximum Number of Snapshots	Unlimited	Unlimited	Effectively 32

[1] For lack of a better term -- how well the command line interface integrates with the system as a whole, this might be subjective.

For a more complete, but less focused comparison see Wikipedia's Comparison of Filesystems

The Rosetta Stone

1. Task: Create a pool of storage from disks /dev/A, /dev/B, and /dev/C (striped or linear concat)
   1. Using ZFS:
      1. # zpool create TESTPOOL A B C
   2. Using BtrFS:
      1. # mkfs.btrfs -d raid0 /dev/A /dev/B /dev/C
   3. Using EXT4 on LVM:
      1. # pvcreate /dev/A /dev/B /dev/C
      2. # vgcreate TESTPOOL /dev/A /dev/B /dev/C
2. Task: Make storage from pool available to system
   1. Using ZFS:
      1. # zfs set mountpoint=/data TESTPOOL
   2. Using BtrFS:
      1. # mkdir /data
      2. # mount -t btrfs /dev/A /data
      3. Update /etc/fstab
   3. Using EXT4 on LVM:
      1. # mkdir /data
      2. # lvcreate -L _SizeOfVolume_ -n DATA TESTPOOL
      3. # mkfs -t ext4 /dev/TESTPOOL/DATA
      4. # mount /dev/TESTPOOL/DATA /data
      5. Update /etc/fstab
3. Task: Add an additional disk to the pool
   1. Using ZFS:
      1. # zfs add TESTPOOL D
   2. Using BtrFS:
      1. # btrfs device add /dev/D /data
      2. # btrfs filesystem balance /data
   3. Using EXT4 on LVM:
      1. # pvcreate /dev/D
      2. # vgextend TESTPOOL /dev/D
4. Task: Add additional space to a filesystem
   1. Using ZFS:
      1. No action needed
   2. Using BtrFS:
      1. No action needed
   3. Using EXT4 on LVM:
      1. # lvextend -L _SizeToIncreaseTo_ /dev/TESTPOOL/DATA
      2. # resize2fs /dev/TESTPOOL/DATA
5. Task: Remove a disk from the pool
   1. Using ZFS: N/A
   2. Using BtrFS:
      1. # btrfs device delete /dev/A /data
      2. # btrfs filesystem balance /data
   3. Using EXT4 on LVM:
      1. # pvmove /dev/A
      2. # vgreduce TESTPOOL /dev/A
6. Task: Replace operational disk
   1. Using ZFS:
      1. # zfs replace TESTPOOL A D
   2. Using BtrFS:
      1. # btrfs device add /dev/D /data
      2. # btrfs device delete /dev/A /data
      3. # btrfs filesystem balance /data
   3. Using EXT4 on LVM:
      1. # pvcreate /dev/D
      2. # vgextend TESTPOOl /dev/D
      3. # pvmove TESTPOOL /dev/A /dev/D
      4. # vgreduce TESTPOOL /dev/A
7. Task: Take a snapshot of a filesystem
   1. Using ZFS:
      1. # zfs snapshot TESTPOOL@snapshot1
   2. Using BtrFS:
      1. # btrfs subvolume snapshot /data /data/snapshot1
   3. Using EXT4 on LVM:
      1. # lvcreate -s /dev/TESTPOOL/DATA -L _SizeToAllocate_ -n snapshot1
8. Task: Rollback a snapshot
   1. Using ZFS:
      1. # zfs rollback TESTPOOL@snapshot1
   2. Using BtrFS:
      1. Not sure...
   3. Using EXT4 on LVM:
      1. # lvconvert --merge /dev/TESTPOOL/snapshot1
9. Task: Delete a snapshot
   1. Using ZFS:
      1. # zfs destroy TESTPOOL@snapshot1
   2. Using BtrFS:
      1. # btrfs subvolume delete /data/snapshot1
   3. Using EXT4 on LVM:
      1. # lvremove /dev/TESTPOOL/snapshot1

Comments

[Jürgen Botz]

For Task 8: Rollback a snapshot, you say "not sure" for btrfs. The answer is that snapshots are completely equivalent to subvolumes, so you just start using (and mounting, if appropriate) the snapshot instead of the "original" subvolume (and delete that one if you want).

[Jürgen Botz]

Oh, and you can put a swapfile on a btrfs filesystem nowadays, as well as any kind of other file that is meant to serve as storage for a block device (such as VM images, etc.). The key is to set the NoCOW attribute on the subvolume or directory those files live in; that done, today's btrfs is no worse for this usecase than any other filesystem.

[Jürgen Botz]

I'm really curious what you mean by CLI integration... the footnote doesn't help. Please explain your "subjective" understanding of this.

[Gordon]

GRUB2 has been able to boot to LVM for a long time.