A sub-100-dollar ZFS data tier from a used office PC

My cluster's newer nodes are small-form-factor office machines, the kind that companies lease for three years and then sell by the pallet. They came with one NVMe boot drive and an empty 2.5 inch SATA bay. Filling that empty bay turned each node into something with a real storage tier, for less than the price of a restaurant dinner.

The recipe is short: a used enterprise SATA SSD in the spare bay, ZFS on top of it, and a handful of pool options that people usually copy from a forum thread without knowing why. The options are the interesting part.

The drive: used enterprise, not new consumer

The drive I put in each node is an Intel S3520 class enterprise SSD. Used, off the second-hand market, for roughly the cost of a new budget consumer drive.

The case for used enterprise over new consumer at the same price is straightforward. Enterprise SATA drives have power-loss protection capacitors, which matter more than any other single feature when the filesystem on top is ZFS and the workload is VM disks. They publish honest endurance ratings, and a lightly used one typically arrives with a few percent of its rated writes consumed. A new consumer drive at the same price has none of the capacitors and a fraction of the endurance.

Before trusting any second-hand drive, read its own account of its history:

smartctl -i -A /dev/sdX | grep -E 'Model|Serial|Reallocated|Power_On_Hours|Wear'

What you want: zero reallocated sectors, wear levelling well under 80 percent used, and a power-on-hours figure consistent with the seller's story. A drive that fails any of those is a drive someone else should own.

The pool: every option earns its place

One drive, one pool. Creation looks like this:

zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -O dnodesize=auto \
  -O normalization=formD \
  tank /dev/disk/by-id/ata-MODEL_SERIAL

Two things before the option list. First, the pool is built on the /dev/disk/by-id/ path, never /dev/sda. Device letters are assigned by enumeration order and change when you add a disk; by-id paths are welded to the drive's serial number and survive anything short of the drive dying. Second, the pool is named tank on every node in the cluster, identically. That naming choice looks cosmetic. It is what makes replication and migration between nodes work as single commands. It deserves its own write-up, and it gets one on Thursday.

The options, and what each one buys:

• ashift=12 tells ZFS the drive writes in 4K blocks. SSDs lie about their sector size for compatibility; taking the lie at face value costs write amplification forever, because ashift is fixed at creation.
• compression=lz4 is close to free on any modern CPU and typically returns 1.5 to 2 times effective space on VM workloads, which are full of text, logs, and zero pages. There is no realistic workload where turning it off wins.
• atime=off stops the filesystem updating a timestamp on every read. Without it, reading a file costs a write. Nothing on a VM data tier cares about access times.
• xattr=sa stores extended attributes inside the inode instead of in hidden directories, one lookup instead of two. Containers touch xattrs constantly.
• acltype=posixacl gives containers the ACL semantics they expect. Some distributions inside containers misbehave subtly without it, and subtle misbehaviour is the worst kind.
• dnodesize=auto lets metadata-heavy datasets grow their dnodes instead of spilling attributes elsewhere. Pairs with xattr=sa.
• normalization=formD makes the filesystem treat Unicode names consistently, so you cannot end up with two files whose names look identical but differ in bytes. You only hit this once, but you remember it.

Then register the pool with the hypervisor:

pvesm add zfspool tank-local --pool tank --content rootdir,images --sparse 1

The storage ID tank-local and the pool name tank are deliberately different names for different layers. --sparse 1 makes volumes thin, so a 32 GB container that holds 4 GB of data consumes 4 GB.

What this tier is for, and what it is not for

With one of these in each node, containers and VMs live on local ZFS with lz4 behind them, snapshots are instant, and scheduled replication can keep a copy of a guest's disk on the other node a few minutes stale. That replication is the foundation the cluster's failover story is built on, and both of the next two posts in this series stand on it.

What a single-drive pool does not give you is redundancy. One drive is one failure domain; ZFS will detect corruption but has no second copy to heal from. My answer to that is nightly backups to a dedicated backup server plus the cross-node replication, which together bound any loss to a small window. A mirror would be better and the bay count says no. So the redundancy lives in the backup schedule instead of in the second bay I do not have.

Total cost per node: one used enterprise SSD, somewhere between 30 and 50 dollars depending on the week. Capability added: snapshots, compression, replication, and a place for guests to live that is not the boot drive. It is the best price-to-capability ratio anywhere in my rack.

The full node build procedure, from bare hardware to cluster member, lives in my runbooks. This drive-and-pool step is section five of it, and it is the section I would keep if I had to throw away the rest.