Mark Caudill

Posted on Feb 27, 2019 • Updated on Jun 26, 2019 • Originally published at mrkc.me

Leveraging LVM Caching

#lvm #linux

Note: I'm migrating relevant blog posts I've written over to dev.to. This is the first one.

I recently implemented LVM caching based largely on Richard WM Jones' post and thought I'd document my own experience here.

I started with an SSD containing /, /home, and swap, two 2 TB HDDs (sdb, and sdc) and an additional SSD (sdd).

[root@bender ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0 447.1G  0 disk
├─sda1            8:1    0   500M  0 part /boot
└─sda2            8:2    0 446.7G  0 part
  ├─fedora-root 253:0    0    50G  0 lvm  /
  ├─fedora-swap 253:1    0  15.7G  0 lvm  [SWAP]
  └─fedora-home 253:2    0   381G  0 lvm  /home
sdb               8:16   0   1.8T  0 disk
sdc               8:32   0   1.8T  0 disk
sdd               8:48   0 119.2G  0 disk

I then used fdisk to create a Linux partition on each HDD and a Linux LVM partition on the SSD. Though, these partition types don't actually matter much for our purposes beyond looking nice.

[root@bender ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0 447.1G  0 disk
├─sda1            8:1    0   500M  0 part /boot
└─sda2            8:2    0 446.7G  0 part
  ├─fedora-root 253:0    0    50G  0 lvm  /
  ├─fedora-swap 253:1    0  15.7G  0 lvm  [SWAP]
  └─fedora-home 253:2    0   381G  0 lvm  /home
sdb               8:16   0   1.8T  0 disk
└─sdb1            8:17   0   1.8T  0 part
sdc               8:32   0   1.8T  0 disk
└─sdc1            8:33   0   1.8T  0 part
sdd               8:48   0 119.2G  0 disk
└─sdd1            8:49   0 119.2G  0 part

I then created a RAID 1 set using the two HDD partitions.

[root@bender ~]# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@bender ~]# echo 'DEVICE /dev/sdb1 /dev/sdc1' > /etc/mdadm.conf
[root@bender ~]# mdadm --detail --scan >> /etc/mdadm.conf
[root@bender ~]# cat /etc/mdadm.conf
DEVICE /dev/sdb1 /dev/sdc1
ARRAY /dev/md0 metadata=1.2 name=bender:0 UUID=2f9fd7a2:cf0d4dd8:579ec7ce:073d0886

Then I created the LVM volumes and an ext4 filesystem on the RAIDed volume.

[root@bender ~]# pvcreate /dev/md0 /dev/sdd1
  Physical volume "/dev/md0" successfully created
  Physical volume "/dev/sdd1" successfully created
[root@bender ~]# vgcreate data /dev/md0
  Volume group "data" successfully created
[root@bender ~]# lvcreate -n vol0 -L 1.5T data /dev/md0
  Logical volume "vol0" created.
[root@bender ~]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vol0 data   -wi-a-----   1.50t
  home fedora -wi-ao---- 380.95g
  root fedora -wi-ao----  50.00g
  swap fedora -wi-ao----  15.69g
[root@bender ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/md0   data   lvm2 a--    1.82t 326.89g
  /dev/sda2  fedora lvm2 a--  446.64g   4.00m
  /dev/sdd1         lvm2 ---  119.24g 119.24g
[root@bender ~]# mkfs.ext4 /dev/data/vol0
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 402653184 4k blocks and 100663296 inodes
Filesystem UUID: 51e8796a-5090-45ce-ae62-4944f2eb9132
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Now for the caching configuration. It's important to specify the actual partition (sdd1) in the following commands to ensure that the cache gets created on that device and not arbitrarily within the volume group.

[root@bender ~]# vgextend data /dev/sdd1
  Volume group "data" successfully extended
[root@bender ~]# lvcreate -L 1G -n vol0_cache_meta data /dev/sdd1
  Logical volume "vol0_cache_meta" created.
[root@bender ~]# lvcreate -L 115G -n vol0_cache data /dev/sdd1
  Logical volume "vol0_cache" created.
[root@bender ~]# lvs
  LV              VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vol0            data   -wi-a-----   1.50t
  vol0_cache      data   -wi-a----- 115.00g
  vol0_cache_meta data   -wi-a-----   1.00g
  home            fedora -wi-ao---- 380.95g
  root            fedora -wi-ao----  50.00g
  swap            fedora -wi-ao----  15.69g
[root@bender ~]# lvconvert --type cache-pool --poolmetadata data/vol0_cache_meta data/vol0_cache
  WARNING: Converting logical volume data/vol0_cache and data/vol0_cache_meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert data/vol0_cache and data/vol0_cache_meta? [y/n]: y
  Converted data/vol0_cache to cache pool.
[root@bender ~]# lvconvert --type cache --cachepool data/vol0_cache data/vol0
  Logical volume data/vol0 is now cached.
[root@bender ~]# lvs
  LV   VG     Attr       LSize   Pool         Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  vol0 data   Cwi-a-C---   1.50t [vol0_cache] [vol0_corig] 0.00   1.43            100.00
  home fedora -wi-ao---- 380.95g
  root fedora -wi-ao----  50.00g
  swap fedora -wi-ao----  15.69g

Then I mounted it at /var/data.

[root@bender ~]# echo '/dev/mapper/data-vol0 /var/data                 ext4    defaults        1 2' >> /etc/fstab
[root@bender ~]# mount -a
[root@bender ~]# df -h /var/data
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/data-vol0  1.5T   69M  1.5T   1% /var/data