Encrypt an Existing Linux Installation Online with the Magic of LVM

So you have an existing Linux installation and your disks aren’t encrypted? You want to encrypt the entire disk including your root partition, and you’d like to do it without taking the server offline. If the server already uses LVM, which is the default for many popular enterprise distros, then you’re in luck.

Because of the awesomeness of LVM, an existing installation can be encrypted with LUKS completely online. This does require a spare disk of equal or greater size to the existing disk. In a virtual environment, this can be achieved by adding an extra virtual disk to the VM. For a desktop or workstation, this can be achieved with an external drive.

What we want to do is:

• Add the spare disk to the volume group.

• Move the data over to the spare disk.

• Remove the original disk from the volume group.

• Encrypt the original disk with LUKS.

• Add it back to the volume group and move the data back over.

• Remove the spare disk from the volume group.

To get started:

• Get the name of the existing physical volume with sudo pvdisplay.

• Get the name of the existing volume group with sudo vgdisplay.

• Determine the name of the spare disk with sudo fdisk -l.

I’m encrypting a VM with a virtual disk (/dev/sda). It has two partitions: a small /boot partition (/dev/sda1) and an LVM physical volume (/dev/sda2). The name of the volume group used in this example is vgpool. I added a second virtual disk. It is unpartitioned and called /dev/sdb. Substitute the values for your environment.

I highly recommend taking a snapshot or backup before continuing. Any time you are messing with storage, there is the possibility for things to go very wrong.

1. Add the spare disk to the volume group.

   $ sudo pvcreate /dev/sdb
     Physical volume "/dev/sdb" successfully created.
   $ sudo vgextend vgpool /dev/sdb
     Volume group "vgpool" successfully extended

2. Move the physical extents from the current physical volume to the spare physical volume. This will take a long time. It will periodically print progress to the screen.

   $ sudo pvmove /dev/sda2 /dev/sdb

3. Remove the original physical volume from the volume group.

   $ sudo vgreduce vgpool /dev/sda2
     Removed "/dev/sda2" from volume group "vgpool"
   $ sudo pvremove /dev/sda2
    Labels on physical volume "/dev/sda2" successfully wiped.

4. Wipe the original disk. Because the disk already had data on it, it is important to securely erase the disk to prevent file recovery of unencrypted data.

   $ sudo yum install cryptsetup
   $ sudo cryptsetup open --type plain -d /dev/urandom /dev/sda2 to_be_wiped
   $ sudo dd if=/dev/zero of=/dev/mapper/to_be_wiped bs=1M status=progress
   dd: error writing '/dev/mapper/to_be_wiped': No space left on device
   $ sudo cryptsetup close to_be_wiped

   
   
   

   The default cipher for plain dm-crypt encryption is aes-cbc. It might be worth changing the cipher to aes-xts, as it might be significantly faster (verify with cryptsetup benchmark). $ sudo cryptsetup open --type plain -d /dev/urandom --cipher aes-xts-plain64 /dev/sda2 to_be_wiped

   Use of /dev/urandom with dd is not required as the encryption cipher is used for randomness.

5. Encrypt the original disk with LUKS using a detached LUKS header stored on the /boot partition.

   The cryptsetup default is to store the LUKS header at the beginning of the partition or disk. If the LUKS header is stored on the original, the volume group will no longer fit when we go to move it back to the original disk from the spare disk, and you’ll get an error like this:

   $ sudo pvmove /dev/sdb /dev/mapper/lvmcrypt
     Insufficient free space: 261887 extents needed, but only 261883 available
     Unable to allocate mirror extents for ol_ods/pvmove0.
     Failed to convert pvmove LV to mirrored.

   LUKS has the ability to use a detached header. Since you probably already have an unencrypted /boot, the header can be stored there.

   $ sudo cryptsetup -y luksFormat /dev/sda2 --header /boot/luksheader.img
   
   WARNING!
   ========
   Header file does not exist, do you want to create it?
   
   Are you sure? (Type 'yes' in capital letters): YES
   Enter passphrase for /boot/luksheader.img:
   Verify passphrase:
   $ sudo cryptsetup luksOpen /dev/sda2 lvmcrypt --header /boot/luksheader.img
   Enter passphrase for /dev/sda2:

6. Add the now encrypted disk back to the volume group.

   $ sudo pvcreate /dev/mapper/lvmcrypt
     Physical volume "/dev/mapper/lvmcrypt" successfully created.
   $ sudo vgextend vgpool /dev/mapper/lvmcrypt
     Volume group "vgpool" successfully extended

7. Move the physical extents from the spare physical volume to the encrypted physical volume. Like before, this will take a long time.

   $ sudo pvmove /dev/sdb /dev/mapper/lvmcrypt

8. Remove the spare physical volume from the volume group.

   $ sudo vgreduce vgpool /dev/sdb
     Removed "/dev/sdb" from volume group "vgpool"
   $ sudo pvremove /dev/sdb
     Labels on physical volume "/dev/sdb" successfully wiped.

   The data is now entirely on the original physical volume but this time encrypted. You can now delete the spare virtual disk in the case of a VM or remove the external drive.

Configure Initramfs

We’ve accomplished what we set out to do: we’ve encrypted a live system without requiring a reboot. But whenever you do reboot, you want your system to come back up. Since we encrypted the root partition, the decryption needs to happen in the initramfs. There are various tools to regenerate the initramfs depending on your distro. Red Hat-derived distros like Oracle Linux usually use Dracut.

Since we used a detached header, configuring the initramfs is a little more complicated. “A LUKS volume with a detached header, at least using udev/blkid, does not have a UUID.” However, unlike in the linked article, there is a PARTUUID since partitioning is used on the disk (assuming /dev/sda1 is /boot and /dev/sda2 is LVM because the Red Hat installer requires /boot on a separate partition when using LVM).

1. Get the PARTUUID of the LUKS device with (sudo blkid /dev/sda2).

2. Add it to /etc/crypttab in the form:

   volume-name encrypted-device key-file options

   • volume-name is the device mapper name (e.g. if the volume-name is lvmcrypt, it will mount the LUKS device as /dev/mapper/lvmcrypt).

   • encrypted-device is the specification of the encrypted device via UUID=, PARTUUID=, or ID=.

   • key-file is the absolute path to a file with the encryption key. If there is no key file, use none.

   • options is a comma-delimited list of options.

   Since we used a detached header, we need to include the header option and force to force the inclusion of the entry in the initramfs (the only place I found where this option is documented is in the pull request that added it), so our crypttab will look something like:

   lvmcrypt PARTUUID=<PARTUUID-identifier> none header=/boot/luksheader.img,force

   where <PARTUUID-identifer> is the PARTUUID of the LUKS device.

   If this is for a laptop or workstation where you are OK with being prompted for a password during boot, you can skip to the last step and regenerate initramfs. The following steps will set up Network-Bound Disk Encryption (NBDE) with Clevis.

3. Bind LUKS volume to Clevis.

   $ sudo yum install clevis clevis-luks clevis-dracut
   $ sudo clevis luks bind -d /dev/sda2 tang '{"url": "http://tang.example.com"}'
   /dev/sda2 is not a LUKS device!

   Unfortunately, Clevis does not currently support detached headers. Clevis mostly consists of shell scripts, so this can be easily hacked. These modifications will have to be remerged whenever updating Clevis, so it would be ideal to get the modifications necessary to support detached headers merged upstream like the author of this post did to get detached headers supported by Dracut. I’m going to start by opening an issue on the Clevis GitHub to see if they are open to a pull request adding this and update this with the results.

   • Navigate to /usr/bin and make a backup copy of clevis-luks-bind.

     $ cd /usr/bin
     $ sudo cp clevis-luks-bind clevis-luks-bind.original

   • Edit clevis-luks-bind. Locate where the command line arguments are parsed with getopts and make the following additions:

     while getopts ":hfyd:s:k:t:o:" o; do (1)
         case "$o" in
         f) FRC+=(-f);;
         d) DEV="$OPTARG";;
         s) SLT="$OPTARG";;
         k) KEY="$OPTARG";;
         t) TOKEN_ID="$OPTARG";;
         o) HEADER="$OPTARG";; (2)
         y) FRC+=(-f)
            YES+=(-y);;
         *) usage;;
         esac
     done
     
     if [ ! -z "$HEADER" ]; then (3)
         LUKSOPTS="--header=$HEADER"
     fi
     
     # Alias cryptsetup to insert options:
     cryptsetup() {
         /usr/sbin/cryptsetup "$LUKSOPTS" "$@"
     }

     1	Modify the getopts spec adding o: to the end. Since h for header wasn’t available, and d for detached is already used for device, I chose o for option.
     2	Add an entry to the case statement to set a $HEADER variable to the value of the o argument.
     3	Add this and the following lines. The syntax for the cryptsetup utility is cryptsetup <options> <action> <action args>. Clevis uses cryptsetup to bind to a LUKS volume. What we want to do is insert the header option into <options> whenever Clevis calls cryptsetup. A quick-and-dirty hack to accomplish this with minimal changes to the code (especially since we’ll have to remerge our changes whenever updating Clevis) is to alias cryptsetup with a function that calls the absolute path to the cryptsetup executable with the header option and then appends the rest of the arguments Clevis originally called cryptsetup with.

   • Now you can include the -o argument we just added to Clevis to bind it to the LUKS volume.

     $ sudo clevis luks bind -o /boot/luksheader.img -d /dev/sda2 tang '{"url": "http://tang.example.com"}'
     The advertisement contains the following signing keys:
     
     -Y7MtTJlw8rGGVgXRBhFtumJIqk
     
     Do you wish to trust these keys? [ynYN] y
     You are about to initialize a LUKS device for metadata storage.
     Attempting to initialize it may result in data loss if data was
     already written into the LUKS header gap in a different format.
     A backup is advised before initialization is performed.
     
     Do you wish to initialize /dev/sda2? [yn] y
     Enter existing LUKS password:

   • Modify /usr/libexec/clevis-luks-askpass to support detached headers. Modifying Clevis to bind to a LUKS volume with a deatched header is only one half of the equation. The Clevis Dracut module also needs to be able to unlock the volume at boot.

     while read -r line; do (1)
       case "$line" in
           Id=cryptsetup:*) d="${line##Id=cryptsetup:}";;
           Socket=*) s="${line##Socket=}";;
       esac
     done < "$question"
     
     [ -b "${d}" ] || continue
     [ -S "${s}" ] || continue
     
     # Detached header hack (2)
     if [ "${d}" = "/dev/disk/by-partuuid/<PARTUUID-identifier>" ]; then (3)
       cryptsetup() {
           /usr/sbin/cryptsetup --header=/boot/luksheader.img "$@" (4)
       }
     fi

     1	Locate the while loop that reads the ask.* files in /run/systemd/ask-password.
     2	Add this and the following lines.
     3	This is an ugly hack to get something that works with minimal changes to the code. All the ask.* file contains is the device, not any of the options. If upstream Clevis supports detached headers in the future, ideally it would do something like look up the device in crypttab to get the header, but I’m not familiar enough with the inner workings of Clevis and Dracut to try and implement that in a quick hack. For now since we know which device we want to unlock and where the header is located for that device, we’ll hardcode it into clevis-luks-askpass. Replace <PARTUUID-identifier> with the PARTUUID of the LUKS volume obtained above.
     4	Like we did to get Clevis to bind to a LUKS volume with a detached header, alias cryptsetup with a function that calls the absolute path to the cryptsetup executable with the header option and then appends the rest of the arguments Clevis originally called cryptsetup with.

4. Set the rd.neednet Dracut kernel command line parameter to make sure network is available in the initramfs for Clevis to be able to contact the Tang server. Create /etc/dracut.conf.d/clevis-nbde.conf:

   kernel_cmdline="rd.neednet=1"

   
   
   

   This step might not be necessary depending on your distro, version of Clevis, and/or version of Dracut. Without it, Clevis failed to unlock the LUKS volume, and I had to type the passphrase in at the prompt. After boot, I checked what the issue was with sudo journalctl | grep clevis and saw a whole bunch of messages like this: Jun 14 10:31:45 example clevis-luks-askpass[869]: Error communicating with the server http://tang.example.com Adding the above Dracut configuration parameter resolved the issue. It appears this issue may not exist in versions of Clevis included with recent versions of Fedora, but at least for the version of Dracut included with Oracle Linux 8.5, the above workaround was still needed.

5. Regenerate initramfs. Include the LUKS detached header in the initramfs.

   $ sudo dracut --install /boot/luksheader.img -f

   If it fails to automatically unlock at boot, you can troubleshoot the issue with sudo journalctl | grep clevis. If it fails to boot at all, you can manually unlock the LUKS volume from the Dracut Emergency Shell, and then it should proceed to boot. Then you can troubleshoot the issue by greping the output of journalctl.