Citrix XenServer 7.1 CentOS VM’s not starting after Kernel update

After I update my CentOS 7 VM kernel, the new kernel stopped booting on my XenServer hypervisor. I’ve been using XenServer version 7.1

Working Kernel version, before yum update, was 3.10.0-693 and, after yum update Kernel version is 3.10.0-1127.19.1

This issue is due to the kernel which I updated has the spectre/meltdown patches in. But our XenServer 7 doesn’t. This difference created the kernel booting problem. Interestingly, this issue only affects CentOS VMs. My other Ubuntu servers can just run updates without issue but CentOS VM’s require the host to be updated or they stop working.

To FIX this issue you should update your XenServer to the latest version or else if there are hotfixes available then you should install it.

Link for XenServer 7.1 hotfixes. https://support.citrix.com/article/CTX230788

To upgrade XenServer 7.1 you should have Citrix License first. But a "free" patch was released for 7.2, due to the massive security issue later.

But if you have been running older XenServer's version, older than 7.1 then you don't have any choice besides updating XenServer to the latest.

Also, CentOS fixed their booting problem on their new Kernel versions which I will discuss later below.

What is Meltdown and Spectre vulnerabilities?

https://spectreattack.com

Someone having access to the VM will be able to read the memory outside the VM itself, and therefore read data from other VMs on this same physical host. An attacker could enter via other application exploits (webserver etc.) on your VMs.

Am I affected?

Meltdown is using a design flaw into Intel CPUs only. This is called by Xen sec team "SP3" (aka rogue data cache load).

You are impacted only if you are using:

• 64-bits PV type VM (HVM/PVHVM aren't affected!)
• Intel CPUs (AMD chip design is a bit different and not affected)
• untrusted VMs, ie untrusted users having VM access (even non-root!)
• All XenServer versions are affected

64-bits PV guests are vulnerable because guest and hypervisor share the same address space, but with different privileges. HVM aren't.

Mitigate Risk:

Migrate all VM’s from PV to HVM or upgrade XenServer to latest or install hotfixes if available.

But if you have been running older version XenServer, older than 7.1 then you have to update your XenServer to resolve this issue.

Back to CentOS 7 booting issue.
It seems the booting issue was fixed in the latest CentOS kernel versions. If the XenServer was up-to-date with the hotfixes then the VM’s should not have any problem. But in my case I didn't update my XenServer, so now I’ve to either install the latest CentOS 7 kernel or downgrade to older previous running one.

Upgrading Kernel Version

Add ELRepo Repository first.

Add ELRepo gpg key to the system.

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

This is important – CentOS will not allow the installation of an unsigned software package. The GPG key provides a digital signature to verify that the software is authentic.

Now add new ELRepo repository.

rpm -Uvh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

When it's complete, check all repository enabled on the system, and make sure ELRepo is on the list.

yum repolist

yum --enablerepo=elrepo-kernel install kernel-ml

kernel-ml – which indicates a mainline release, with a shorter support term but with more frequent updates.

OR INSTALL (Install either ml or lt)

yum --enablerepo=elrepo-kernel install kernel-lt

kernel-lt – signifying a stable long-term support release.

listout all your existing kernel version on your system.

awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg

CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)

We have sucessfully installed CentOS latest kernel version 4.4.234-1. Now set this as a default kernel.

Set Default Kernel Version

First listout all existing kernel versions.

awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)

NOTE: The first entry is denoted as Zero. Let us modify the Kernel Version to 4.4.234-1 which is at line number 1 but denoted as entry 0.

Set default kernel.

grub2-set-default 0

Next, Rebuild grub.cfg file with 'gurb2-mkconfig' command.

grub2-mkconfig -o /boot/grub2/grub.cfg

To check default Kernel Version

cat /boot/grub2/grubenv |grep saved

saved_entry=CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)

Reboot Server

reboot

Above solution will resolves the booting issue.

If you don't want to update CentOS kernel to latest one and want to boot from the existing previous working kernel then reboot your system and choose kernel version which was working fine before from grub menu and select enter and follow the below steps.

Set Default Kernel Version but first list-out all existing kernel versions.

awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)

NOTE: The first entry is denoted as Zero. So currently the Server is booted to 0th entry as per the above command output. Let us modify the Kernel Version to 3.10.0-693 which is at line number 4 but denoted as entry 3.

Set default kernel.

grub2-set-default 3

Next, Rebuild grub.cfg file with 'gurb2-mkconfig' command.

grub2-mkconfig -o /boot/grub2/grub.cfg

Check default Kernel Version

cat /boot/grub2/grubenv | grep saved

saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)

Reboot Server

reboot