DEV Community

Cover image for One command tells you if your next Kubernetes upgrade will brick a node
Sai Pisey
Sai Pisey

Posted on Originally published at saipisey.com

One command tells you if your next Kubernetes upgrade will brick a node

#kubernetes #devops #sre #linux

Since v1.35, the kubelet setting failCgroupV1 defaults to true. On a node that still runs cgroup v1, the kubelet refuses to start, so the node goes NotReady and stays there. On a control-plane node it is worse: the kubelet starts the API server and etcd as static pods, so after a reboot they do not come back either.

After upgrading the kubelet to v1.35: a cgroup v2 node stays Ready, cgroup v1 nodes go NotReady

kubeadm warns you first: its SystemVerification preflight check errors on init, join and upgrade once the kubelet is v1.35 or later. Do not skip it with --ignore-preflight-errors.

Under the hood, cgroup v1 and v2 lay out the same pod very differently, which is why one stat call can tell them apart:

cgroup v1 keeps one tree per controller; cgroup v2 keeps one unified tree

Check one node 

$ stat -fc %T /sys/fs/cgroup/
cgroup2fs
Enter fullscreen mode Exit fullscreen mode
  • cgroup2fs: cgroup v2. Nothing to do.
  • tmpfs: cgroup v1. After an upgrade to v1.35 or later, its kubelet will not start.

Check every node without SSH 

$ for n in $(kubectl get nodes -o name); do
    printf "%s " "$n"
    kubectl debug "$n" -n default -q -i --profile=general --image=busybox \
      -- chroot /host stat -fc %T /sys/fs/cgroup
  done
node/worker-1 cgroup2fs
node/worker-2 tmpfs
Enter fullscreen mode Exit fullscreen mode

kubectl debug mounts the node at /host; chroot /host uses the node's GNU stat

Keep the chroot /host. Without it the command runs busybox's own stat, which prints UNKNOWN instead of cgroup2fs on a healthy v2 node. I hit that while testing on a v1.35 cluster.

Which nodes are likely on cgroup v1

CentOS 7, RHEL 7 and 8, Ubuntu 18.04 and 20.04, and Amazon Linux 2 all default to cgroup v1. Self-managed kubeadm clusters on long-lived VMs are where this bites.

The full post covers what a node needs for cgroup v2, which nodes can switch in place and which need a new OS image, how to check the kubelet and containerd cgroup drivers, and the failCgroupV1: false escape hatch with its expiry date.

Read the full post on saipisey.com →

Top comments (0)