DEV Community

Indra Gusti Prasetya
Indra Gusti Prasetya

Posted on • Originally published at indragustiprasetya.com

Fix kubelet cgroup v1 error on Kubernetes 1.35+

#tutorial

A node comes back from a Kubernetes 1.35 upgrade stuck NotReady, kubelet dead, and journalctl blames cgroup v1. This walks you through the one-command rescue that gets the node serving again, then the migration to cgroup v2 that actually keeps it fixed past v1.38.

The short version first, because you probably found this page mid-incident: as of v1.35 the kubelet config field failCgroupV1 defaults to true, so kubelet deliberately refuses to initialize on any node still booted into the cgroup v1 hierarchy. This is not a bug and not a flaky restart. It is a validation gate that landed via KEP-5573 (implemented in issue #132925), after cgroup v1 sat in "maintenance mode" since v1.31 under KEP-4569.

Here is the trap. The one-line workaround everyone pastes, failCgroupV1: false, is the correct emergency stop, and it is also a countdown timer. Per KEP-5573, removal of cgroup v1 support happens "no earlier than 1.38," and when the code goes, the override flag goes with it. A node you "fixed" with the flag in 1.35 will simply break again on a later upgrade, this time with nothing to set. So we do both jobs: one command to stop the bleeding, then the OS migration to v2 so it stays fixed. It's the same reason I treat migrating Ingress-NGINX to Gateway API before its retirement as real work and not a flag flip. Deprecation windows close.

This is for anyone running kubeadm, k3s, RKE2, or a hand-rolled kubelet on older distros (RHEL 8, Amazon Linux 2, Ubuntu 20.04) that still default to cgroup v1. If you already run current node images, you may only hit this on one forgotten machine.

Prerequisites

  • A Kubernetes 1.35+ node where kubelet fails to start, plus control-plane access to kubectl drain/uncordon.
  • Root/SSH on the affected node and the ability to reboot it.
  • containerd (1.7+ or 2.x) or CRI-O as the runtime, with systemd as init.
  • Linux kernel 5.8+ on the node. cgroup v2 technically needs 4.15+, but 5.8+ gives you full controller parity.

Step-by-step

1. Confirm the node is actually on cgroup v1 

stat -fc %T /sys/fs/cgroup
# cgroup2fs  -> already v2 (your problem is elsewhere)
# tmpfs      -> cgroup v1, this is the cause
Enter fullscreen mode Exit fullscreen mode

tmpfs means the legacy v1 hierarchy is mounted. Do not skip this. If it already prints cgroup2fs, kubelet is failing for some other reason and the rest of this guide won't help you. Cross-check the log so you're chasing the right thing:

journalctl -u kubelet -n 40 --no-pager | grep -i cgroup
Enter fullscreen mode Exit fullscreen mode

Kubelet aborts during config validation with a message stating it is configured not to run on a host using cgroup v1 (the SUSE and minikube trackers document the same string). That exact wording is your fingerprint. It's a refusal to start, not a crash, which is why nothing in the pod logs tells you anything useful.

2. Get the node back NOW with the override (temporary)

Edit the on-node kubelet config (the kubeadm default path is shown) and add the field:

sudo sed -n '/^failCgroupV1/!p' /var/lib/kubelet/config.yaml >/dev/null  # confirm it isn't set
echo 'failCgroupV1: false' | sudo tee -a /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

One thing that catches people: editing the cluster-wide kube-system/kubelet-config ConfigMap alone does not fix a running node. That ConfigMap is only read when a node joins or runs kubeadm upgrade node. You have to change the file on disk and restart. Treat this as buying a maintenance window and nothing more. It's the same "the config you can see isn't the config in effect" gotcha you hit with kubelet's user-namespace behavior in 1.36: the node reads from disk, not from the API you were staring at.

3. Drain the node before the real migration 

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=120s
Enter fullscreen mode Exit fullscreen mode

Switching cgroup hierarchies needs a reboot, so evict workloads first. --ignore-daemonsets is required because DaemonSet pods can't be evicted the normal way, and without it the drain just refuses.

4. Boot the node into the cgroup v2 unified hierarchy

On Ubuntu/Debian, add the kernel parameter and rebuild GRUB:

# /etc/default/grub -> append to GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"
sudo update-grub          # Debian/Ubuntu
# RHEL/CentOS/Rocky: sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
Enter fullscreen mode Exit fullscreen mode

systemd.unified_cgroup_hierarchy=1 tells systemd to mount only the v2 tree at boot. On many modern images (Ubuntu 22.04+, RHEL 9, Amazon Linux 2023, Flatcar, Bottlerocket) v2 is already the default and this step is a no-op. If that's you, the real fix is moving off the old base image, not editing GRUB.

5. Pin the runtime and kubelet to the systemd cgroup driver

cgroup v2 only works cleanly when the runtime and kubelet agree on the driver. For containerd:

# /etc/containerd/config.toml, under the runc runtime options block.
# containerd 1.x: [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
# containerd 2.x: [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl restart containerd
Enter fullscreen mode Exit fullscreen mode

Then set the matching driver in kubelet config:

# /var/lib/kubelet/config.yaml
cgroupDriver: systemd
Enter fullscreen mode Exit fullscreen mode

Since Kubernetes 1.34, CRI cgroup-driver lookup is GA, so kubelet can query the runtime for its driver. Setting cgroupDriver: systemd explicitly on both sides anyway removes any ambiguity, and I'd rather be explicit than debug a silent negotiation later. Once you're on v2, delete the failCgroupV1: false line from step 2 so the node fails loudly if it ever regresses to v1. Leaving the override in place just re-arms the same trap for the v1.38 upgrade.

6. Restart kubelet and return the node to service 

sudo systemctl restart kubelet
kubectl uncordon <node>
Enter fullscreen mode Exit fullscreen mode

Verify it works

Confirm the hierarchy flipped and kubelet is healthy:

stat -fc %T /sys/fs/cgroup      # expect: cgroup2fs
systemctl is-active kubelet     # expect: active
kubectl get node <node>         # expect: Ready, version v1.35.x+
Enter fullscreen mode Exit fullscreen mode

Now confirm the driver actually negotiated as systemd. A silent mismatch here is the classic post-migration trap, and the node looks fine right up until a pod won't schedule:

journalctl -u kubelet | grep -i "cgroup driver"
# expect a line reporting the systemd cgroup driver, not cgroupfs
Enter fullscreen mode Exit fullscreen mode

Finally, schedule a throwaway pod to prove the runtime can create v2 cgroups end to end:

kubectl run cgtest --image=busybox --restart=Never --rm -it --command -- true
Enter fullscreen mode Exit fullscreen mode

If that pod runs and cleans itself up, the whole chain works: v2 hierarchy, agreed driver, runtime creating cgroups.

Common pitfalls

  • Treating failCgroupV1: false as the fix. It is deprecated on arrival. KEP-5573 sets removal "no earlier than 1.38," so the flag you leaned on in 1.35 will not exist on a later upgrade. Migrate the OS during this window, not after it slams shut.
  • Driver mismatch after migration. If containerd stays on SystemdCgroup = false (cgroupfs) while kubelet uses systemd, pods fail to start or the node flaps NotReady, even though you are correctly on cgroup v2. Both must say systemd. This is the single most common "I migrated and it's still broken" cause I see.
  • Editing the ConfigMap instead of the node file. Changing kube-system/kubelet-config does nothing for a running node until it re-reads config on join or upgrade. Fix /var/lib/kubelet/config.yaml on the box itself.
  • The distro you forgot. RHEL 8, Amazon Linux 2, and Ubuntu 20.04 still default to cgroup v1. EKS nodes on the retired AL2 AMI are the frequent surprise, and the fix there is AL2023, not a kubelet flag. Managed GKE/EKS current node images are already v2, so the bite lands on self-managed and older nodes. This is exactly the kind of node-level assumption worth catching before it becomes an outage, the same way you'd audit for a default-deny egress gap rather than discover it in production.
  • kind/minikube on a v1 host. These run nested, so the host's cgroup mode leaks in, and on a v1 host you hit the same validation. Move the host to v2, or for local-only clusters, minikube exposes a skip per issue #22315.

Wrap-up

You now have a node booting into the cgroup v2 unified hierarchy, a runtime and kubelet that agree on the systemd driver, and no reliance on an override Kubernetes will delete around v1.38. Make it durable: bake systemd.unified_cgroup_hierarchy=1 (or a v2-default base image) and SystemdCgroup = true into your node image, Ansible, or Terraform so every new node joins on v2. Then add one fleet check, stat -fc %T /sys/fs/cgroup across all nodes, to your pre-upgrade gate. Do that and v1.38 never strands a node on you, because there won't be a v1 node left to strand.

Sources

Originally published at indragustiprasetya.com

Top comments (0)