DEV Community

R (RYUSLOG DEV)
R (RYUSLOG DEV)

Posted on Edited on Originally published at ryuslog.com

Kubernetes 1.35: kubelet will not start on cgroup v1

Introduction & Quick Summary

Symptom. After upgrading a cluster's nodes to Kubernetes 1.35, new nodes never become Ready. The kubelet on the node exits almost immediately:

$ systemctl status kubelet
Active: failed (Result: exit-code)

$ journalctl -u kubelet -n 20 --no-pager
... kubelet: cgroup v1 is no longer supported; failing to start
Enter fullscreen mode Exit fullscreen mode
$ kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
worker-3      NotReady   <none>   4m    v1.34.9
Enter fullscreen mode Exit fullscreen mode

The node object is registered by an older kubelet or not at all, pods stay Pending, and kubectl describe node worker-3 shows Ready False with KubeletNotReady.

Cause. Kubernetes 1.35 removed cgroup v1 support. A node booting on cgroup v1 with a 1.35 kubelet does not join, and the failure is at the host, not in any Kubernetes object. The same release also removed the --pod-infra-container-image kubelet flag, so custom node bootstrap configurations that pass it exit on startup with an unknown flag.

Fix. Move the node to cgroup v2, or set failCgroupV1: false while you plan the move.

# kubelet configuration on the node (not a Kubernetes object)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false
Enter fullscreen mode Exit fullscreen mode
# confirm what the host booted with before touching the kubelet
stat -fc %T /sys/fs/cgroup     # tmpfs or cgroup2fs means cgroup v2
grep -o 'cgroup[^ ]*' /proc/cmdline
Enter fullscreen mode Exit fullscreen mode

Core Mechanics, Signatures & Reproducible Setup

Kubernetes 1.25 added cgroup v2 support; 1.35 removed v1. The kubelet reads the host hierarchy at startup and refuses to run when the only hierarchy is v1, because every resource limit, QoS class and eviction threshold it enforces is expressed through the v2 unified hierarchy. This is not a warning you can ignore: the process exits.

Check the whole fleet in one pass rather than one node at a time:

for n in $(kubectl get nodes -o name | cut -d/ -f2); do
  printf "%s " "$n"
  kubectl debug node/$n -it --image=busybox -- \
    sh -c 'stat -fc %T /sys/fs/cgroup' 2>/dev/null | tail -1
done
Enter fullscreen mode Exit fullscreen mode

The supported configurations are uncomplicated: Amazon Linux 2023 and Bottlerocket default to cgroup v2, and most distributions from 2022 onward do too. The risk is a launch template, nodeadm config, or custom AMI that put the host back on v1 with systemd.unified_cgroup_hierarchy=0 on the kernel command line, or an older base image that never had v2.

Reproduce the failure in a scratch cluster by starting a node with that kernel argument, then roll the kubelet to 1.35 and watch it exit. Doing it once in a test cluster is cheaper than discovering it during a production node pool rotation.

Two related removals in the same upgrade:

# this flag no longer exists; a node that passes it never becomes Ready
grep -rn "pod-infra-container-image" /etc/kubernetes /var/lib/kubelet \
  /etc/systemd/system/kubelet.service.d 2>/dev/null
Enter fullscreen mode Exit fullscreen mode

And containerd 1.x is supported for the last time in 1.35, so the next upgrade requires containerd 2.0 or later. Find the nodes that are behind:

kubectl get --raw /metrics | grep kubelet_cri_losing_support
Enter fullscreen mode Exit fullscreen mode

Key Comparison Table

Kubernetes version cgroup v1 node containerd 1.x kube-proxy ipvs
1.33 works supported works
1.34 works, warning supported works
1.35 kubelet will not start last supported release deprecated, warns
next release removed removed removal planned
What you see Where it comes from First check
--- --- ---
Node never registers kubelet exited on cgroup v1 journalctl -u kubelet -n 50
unknown flag: --pod-infra-container-image bootstrap user data or AMI nodeadm config, Karpenter EC2NodeClass
Node joins then goes NotReady runtime not ready or sandbox image missing crictl info, systemctl status containerd
kubelet_cri_losing_support metric rises containerd 1.x containerd --version per node
kube-proxy logs a deprecation warning mode: ipvs switch to nftables in a separate window

Practical Edge Cases, Error Fixes & Deep Dive Code Examples

1. cgroup v1 hides in the kernel command line

stat -fc %T /sys/fs/cgroup returning tmpfs means v1 (v2 returns cgroup2fs), but the reason is often a single kernel argument in the launch template. Remove it rather than setting failCgroupV1: false and calling it done:

# before
user_data = <<-EOF
#!/bin/bash
sed -i 's/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 /' /etc/default/grub
update-grub
EOF

# after: nothing; AL2023 already boots with cgroup v2
Enter fullscreen mode Exit fullscreen mode

Rebuild the node group rather than patching running nodes: the kernel argument is read at boot, so an in-place change needs a reboot anyway, and a rolling replacement is the same cost with a cleaner state.

2. failCgroupV1: false as a documented escape hatch

The field exists so that a managed platform can keep working while hosts are migrated; Bottlerocket's 1.35 image sets it itself. Use it deliberately, with a date next to it, because it keeps you on a mode that a future release removes:

# kubeadm clusters: drop it in and restart
sudo tee /var/lib/kubelet/config.yaml.d/99-cgroup.yaml >/dev/null <<'EOF'
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false
EOF
sudo systemctl restart kubelet
Enter fullscreen mode Exit fullscreen mode

Confirm the node joins and that the runtime reports a v2 hierarchy once you have moved the host, then delete the override.

3. The removed --pod-infra-container-image flag

A kubelet started with a flag it no longer knows exits at once with a usage error, so the node never registers and the log line is easy to miss among the normal startup noise. The flag lives on the host, in bootstrap user data, spec.kubelet.flags in a nodeadm NodeConfig, Karpenter EC2NodeClass user data, and custom AMI build scripts. No cluster-side query can see it:

kubectl get nodes -o wide
# for each node that will not join
gcloud compute instances describe ... # or aws ssm send-command
Enter fullscreen mode Exit fullscreen mode

Remove the flag and let the runtime supply the sandbox image. EKS-optimized AMIs never set it; custom images often do.

4. Ingress NGINX is on its way out

Ingress NGINX received best-effort maintenance until March 2026 and is archived after that, which is a schedule problem rather than a 1.35 error. A cluster that depends on ingress-nginx annotations is a cluster with a migration to plan. Enumerate the exposure before you promise a timeline:

kubectl get ingress -A -o json | jq -r '.items[] | [.metadata.namespace, .metadata.name,
  ([.metadata.annotations // {} | keys[] | select(startswith("nginx.ingress.kubernetes.io"))] | length)] | join("  ")'
Enter fullscreen mode Exit fullscreen mode

Annotations have no Gateway API equivalent in some cases, so the count of annotations per Ingress is a reasonable first estimate of the work, not just the count of objects.

5. Version skew when you cannot move every node at once

A 1.35 API server supports kubelets back to 1.32, so the control plane can move first while node pools follow. The order matters more than the speed: API servers one at a time, then controllers, then kubelets node by node after a drain, then kube-proxy no newer than the API server.

kubectl drain worker-3 --ignore-daemonsets --delete-emptydir-data --timeout=300s
# upgrade kubelet and kubeadm on the node
kubectl uncordon worker-3
Enter fullscreen mode Exit fullscreen mode

Do not upgrade a node pool whose hosts are still on cgroup v1 with the new kubelet: the drain succeeds and the node never comes back, which turns a routine rotation into an outage.

6. Verifying the upgrade before the maintenance window

kubectl speaks to a 1.35 API server from versions 1.34 through 1.36, so the client is not the problem. The problems are all in the manifest and host layers. Check the manifest layer with the metrics the API server already exposes:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
grep -rn "storage.k8s.io/v1alpha1" ./manifests ./charts
Enter fullscreen mode Exit fullscreen mode

A non-empty metric means a client called a deprecated API since the last API server restart. Convert manifests with kubectl convert where a replacement exists, and validate the YAML you produce with the YAML validator before applying it to a test cluster.

Common Compilation/Runtime Mistakes & Version Gotchas

  • Assuming Ready means the node is on the new version. Check kubectl get nodes -o wide for the kubelet version, not just the status.
  • Setting failCgroupV1: false and forgetting it. The next release removes the escape hatch entirely.
  • Patching a node in place instead of replacing it. Kernel arguments and container runtime versions belong to the image, not the running process.
  • Upgrading kube-proxy past the API server. It is unsupported and produces confusing service failures.
  • Reading a Pending pod as a scheduling bug. With a node that never joined there is nothing to schedule onto; look at nodes first.
  • Leaving ipvs mode because it still works. It warns at startup today and disappears later; nftables needs kernel 5.13 or newer, which is the real constraint.
  • Testing on a single-node cluster. cgroup and runtime differences only show up on real hosts, which is the whole point.

Reference: Kubernetes v1.35 release announcement and cgroup v2 documentation.

Top comments (1)

Collapse
 
onelifevibe profile image
OneLifeVibe •

The detail that cgroup v1 often hides in a single kernel argument stood out. Checking stat -fc %T /sys/fs/cgroup before touching the kubelet is a good first step, because the symptom looks like a scheduling problem when the node never actually joined. I also appreciated the warning to treat failCgroupV1: false as a temporary escape hatch rather than a fix, since the next release removes it entirely. Did you run into many clusters where that argument was still set in a launch template long after the base image had moved on?