I tried to bring up a kind cluster on this machine using the current stable Kubernetes, 1.37.0, and the control plane never came up. kubectl get nodes never returned anything, because the kubelet never got past its own startup checks. systemctl status kubelet inside the node container showed this:
Active: activating (auto-restart) (Result: exit-code) since Wed 2026-10-21 05:20:55 UTC; 959ms ago
Restarting every second, forever. The actual error, from the journal:
E1021 05:20:25.129684 1003 run.go:72] "command failed" err="failed to validate kubelet
configuration, error: kubelet is configured to not run on a host using cgroup v1.
cgroup v1 support is unsupported and will be removed in a future release,
path: &TypeMeta{Kind:,APIVersion:,}"
This machine still runs cgroup v1. Starting with Kubernetes 1.35, that alone is now fatal, by default, with no grace period.
Cgroup v1 is not some exotic setup. It's what you get on older kernels, older base images, and any host nobody has gone back to reconfigure since before cgroup v2 was the default. The failure mode here isn't "feature doesn't work as well as hoped." It's "kubelet will not run at all, and it will not tell you that until you're already mid-upgrade."
Confirming the boundary myself
The behaviour is in a kubelet field called FailCgroupV1, and Kubernetes' own blog, published two weeks before I ran this, says the default flipped to true in 1.35. I didn't want to take that on faith, so I pulled three kindest/node images and watched each one's kubelet directly with docker exec and journalctl, rather than waiting on the whole cluster to come up.
| image | kubelet on a cgroup v1 host |
|---|---|
kindest/node:v1.34.0 |
starts, logs one warning event, stays running |
kindest/node:v1.35.0 |
exits immediately, restarts, exits again |
kindest/node:v1.37.0 |
same as 1.35 |
On 1.34.0 the kubelet logged a single warning event and then just ran:
Message:cgroup v1 support is in maintenance mode, please migrate to cgroup v2
Type:Warning
On 1.35.0 and 1.37.0 it never got that far. Both produced the exact same fatal error above and both crash-looped indefinitely. The boundary is real, and it's exactly where the blog post says it is. I didn't test 1.36.1, the release in between, so I'm only reporting the boundary at 1.34 versus 1.35/1.37, not claiming anything about 1.36 specifically.
What the crash loop actually costs
I measured the 1.37.0 node's restart counter twice, thirty seconds apart, with systemctl show kubelet -p NRestarts:
| check | elapsed since kubelet first ran | restarts |
|---|---|---|
| first | ~60s | 58 |
| second | ~90s | 82 |
That's 24 restarts in 30 seconds, or a full crash-and-respawn roughly every 1.25 seconds. It doesn't back off and it doesn't stop; I left a 1.35.0 node running longer and it reached 137 restarts with no sign of slowing. The node sat at 9.35% CPU doing nothing useful, and its systemd journal had already grown to 8.0MB purely from one error message looping. A single misconfigured cgroup v1 node, left alone, will burn CPU and fill its own log forever rather than failing once and going quiet.
The documented fix stops the kubelet crash, nothing else
The blog's workaround is a kubelet config line:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false
I applied this to a fresh 1.37.0 node through a kind config patch. It worked, exactly as advertised, at the kubelet level:
$ docker exec c137wa-control-plane systemctl show kubelet -p NRestarts
NRestarts=0
Stable, no crash. But the blog's fix only covers the kubelet. On a real cluster you also run kubeadm init, and kubeadm runs its own, separate preflight check before it ever touches the kubelet config. I ran that phase by hand, outside of kind, to see it fire on its own:
$ kubeadm init phase preflight --config=/kind/kubeadm.conf
[preflight] Some fatal errors occurred:
[ERROR SystemVerification]: cgroups v1 support is deprecated and will be removed
in a future release. Please migrate to cgroups v2. To explicitly enable cgroups v1
support for kubelet v1.35 or newer, you must set the kubelet configuration option
'FailCgroupV1' to 'false'. You must also explicitly skip this validation. For more
information, see https://git.k8s.io/enhancements/keps/sig-node/5573-remove-cgroup-v1
Read that last sentence again: kubeadm's own error message says you must also explicitly skip its validation, separately from the kubelet setting. The fix is two flags, not one, and the first one I tried, failCgroupV1: false alone, would have left a plain kubeadm init dead on preflight regardless of what the kubelet config said. Adding the second flag does exactly what the error promises:
$ kubeadm init phase preflight --config=/kind/kubeadm.conf --ignore-preflight-errors=SystemVerification
[WARNING SystemVerification]: cgroups v1 support is deprecated and will be removed
in a future release. ...
Same text, downgraded from a fatal error to a warning. That's the actual two-part workaround: failCgroupV1: false in the kubelet config, and --ignore-preflight-errors=SystemVerification on every kubeadm init, join, and upgrade. The blog post I checked this against only documents the first half.
The override has a quiet side effect
With failCgroupV1: false applied and the kubelet running cleanly, I still found this in the same startup log, a few lines after the config dump:
I1021 05:21:07.455963 278 server.go:682] "Warning: MemoryQoS feature only works with
cgroups v2 on Linux, but enabled with cgroups v1"
Memory QoS has been a stable kubelet feature for several releases. Keeping cgroup v1 alive with the override doesn't just let an old node limp along unchanged — it silently drops a scheduling feature that's supposed to be on by default, with nothing in the fatal-error path warning you about it. If you adopt the override as a stopgap because migrating every node to cgroup v2 isn't a today problem, you've also quietly turned off memory QoS on all of them, and the only way you'd notice is reading the full startup log rather than grepping it for "error".
kind hides half of this by default
While building the test above I noticed kind never actually runs the kubeadm preflight phase I quoted. The kubeadm config kind generates for every node includes this:
skipPhases:
- preflight
So if your only exposure to this change is through kind, you will never see the SystemVerification error or its warning at all — kind skips that check unconditionally, on every version, cgroup v1 or not. What you'll see instead is purely the kubelet's own internal validation killing the process a layer further down, which is the crash loop at the top of this post. The two checks exist independently in real Kubernetes, kubeadm's and kubelet's own, and testing through kind only exercises one of them.
What I got wrong on the way
My original plan was to demonstrate this against a fully working cluster: bring up kind, show kubectl get nodes going Ready, then break and fix it. I couldn't get that far. Every node I started, on every Kubernetes version including 1.34, failed to actually run its static pods, with containerd reporting:
OCI runtime create failed: runc create failed: unable to start container process:
can't get final child's PID from pipe: EOF
That's a sandboxing limitation of the nested container environment this ran in, not a Kubernetes bug, and it hit 1.34 exactly as hard as 1.37. Once I confirmed the failure was identical and version-independent, I stopped trying to get a live cluster and tested the kubelet and kubeadm directly with docker exec and systemctl/journalctl instead, which turned out to be the right layer for this question anyway, since the thing I was measuring happens entirely before any pod is ever scheduled.
Run it yourself
This needs docker and the kind and kubectl binaries; no cloud account.
curl -sL -o /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-linux-amd64
chmod +x /usr/local/bin/kind
# default behaviour: crash loop on a cgroup v1 host
kind create cluster --name c137 --image kindest/node:v1.37.0 --retain || true
docker exec c137-control-plane systemctl show kubelet -p NRestarts
docker exec c137-control-plane journalctl -u kubelet --no-pager | grep "command failed" | tail -1
kind delete cluster --name c137
# the two-part workaround
cat > /tmp/kind-workaround.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: KubeletConfiguration
failCgroupV1: false
EOF
kind create cluster --name c137wa --image kindest/node:v1.37.0 \
--config /tmp/kind-workaround.yaml --retain || true
docker exec c137wa-control-plane systemctl show kubelet -p NRestarts
docker exec c137wa-control-plane journalctl -u kubelet --no-pager | grep -i memoryqos
kind delete cluster --name c137wa
Check grep cgroup /proc/filesystems and stat -fc %T /sys/fs/cgroup first; if that prints cgroup2fs you're already on v2 and none of this applies to you.
What to check
Run stat -fc %T /sys/fs/cgroup on every node you plan to run Kubernetes 1.35 or later on. If it says tmpfs rather than cgroup2fs, you're on cgroup v1, and an in-place upgrade to 1.35+ will not degrade gracefully, it will crash-loop every kubelet on every affected node the moment the new binary starts. That's not a staged rollout risk, it's an outage the instant the new kubelet binary lands, with no warning beforehand unless someone already read the 1.35 changelog.
Migrate to cgroup v2 before the upgrade if you can. If you genuinely can't yet, set both halves of the override, kubelet's failCgroupV1: false and kubeadm's --ignore-preflight-errors=SystemVerification, and then go looking in your kubelet logs for the Memory QoS warning, so you know, deliberately, what you gave up to keep the node alive, rather than finding out later from a scheduling decision that doesn't make sense. Cgroup v1's fallback path is on record as scheduled for removal in 1.38, two releases from now, so this buys you a reprieve measured in months, not a long-term plan. Put the migration on the calendar now, while the override still exists to buy you the time to do it properly.
Top comments (4)
Good catch on the two-part fix, most people only find the kubelet half. Treat failCgroupV1: false as a migration bridge rather than a permanent setting, since cgroup v1 is already in maintenance mode and the journal error says outright it will be removed in a future release. Before upgrading a fleet, inventory every node with stat -fc %T /sys/fs/cgroup so no host surprises you mid-rollout. And take the Memory QoS warning seriously: those protections do not apply on cgroup v1, which tends to surface as OOM kills under load later.
Yeah, migration bridge is exactly how I'd treat it too. The flag buys you time to move the node to v2, not a reason to stop thinking about it. And the error message literally tells you it's going away, so anyone leaving it on forever is choosing to get surprised later.
Fleet inventory first is the right order. Finding out which node is still on tmpfs from a crash-looping kubelet halfway through a rollout is the worst possible way to learn it lol.
On Memory QoS, agreed it's the sneaky part, and I need to fix something in the post because of it. I wrote it's been stable for several releases, that's wrong. It was alpha and off by default all the way to 1.36, and only went beta and on by default in 1.37, which is the version I saw the warning on. So on 1.35 and 1.36 you probably won't hit it unless you turned the gate on yourself, but anyone landing on 1.37+ with the v1 override loses memory.high throttling, and pods just run into the hard limit and get OOM killed instead of slowing down first. Same as v1 always was, but now it's quietly not what the defaults promise.
This will come handy on my upgrade thanks
You're most welcome!