Kubernetes 1.35+ Fails to Start on cgroup v1 Nodes: Disable `failCgroupV1` or Upgrade to cgroup v2

Introduction

Kubernetes administrators must act immediately to address a critical compatibility issue introduced in kubelet version 1.35 and later. The kubelet, responsible for managing pods on cluster nodes, now terminates itself upon detecting cgroup v1 due to the failCgroupV1 setting, which defaults to true. This behavior is not a warning or a graceful degradation—it is a hard enforcement mechanism that renders affected nodes inoperable. Organizations failing to verify and rectify cgroup version compatibility face immediate cluster instability and unplanned downtime, as evidenced by widespread reports of control-plane failures in production environments.

Mechanism of Failure

The kubelet’s termination process on cgroup v1 nodes unfolds in three distinct phases:

1. Version Detection: During initialization, the kubelet interrogates the /sys/fs/cgroup/ filesystem to determine the cgroup version. It identifies v1 by the presence of tmpfs and v2 by cgroup2fs. This check is binary, with no fallback or partial compatibility.
2. Termination Enforcement: If failCgroupV1: true, the kubelet invokes an immediate os.Exit(1) upon detecting v1. This abrupt termination bypasses graceful shutdown procedures, halting all pod management capabilities on the node.
3. Control-Plane Collapse: On control-plane nodes, this termination prevents critical static pods (e.g., API server, etcd) from restarting post-reboot. Without these components, the cluster loses its ability to manage workloads, effectively rendering it non-functional.

Root Cause: Pervasive cgroup v1 Dependency

The urgency of this issue is compounded by the prevalence of cgroup v1 in enterprise environments. Major Linux distributions default to v1 due to kernel version constraints:

• CentOS 7
• RHEL 7/8
• Ubuntu 18.04/20.04
• Amazon Linux 2 (AL2)

These distributions ship with kernels below 5.8, the minimum required for cgroup v2. Cloud providers, including AWS, further exacerbate the issue by not supporting v2 on AL2 instances, even with kernel arguments. Upgrading to cgroup v2 necessitates a full OS migration, as v1 and v2 are mutually exclusive at the kernel level.

Temporary Mitigation: Disabling failCgroupV1

Kubernetes provides a temporary workaround by setting failCgroupV1: false in the kubelet configuration. This bypasses the version check, allowing v1 nodes to remain operational. However, this solution is explicitly time-bound: KEP-5573 mandates the removal of this option no earlier than version 1.38. Administrators must treat this as a transitional measure, not a long-term fix.

Non-Disruptive Node Assessment

To evaluate cgroup compatibility without direct node access, execute the following command:

for n in $(kubectl get nodes -o name); do printf "%s " "$n"; kubectl debug "$n" -n default -q -i --profile=general --image=busybox -- chroot /host stat -fc %T /sys/fs/cgroup; done

This script leverages kubectl debug to inspect the cgroup filesystem on each node. Critical considerations:

• The chroot /host directive is essential to correct BusyBox’s misidentification of cgroup v2 as UNKNOWN.
• Each execution leaves a `node-debugger- pod in the default namespace. Remove these artifacts with:


kubectl delete pods -n default -l app=node-debugger


Consequences of Inaction

Failure to address this incompatibility results in:

• Immediate kubelet failures on all cgroup v1 nodes post-upgrade to Kubernetes 1.35+.
• Permanent control-plane instability, as static pods fail to restart after node reboots.
• Cluster-wide service disruptions due to control-plane node unavailability.

Given the widespread adoption of Kubernetes 1.35+ and the dominance of cgroup v1, the risk is both immediate and systemic. Proactive verification and remediation are not optional.

For a comprehensive remediation checklist and advanced troubleshooting steps, consult this technical guide.

Technical Background

The Kubernetes 1.35+ startup failure on cgroup v1 nodes stems from a critical design decision: the failCgroupV1 flag now defaults to true. This change serves as a proactive enforcement mechanism, compelling administrators to address the inherent incompatibility between Kubernetes 1.35+ and the legacy cgroup v1 system. By halting kubelet startup on v1 nodes, this default behavior prevents silent failures and ensures alignment with the container orchestration ecosystem's shift toward cgroup v2.

The Role of cgroup v1 and v2

Control groups (cgroups) are a Linux kernel feature designed to allocate, monitor, and isolate system resources (CPU, memory, I/O, etc.) among process groups. cgroup v1, the legacy implementation, has been the default in most Linux distributions. However, its architecture suffers from critical limitations:

• Hierarchical Fragmentation: Resources are managed through disjointed controllers (e.g., cpu, memory), leading to configuration complexity and potential resource contention.
• Incompatible Hierarchies: Controllers cannot share a unified hierarchy, preventing cohesive resource management across subsystems.

cgroup v2, introduced in Linux kernel 4.5 (2016), resolves these issues by consolidating all controllers under a single, unified hierarchy. Kubernetes 1.35+ enforces this transition by refusing to initialize on v1 systems, as v2 is now the standard for containerized workloads.

Mechanism of failCgroupV1 Enforcement

The kubelet's startup sequence now incorporates a rigorous cgroup version check. The causal mechanism is as follows:

1. Filesystem Detection: The kubelet inspects /sys/fs/cgroup/ to determine the cgroup filesystem type. A tmpfs mount indicates v1, while cgroup2fs confirms v2.
2. Enforcement Logic: With failCgroupV1: true, detection of v1 triggers an immediate os.Exit(1) call, terminating the kubelet process before pod management can commence.
3. Systemic Impact: On control-plane nodes, this termination prevents critical static pods (API server, etcd) from initializing, rendering the cluster non-functional post-reboot or upgrade.

This enforcement is a deliberate measure to preempt operational degradation and ensure compatibility with modern container orchestration requirements.

Distribution-Specific Constraints

The transition challenge is compounded by the prevalence of cgroup v1 in widely deployed distributions:

• Kernel Version Limitations: Distributions such as CentOS 7, RHEL 7/8, Ubuntu 18.04/20.04, and Amazon Linux 2 ship with kernels below 5.8, the minimum version for stable cgroup v2 support. Enabling v2 via boot parameters (e.g., systemd.unified\_cgroup\_hierarchy=1) on these kernels fails due to missing kernel features.
• Cloud Provider Restrictions: AWS explicitly prohibits cgroup v2 on Amazon Linux 2 (AL2) images, even with kernel upgrades. Migration to AL2023 or another v2-compatible image is mandatory for compliance.

Temporary Mitigation: failCgroupV1: false

Setting failCgroupV1: false in the kubelet configuration bypasses the version check, enabling v1 nodes to start. However, this is a finite solution. Kubernetes Enhancement Proposal KEP-5573 mandates the removal of this option by version 1.38, necessitating a full transition to cgroup v2.

Risk Propagation Mechanism

Inaction carries the following cascading risks:

1. Upgrade Initiation: Upgrading to Kubernetes 1.35+ on a cgroup v1 node.
2. Immediate Failure: Kubelet termination halts pod scheduling and management.
3. Control-Plane Collapse: Static pods (API server, etcd) fail to initialize, disrupting core cluster services.
4. Cluster-Wide Outage: Loss of control-plane functionality renders the entire cluster inoperable.

This risk is systemic due to the widespread reliance on cgroup v1 and the rapid adoption of Kubernetes 1.35+ in production environments. Immediate verification and remediation are imperative to ensure cluster stability.

Affected Scenarios

Kubernetes 1.35+ introduces a critical incompatibility with cgroup v1, manifesting in six distinct scenarios tied to specific cluster lifecycle events. This issue stems from the kubelet's enforced termination upon detecting cgroup v1, a behavior governed by the failCgroupV1: true default setting. The following breakdown elucidates the causal mechanisms and observable effects in each scenario:

• Cluster Initialization (kubeadm init)

During kubeadm init, the SystemVerification preflight checks explicitly validate cgroup compatibility. Detection of cgroup v1 triggers an immediate os.Exit(1) via the kubelet's stat command, which identifies /sys/fs/cgroup as tmpfs. This termination halts the initialization process, preventing the deployment of critical static pods such as the API server and etcd. Consequently, the cluster remains non-functional, with core components failing to initialize.

• Node Joining (kubeadm join)

When a new node attempts to join the cluster via kubeadm join, the same preflight checks are enforced. If cgroup v1 is detected, the kubelet terminates, leaving the node in an unjoined state. The causal sequence mirrors cluster initialization: tmpfs detection → kubelet termination → failure to register with the API server. This renders the node unusable within the cluster.

• Control-Plane Node Reboot

Rebooting a control-plane node running cgroup v1 triggers kubelet termination due to the failCgroupV1: true setting. As the kubelet fails to start, static pods—including the API server and etcd—cannot be managed or restarted. This results in a collapsed control plane, characterized by cluster-wide API unavailability and data store inaccessibility, effectively paralyzing the entire cluster.

• Worker Node Reboot

Worker nodes operating on cgroup v1 experience kubelet termination post-reboot, halting all pod scheduling and management. While less critical than control-plane failures, this scenario leads to workload eviction and service disruption. The underlying mechanism is consistent: tmpfs detection → kubelet exit → pods persist in Pending or Unknown states, disrupting application availability.

• In-Place OS Upgrades

In-place upgrades from cgroup v1 to v2 (e.g., CentOS 7 to 8) frequently fail due to kernel version incompatibilities. Legacy distributions such as CentOS 7 and RHEL 7/8 ship with kernels <5.8, which lack full cgroup v2 support. The risk arises from incomplete cgroup migration, leaving residual v1 hierarchies that trigger kubelet termination even after partial upgrades. This results in a hybrid, non-functional cgroup configuration.

• Cloud Provider Constraints (e.g., AWS AL2)

Certain cloud providers, such as AWS, enforce cgroup v1 on specific OS images (e.g., Amazon Linux 2). Attempts to enable cgroup v2 via kernel arguments are blocked by AMI-level restrictions. The causal mechanism is AWS's explicit prohibition of v2 configurations, necessitating a full OS migration to compatible images (e.g., AL2023) to resolve the incompatibility.

These scenarios underscore the systemic risk posed by cgroup v1 dependency in Kubernetes 1.35+. The root cause—pervasive v1 adoption in legacy distributions—combined with the kubelet's enforced termination policy, initiates a critical failure cascade during essential cluster operations. Immediate verification of cgroup versions and proactive remediation are imperative to avert cluster-wide outages and ensure operational continuity.

Mitigation and Solutions

Kubernetes 1.35 and later versions enforce a strict compatibility check with cgroup v1, causing the kubelet to terminate immediately upon detection of cgroup v1 on nodes. This behavior stems from the removal of cgroup v1 support in the Linux kernel 5.8 and subsequent Kubernetes enhancements (KEP-5573). Failure to address this incompatibility results in node unavailability, pod scheduling disruptions, and potential cluster-wide outages. This section provides technical solutions and workarounds to ensure cluster stability and prevent downtime.

Verifying cgroup Version Without SSH Access

To assess cgroup compatibility across nodes without SSH access, execute the following command:


for n in $(kubectl get nodes -o name); do printf "%s " "$n"; kubectl debug "$n" -n default -q -i --profile=general --image=busybox -- chroot /host stat -fc %T /sys/fs/cgroup; done


This script iterates over each node, deploys a debug pod, and inspects the cgroup filesystem type. Output of cgroup2fs confirms v2 compatibility, while tmpfs indicates v1. Critical implementation details include:

• Chroot Requirement: The chroot /host directive is mandatory. Without it, BusyBox’s stat utility misidentifies cgroup v2 as UNKNOWN due to incorrect filesystem path resolution.
• Resource Cleanup: Debug pods persist as node-debugger-* resources post-execution. Remove them using:


kubectl delete pods -n default -l app=node-debugger


Temporary Mitigation: Disable failCgroupV1 Feature Gate

If immediate OS upgrades are unfeasible, disable the failCgroupV1 feature gate in the kubelet configuration file (/var/lib/kubelet/config.yaml) by setting failCgroupV1: false. This bypasses the cgroup version check, allowing the kubelet to start on v1 nodes. This is a temporary measure, as KEP-5573 mandates its removal by Kubernetes 1.38. The mechanism involves suppressing the kubelet’s termination logic, enabling it to manage pods despite v1 detection.

Permanent Solution: OS Upgrade Requirements

To resolve the incompatibility permanently, upgrade nodes to an OS distribution supporting cgroup v2. Distributions such as CentOS 7, RHEL 7/8, Ubuntu 18.04/20.04, and Amazon Linux 2 default to cgroup v1 and require migration. The causal factors include:

1. Kernel Incompatibility: Kernels prior to 5.8 lack stable cgroup v2 support, rendering v2 unusable on these systems.
2. Mutual Exclusivity: cgroup v1 and v2 cannot coexist on a single node, necessitating a clean OS installation or in-place migration.
3. Cloud Provider Constraints: AWS, for instance, does not support cgroup v2 on Amazon Linux 2, requiring migration to AL2023 or compatible images.

Critical Edge Cases and Risk Mechanisms

Address the following edge cases to mitigate risks:

• Control-Plane Node Failures: On cgroup v1 control-plane nodes, kubelet termination prevents static pods (API server, etcd) from restarting, immediately rendering the control plane non-functional and collapsing the cluster.
• In-Place OS Upgrades: Partial upgrades or kernel updates without full cgroup v2 support trigger kubelet termination, halting pod management and causing workload failures.
• Cloud Provider Lock-In: AWS’s restriction on cgroup v2 for Amazon Linux 2 propagates risk through AMI-level constraints, leaving nodes vulnerable until migration to a compatible OS image.

Practical Remediation Strategies

To minimize disruption, implement the following measures:

• Pre-Upgrade Verification: Assess cgroup versions across all nodes using the provided script before upgrading to Kubernetes 1.35+.
• Scheduled OS Migrations: Plan OS upgrades during maintenance windows, ensuring compatibility with cgroup v2 and Kubernetes requirements.
• Control-Plane Prioritization: Verify and remediate control-plane nodes first, as their failure has the most severe cluster-wide impact.

For a comprehensive checklist and detailed migration steps, refer to this resource.

Conclusion and Actionable Guidance

Kubernetes 1.35+ enforces a critical compatibility shift by defaulting the failCgroupV1 flag to true. This mechanism triggers an immediate termination of the kubelet process on nodes running cgroup v1. Upon detecting the tmpfs mount at /sys/fs/cgroup/—a definitive marker of cgroup v1—the kubelet invokes os.Exit(1), forcibly halting all pod management operations. This results in static pod failures on control-plane nodes (e.g., API server, etcd), rendering the cluster non-operational.

Critical Insights

• Immediate Operational Risk: Clusters running Kubernetes 1.35+ on cgroup v1 nodes face systemic failure due to kubelet termination. This risk is compounded by the pervasive adoption of cgroup v1 in legacy distributions (CentOS 7, RHEL 7/8, Ubuntu 18.04/20.04, Amazon Linux 2).
• Kernel and Distribution Constraints: Kernels below version 5.8 lack stable cgroup v2 support, and cloud providers such as AWS explicitly prohibit cgroup v2 on Amazon Linux 2, necessitating OS migration to resolve compatibility.
• Temporary Mitigation: Setting failCgroupV1: false circumvents the termination check but is a deprecated workaround slated for removal in Kubernetes 1.38, as outlined in KEP-5573.

Technical Remediation Steps

To eliminate risks and ensure long-term cluster stability, execute the following steps in sequence:

1. Accurately Identify cgroup Versions: Deploy the kubectl debug command with chroot /host to reliably determine cgroup versions across nodes, avoiding misidentification of v2 as UNKNOWN. Post-assessment, terminate all `node-debugger-*` pods to reclaim resources.
2. Execute OS Migrations: Transition to cgroup v2-compatible OS images (e.g., CentOS 8, RHEL 9, Ubuntu 22.04, AL2023) during scheduled maintenance windows. In-place upgrades without full cgroup v2 support will trigger kubelet termination, necessitating node replacement or reimaging.
3. Prioritize Control-Plane Remediation: Address control-plane nodes first to prevent cluster-wide outages caused by static pod failures. Ensure etcd quorum is maintained during rolling upgrades.
4. Prepare for KEP-5573 Enforcement: Proactively ensure all nodes are cgroup v2-compatible before Kubernetes 1.38, as the failCgroupV1 escape hatch will be removed, rendering non-compliant nodes inoperable.

Edge-Case Handling

• Cloud Provider Restrictions: AWS’s prohibition of cgroup v2 on Amazon Linux 2 mandates migration to AL2023 or compatible images. Kernel arguments alone cannot bypass this restriction, requiring full OS transition.
• Partial Upgrade Risks: Incomplete OS migrations or kernel upgrades without unified cgroup hierarchy support will trigger kubelet termination, halting pod scheduling and management.

By systematically addressing cgroup compatibility, Kubernetes administrators can preempt catastrophic failures and ensure readiness for future Kubernetes releases. Refer to the comprehensive migration checklist for detailed procedural guidance and edge-case resolution.