Fix Kubernetes CPU Throttling & CFS Quotas

#devops #linux #kubernetes #sre

The P99 latency alert fires while Grafana reports 20% CPU usage. Welcome to Kubernetes CPU throttling, where Linux CFS Quotas forcibly freeze containers in microscopic 100ms windows despite low average usage.

Here is the complete SRE blueprint to detect micro-freezes, tune runtimes, and bypass CFS limits using bare-metal static core pinning.

Phase 1: The Dashboard Illusion (100ms CFS Quota)

Standard CPU metrics average usage over 1 to 5 minutes, hiding microscopic kernel freezes. Kubernetes CPU limits are actively enforced by the Linux kernel's Completely Fair Scheduler (CFS) in 100-millisecond windows.

100m Limit: Grants 10ms of execution time per 100ms window.
500m Limit: Grants 50ms of execution time per 100ms window.

When a bursty, multi-threaded application consumes its 100ms allocation in the first 10ms, the kernel forcibly freezes the container for the remaining 90ms. Your dashboard reports low average CPU usage, but your application was completely dead for 90% of that second.

Phase 2: Detecting Throttling via PromQL

Stop tracking raw CPU usage (container_cpu_usage_seconds_total). Instead, track the percentage of 100ms windows where the kernel parked your application:

# Throttling Ratio Query (%)
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{container!="", container!="POD"}[5m])
) 
/ 
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{container!="", container!="POD"}[5m])
) * 100

🚨 SRE Rule: If this metric exceeds 15–25%, your application is suffering from artificial latency spikes caused by cgroup throttling.

Phase 3: Fixing Thread Amplification (JVM & Go)

On a bare-metal node with 64 physical cores, Java and Go runtimes inspect the host OS, detect 64 cores, and spawn 64 Garbage Collection or worker threads. If your container limit is set to 2 vCPUs, all 64 threads wake up simultaneously and burn through your 100ms quota in milliseconds.

1. Java / JVM (JDK 11+)

Rely on UseContainerSupport so the JVM auto-detects container cgroup limits:

ENV JAVA_OPTS="-XX:+UseContainerSupport"

2. Golang

Import Uber's automaxprocs library to auto-tune GOMAXPROCS to cgroup limits rather than host core counts:

import _ "go.uber.org/automaxprocs"

Phase 4: The 2x P99 Right-Sizing Rule

Removing CPU limits entirely leaves nodes vulnerable to CPU Exhaustion DoS attacks and Node Starvation risks. Instead of removing limits, right-size using telemetry:

Measure P50 (median) CPU usage over 7 days $\rightarrow$ Set as requests.cpu.
Measure P99 (peak burst) CPU usage over 7 days $\rightarrow$ Multiply by 2 $\rightarrow$ Set as limits.cpu.

Phase 5: Eradicating Throttling on Bare Metal

Public Cloud VMs suffer from "Double Throttling"—K8s CFS limits combined with Hypervisor Steal Time from noisy neighbors.

On ServerMO Dedicated Bare Metal, enable Kubelet's cpuManagerPolicy: static. By deploying pods in the Guaranteed QoS class (requests equal limits using integer CPU values), Kubernetes completely bypasses the CFS 100ms quota system and pins the container directly to dedicated physical CPU cores.

apiVersion: v1
kind: Pod
metadata:
  name: high-performance-api
spec:
  containers:
  - name: api
    image: enterprise-api:v1.2
    resources:
      limits:
        cpu: "4"        # Integer value required
        memory: "8Gi"
      requests:
        cpu: "4"        # Must equal limits for Guaranteed QoS
        memory: "8Gi"

👉 Read the full SRE guide on ServerMO:

Stop Kubernetes CPU Throttling | ServerMO

DEV Community