hard‑capped CPU in k8s: stopping silent throttling before it crashes your node

in a typical autoscaling deployment you’ll see pods spin up, hit their request rates, and then the node suddenly starts paging. the symptom is a spike in latency without any OOM kills – a classic silent CPU throttle that the kernel reports only in cgroup stats.

the silent throttle

kubernetes lets you specify requests and limits, but the default cpu limit is soft when you use the cpu shares model on older cgroup v1 setups. the scheduler assumes the node can overcommit, and the kernel throttles containers that exceed their share without raising an alert. the result is a cascade: one noisy neighbor starves others, the node’s load average climbs, and latency budgets break.

a reproducible pod spec

apiVersion: v1
kind: Pod
metadata:
  name: throttle-demo
spec:
  containers:
  - name: app
    image: nginx:stable-alpine
    resources:
      requests:
        cpu: "250m"
      limits:
        cpu: "500m"   # hard cap with cgroup v2
    env:
    - name: LOAD_DURATION
      value: "30"
  - name: load-generator
    image: busybox
    command: ["sh", "-c", "while true; do dd if=/dev/zero of=/dev/null bs=1M count=10; sleep 1; done"]
    resources:
      limits:
        cpu: "500m"

the limits line forces a hard quota when the node runs a recent kernel with cgroup v2 enabled. you can verify the enforcement with kubectl top pod throttle-demo and cat /sys/fs/cgroup/cpu/kubepods/.../cpu.max.

trade‑offs to watch

• over‑provisioning vs latency – setting limits too low guarantees throttling under normal load; too high defeats the purpose and re‑introduces noisy‑neighbor risk.
• cost impact – hard caps improve bin‑packing efficiency, but you may need more nodes to meet peak demand, increasing egress and instance fees.
• observability gap – most dashboards surface CPU usage but not throttling time. instrument your pods with cpu.throttled.time from /sys/fs/cgroup/cpu.stat and forward it to Prometheus; set alerts on a threshold of >5% throttled time over a 5‑minute window.
• failure mode testing – simulate a burst with a temporary load-generator sidecar or a chaos‑engineered pod that spikes CPU. verify that the throttled pod degrades gracefully (e.g., returns 503 with a Retry‑After) rather than timing out.

cost‑aware scaling

pair hard caps with a custom HPA that uses the throttling metric as a scaling signal instead of pure CPU utilization. this prevents the controller from scaling up based on a single hot pod that’s already hitting its quota.

---

how do you validate that your resource limits actually protect the node under load without throttling critical paths