Kubernetes CPU throttling: p99 in seconds, no restarts, and a limit that's too low

Your service is slow and nothing looks wrong. The p99 latency is in seconds, the median is fine, every pod is Running and Ready with 0 restarts, and the nodes are nearly idle. That pattern usually means a CPU limit is throttling the container: it isn't killed, it's paused. Compare kubectl top pod with the CPU limit in kubectl describe pod. If the two numbers match, the pod is capped.

I broke a small Go service on purpose to show this on a real cluster: a "cost review" lowered its CPU limit while a load generator sent it steady traffic. Every output below is copied from that recording (I lined up the columns).

What throttling looks like from outside

A throttled pod passes every health check you'd normally look at. The latency alert fired because the p99 stayed above half a second, but the error rate was flat and the pod looked healthy:

$ kubectl get pods -n go-api
NAME                      READY   STATUS    RESTARTS   AGE
go-api-649df775bd-md4lq   1/1     Running   0          4m15s

The nodes weren't busy either, so this wasn't a neighbour stealing CPU:

$ kubectl top nodes
NAME                    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
k3d-snowops-agent-0-0   151m         3%       860Mi           10%
k3d-snowops-agent-1-0   154m         3%       444Mi           5%
k3d-snowops-server-0    399m         9%       3085Mi          38%

kubectl rollout history showed a second revision, so the pod template had changed. That's the "what changed?" clue, but the history doesn't say what.

Compare top with the limit

kubectl top pod shows what a container uses, and kubectl describe pod shows what it's allowed. Put them side by side:

$ kubectl top pod -n go-api
NAME                      CPU(cores)   MEMORY(bytes)
go-api-649df775bd-md4lq   11m          29Mi

$ kubectl describe pod go-api-649df775bd-md4lq -n go-api
...
    Limits:
      cpu:     11m
      memory:  128Mi
    Requests:
      cpu:     11m
      memory:  32Mi
...

(I cut the rest of the describe output.) 11m of CPU looks like an idle pod. It isn't. It's exactly the limit, and a pod that sits on its limit in every sample is using everything it's allowed. The lab picks this limit from a measurement of your machine, so your number will differ. The match is what matters.

Grafana's Pod Resources dashboard made it plain. The CPU throttled (share of periods) panel had go-api's pod throttled in nearly every period:

Grafana panel "CPU throttled (share of periods)": the go-api pod sits near 100% for the whole five minutes

That panel is the ratio of two cAdvisor counters, which you can graph on any cluster with Prometheus:

sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[2m]))
  / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{container!=""}[2m]))

Why a CPU limit slows a pod instead of killing it

A CPU limit is enforced as a quota of CPU time per scheduling period. The kernel gives the container its limit's share of each period (100 ms by default), and once the container has spent it, the container waits for the next period (kubernetes.io, how pods with resource limits are run). A memory limit is different: go over it and the container is killed with OOMKilled. A CPU limit only holds the container back (kubernetes.io, assign CPU resources).

So most requests land early in a period and finish fast. The unlucky ones arrive after the quota is spent, queue behind the work in progress, and wait one period or several. That's how the median stays flat while the p99 climbs to seconds, with no error anywhere.

Why does kubectl top show low CPU while the pod is throttled?

Because a capped pod can never use more than its cap. kubectl top reports what the container actually used, averaged over a short window (kubernetes.io, resource metrics pipeline). It can't show what the pod wanted. A pod pinned at its limit reads as a small, flat, round number. And a multi-threaded app can be throttled even when its average is below the limit, because several threads can burn a period's quota in a few milliseconds. Watch the throttled ratio, not only usage.

Adding replicas didn't fix it

Scaling out is the usual reflex for a slow service, so I tried it:

$ kubectl scale deploy/go-api --replicas=3 -n go-api
deployment.apps/go-api scaled

$ kubectl top pod -n go-api
NAME                      CPU(cores)   MEMORY(bytes)
go-api-649df775bd-gx4p8   8m           7Mi
go-api-649df775bd-md4lq   9m           25Mi
go-api-649df775bd-n2q6c   9m           7Mi

Each pod got a third of the traffic and still sat just under the same 11m ceiling. The throttled panel showed all three throttled in roughly half or more of their periods:

!The same panel after scaling to three replicas: the three go-api pods are still throttled in about 45% to 100% of periods

The same panel after scaling to three replicas: the three go-api pods are still throttled in about 45% to 100% of periods

The lab's check stayed red: FAIL: go-api/go-api is still not sized for the load it serves. Replicas spread the load, but every pod keeps a ceiling below what it needs, and the cost review that set the limit now pays for three pods instead of one.

Measure first, then set the limit

You can't size a limit while the pod is capped, because kubectl top reads the cap. So the fix has two steps. I scaled back to one replica and lifted the limit far above anything it had used, as a measuring step:

$ kubectl scale deploy/go-api --replicas=1 -n go-api
$ kubectl set resources deploy/go-api -n go-api --limits=cpu=500m
deployment.apps/go-api resource requirements updated

A minute into steady load, kubectl top showed the real demand:

$ kubectl top pod -n go-api
NAME                      CPU(cores)   MEMORY(bytes)
go-api-6d5ddc4d8b-db7gc   24m          7Mi

24m, and 23m a minute later. More than twice the old limit. Then I set it properly: a request near the steady use, so the scheduler is told the truth, and a limit with plenty of headroom for bursts:

$ kubectl set resources deploy/go-api -n go-api --requests=cpu=50m --limits=cpu=200m
deployment.apps/go-api resource requirements updated

$ kubectl rollout status deploy/go-api -n go-api
Waiting for deployment "go-api" rollout to finish: 1 old replicas are pending termination...
deployment "go-api" successfully rolled out

(I left out the scale command's output and one repeated Waiting line.) The request is the pod's guaranteed share of a busy node. The limit is the ceiling.

Proving the fix

The fix shows on the user signal, not only on the resource signal. The server-side p99 fell from about 8 seconds back down to the median:

Grafana "Latency percentiles (server-side)": p99 drops from about 8 s to near zero after the limit is lifted, while p50 stays flat at zero the whole time

On the throttled panel, the old pods' lines ended and the new pod drew no line at all. That panel only draws pods throttled in more than 5% of periods, so the new pod is under that. The lab's check then passed and closed the incident.

Questions people ask next

Should I remove CPU limits altogether?

It's a common choice for latency-sensitive services, and a reasonable one. Without a limit the pod can use idle CPU on the node, and its request still guarantees it a share when the node is busy. Two catches: a namespace LimitRange may add a default limit nobody chose (check kubectl describe limitrange -n <ns>), and a pod only gets the Guaranteed QoS class if its limits equal its requests. Keep a limit where you need either of those. And this is about the CPU limit only: memory can't be throttled, so keep the memory limit.

How much throttling is a problem?

There's no universal threshold, so judge it against latency. In this lab, a limit of 1.5 times the measured demand still left the app throttled in about a sixth of its periods with p99 near 150 ms. At twice the demand, throttling was near zero and p99 went back to a few milliseconds. Graph the throttled ratio next to your p99 and look for the two moving together.

Other causes of slow pods with no restarts

Throttling is the cause where the pod's own CPU reads flat at a round number that equals its limit. If it doesn't, look elsewhere:

• A busy node: kubectl top nodes near 100%, and a neighbour taking the CPU. The fix is placement or requests, not this pod's limit.
• Too few replicas for the traffic: CPU per pod is high but below the limit. That's a job for scaling, or a HorizontalPodAutoscaler.
• A slow dependency: the app waits on a database or another service. Traces or the dependency's own latency show it.
• Memory pressure: a tight memory limit causes garbage-collection pauses before it causes OOMKilled.

Try it yourself

On any test cluster with steady traffic on a Deployment, set a CPU limit far below what it uses:

kubectl set resources deploy/<app> -n <ns> --limits=cpu=10m

Then work through it: p99 against the median, pods, nodes, top against the limit, the throttled ratio, then measure and set.

I ran this drill in SnowOps Labs, an open-source Kubernetes lab where you inject faults like this one into a real app and fix them (GitHub). Disclosure: I build it. It measures your machine before it sets the limit, runs the load for you, and checks the limit you set, not just the moment's latency, so scaling out or stopping the traffic won't pass.

The ladder to remember

1. Compare p99 with the median. Slow but not broken means something is waiting.
2. kubectl get pods and kubectl top nodes: no restarts, no busy node.
3. kubectl top pod against the limit in kubectl describe pod. Equal means capped.
4. Check the throttled ratio (container_cpu_cfs_throttled_periods_total).
5. Lift the limit, measure the real use under load.
6. Set a request near that use and a limit with headroom, or remove the CPU limit.

---

*This is part of a series I'm writing on debugging Kubernetes in production. I am building Snowopslabs which I have used to simulate this incident and resolve it. Check out this tool if you want to play around with Kubernetes incidents and scenarios and learn along the way.