The CPU Graph Said 40%. We Were Being Throttled Every Second.

The problem

3 AM, paged for p99 latency on a checkout service that was, by every dashboard we had, fine. Average CPU utilization: 38%. Memory: comfortable. Error rate: near zero. And yet roughly one request in twenty was taking 800ms-1200ms instead of the usual 40ms, in bursts that lasted a few seconds and then vanished.

We did what everyone does first. Checked GC logs — no long pauses. Checked the database — query times were flat. Checked the network — no retries, no timeouts upstream. Checked lock contention in the app — nothing held for more than a few microseconds. Every obvious suspect had an alibi.

The graph that was supposedly clearing us — CPU usage — was the thing lying to us.

Why it happens

Kubernetes (and Docker, and anything built on cgroups) doesn't enforce a CPU limit as "don't average more than X cores." It enforces it as a hard budget inside the CFS (Completely Fair Scheduler) bandwidth controller: every 100ms accounting period, your container gets a fixed slice of CPU-time — say 200ms of combined core-time for a "2 CPU" limit — and once that slice is spent, every thread in the container is frozen until the next period starts. No exceptions, no borrowing from a quiet neighbor period.

That period is 100ms. Your dashboard is almost certainly scraping and averaging over 15s, 30s, or 60s windows. A service can burn its entire 100ms quota in the first 8-10ms of a period — because a burst of concurrent requests briefly spun up enough threads to blow through the budget — sit frozen for the remaining ~90ms, and repeat that dozens of times a second. Averaged over a minute, that's "38% CPU usage." Inside any individual 100ms window, it's 100% then 0%, over and over, and every request that happened to land in a frozen window paid for it in latency.

This is the part that trips up even experienced engineers: CPU limits cap peak usage inside a tiny accounting window, not usage averaged over the window your monitoring actually shows you. The two numbers can point in completely opposite directions.

What to do about it

The metric almost nobody has on a dashboard is throttling, not usage. On the node:

cat /sys/fs/cgroup/cpu.stat
# nr_periods 48213
# nr_throttled 9104
# throttled_usec 182004112

nr_throttled divided by nr_periods is your throttling ratio. Ours was north of 18% — meaning nearly one in five scheduling periods, the container was frozen mid-request. If you're on Kubernetes with cAdvisor/Prometheus, the same signal is container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total:

rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])

Anything consistently above ~5% is worth investigating; above 20-25%, it's very likely a real contributor to tail latency, not a coincidence.

Once you can see it, the fixes are unglamorous but effective, roughly in order of how fast they help:

1. Raise the CPU limit (or remove it and rely on requests + node-level capacity). Immediate relief, costs more compute headroom.
2. Right-size your concurrency. If your runtime spins up more worker threads than your quota can actually sustain in a 100ms burst, you're manufacturing your own throttling. Cap thread/worker pool size to something the quota can plausibly cover.
3. Watch the ratio, not the average. Add the throttling metric to the same dashboard as CPU usage so the two numbers can't tell different stories unchallenged.

We ended up doing both 1 and 2 — a modest limit bump plus trimming an overeager thread pool that had been sized for a machine, not a slice of one. The throttling ratio dropped from ~18% to under 1%, and the "random" p99 spikes went with it.

Key takeaways

• A CPU limit is a hard budget per 100ms accounting period, not a cap on the average your dashboard shows you.
• Low average CPU usage does not rule out throttling — bursty workloads can be throttled constantly while looking idle on a 1-minute chart.
• Check nr_throttled in cpu.stat (or container_cpu_cfs_throttled_seconds_total in Prometheus) directly — it's a different signal than usage, and most teams never graph it.
• If concurrency inside a container regularly exceeds what its quota can sustain in one period, you're throttling yourself before any external load shows up.