A Go API had a p99 of 1.4 seconds against a median of 18 milliseconds. No slow queries, no lock contention, no garbage collection pauses worth mentioning. The CPU dashboard showed the pods sitting at roughly 35 percent of their limit, all day, never close to the ceiling. For a week I believed the problem could not be CPU, because the graph said there was plenty left.
The graph was an average over thirty seconds. The kernel does not average over thirty seconds. With limits.cpu: 500m, the container gets a CFS quota of 50 milliseconds of CPU time per 100 millisecond period, shared across every thread in the cgroup. The service ran on 64-core nodes, so the Go runtime set GOMAXPROCS to 64 and happily scheduled work on dozens of threads at once. Those threads burned the entire 50 millisecond quota in about two milliseconds of wall clock, and then every thread in the container was descheduled until the next period began. Any request unlucky enough to be in flight ate a 98 millisecond stall, sometimes several in a row.
container_cpu_cfs_throttled_periods_total over total periods was 41 percent. That metric was available the whole time and on nobody's dashboard. Averaged utilisation of 35 percent and throttling in 41 percent of periods are completely consistent, which is the part that had seemed impossible: the container is idle most of each period precisely because it is being held down.
Three changes fixed it. We set GOMAXPROCS from the cgroup quota using automaxprocs, so the runtime believes it has the two cores it actually has and stops creating burst parallelism it cannot spend. We raised limits to roughly twice the observed steady-state request rather than sizing them by guesswork. And on the two latency-sensitive services we removed the CPU limit entirely, keeping requests, which makes them burstable and relies on requests for scheduling fairness.
The throttling ratio is now on the service dashboard next to utilisation, with an alert above five percent.
A CPU limit is not a ceiling on how much you use. It is a hard stop you hit in the middle of a request, and average utilisation will never show it to you.
– Sergey Shinder
Top comments (0)