Only p99 is slow at low CPU usage: the cause is the CPU limit

When a container has a CPU limit (the amount of CPU the container may use, set with --cpus in Docker and resources.limits.cpu in Kubernetes), it gets a fixed amount of CPU time for each 100ms period. Once it uses that up, the kernel stops every thread in the container until the end of the period. This happens when processing overlaps within one period, and the only requests delayed are the ones being processed when the container is stopped and the ones that arrive while it is stopped.

As long as the container is stopped in only some periods, p50 (the value at the 50% position when response times are sorted from fastest to slowest) barely changes, while p99 (the value at the 99% position) and the rest of the slow end go up. The container uses no CPU while it is stopped, so CPU usage stays low. The number of periods in which the container was stopped is counted in nr_throttled in cpu.stat (a file where the kernel writes cumulative CPU usage counters for each container).

I reproduced this with Docker and checked the conditions under which the container is stopped and the effect of raising the limit.

What is happening

A --cpus 0.5 limit is enforced as up to 50ms of CPU time in each 100ms period.¹ This 50ms is drawn down by the total CPU time used by all threads in the container.

Once the 50ms is used up, every thread in the container stops until the next period starts.² The kernel stopping a container's threads this way is called throttling. A request being processed stops partway through, and a request that arrives during the stop does not start processing until the next period.

Even when average CPU usage is low, requests arriving at random intervals sometimes overlap within one period. If three requests that each use 20ms of CPU overlap in one period, they add up to 60ms, which exceeds 50ms.

The diagram below shows A and B arriving at 0ms, C at 10ms, D at 40ms, E at 100ms, and F at 105ms. Each character is 5ms. # is time spent processing, . is time spent stopped, and - is time spent waiting before processing starts. The number of # in a column is the number of CPUs in use at that point.

ms     0   20  40  60  80  100 120 140 160 180 200
       |                   |                   |
A      ####
B      ####
C        ##................##
D              ------------###.................#
E                          ###.................#
F                           ##.................##

In the 0–100ms period, the # of A, B, and C add up to 10 at the 20ms mark, and the CPU time used reaches 50ms. A and B finish and return, but C, still being processed, is stopped until 100ms. D arrives at 40ms and does not start processing until 100ms.

In the next period, E and F join C and D, which were carried over from the previous period, and the 50ms runs out again at 115ms. C returns at 110ms, but D, E, and F are stopped until 200ms. D returns at 205ms, 165ms after it arrived.

Measurements

I ran an HTTP server that uses 20ms of CPU per request before responding (4 workers) in a container with --cpus 0.5, and sent requests at random intervals from outside the container for 60 seconds.³ I used three average arrival rates: 5, 10, and 20 per second. The throttle rate is the fraction of periods in which the container was stopped: nr_throttled divided by nr_periods (the number of elapsed periods).

Arrivals per second	CPU usage (fraction of limit)	Throttle rate	p50	p99
5	20.5%	2.8%	20.5ms	67.0ms
10	43.5%	12.2%	20.6ms	138.6ms
20	87.1%	70.1%	91.7ms	733.7ms

Even at 5 and 10 per second, where CPU usage stays below 50%, p99 reaches 67.0ms and 138.6ms. p50 stays at 20ms.

Checking your own environment

If the throttle rate rises at the times p99 goes up, the container is being stopped by its CPU limit. If the throttle rate stays at 0, the slowness has a cause other than the CPU limit.

Calculate the throttle rate from cpu.stat. From inside the container, read /sys/fs/cgroup/cpu.stat. From the host, read /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat.⁴

Both nr_throttled and nr_periods are cumulative since the container started, so read them twice while p99 is elevated and divide the increase in nr_throttled by the increase in nr_periods.

Fix

Raise the CPU limit until the throttle rate is close to 0. At an average of 10 arrivals per second, raising the limit to 1.0 brings the throttle rate to nearly 0, and p99 drops to the low 30ms range, the same as at 1.5 and 2.0, where the throttle rate is 0. The container uses about 0.2 cores on average, and the limit needs to be about 5 times that.

--cpus	Throttle rate	p99
0.5	12.2%	138.6ms
0.75	2.1%	44.5ms
1.0	0.38%	33.2ms
1.5	0%	31.1ms
2.0	0%	33.0ms

Raising the limit increases the CPU the container can use at once in periods where requests overlap. Note that at times when the host's CPU has no headroom, the container competes for that extra CPU with the other containers on the same host.

---

1. The kernel 6.17 cgroup v2 documentation on cpu.max. ↩

2. The kernel 6.17 documentation on CFS bandwidth control. ↩

3. GitHub Actions ubuntu-24.04 (4 vCPU), kernel 6.17, cgroup v2, Docker cgroup driver systemd. 294 / 609 / 1228 requests were sent, with 0 errors. Each condition ran in a separate job, so the CPU model differs between jobs. The code is in show-your-work, and the results are in the CI run. ↩

4. This is the location when Docker's cgroup driver is systemd. <id> is the full container ID. ↩