DEV Community

Cover image for Your CPU Quota Is a Budget, Not a Percentage: Practical cgroup v2 CPU Control on Linux
Lyra
Lyra

Posted on

Your CPU Quota Is a Budget, Not a Percentage: Practical cgroup v2 CPU Control on Linux

#linux #devops #systemd #performance

A batch job gets CPUQuota=50%. The box still goes to 100% CPU, the API on the same host starts missing deadlines, and someone "fixes" it by raising the quota to 200%. Now the batch job is eating two full cores and the API is still slow.

The number felt like a percentage of the machine. The kernel treats it as microseconds of CPU time per period. Those are the same thing only if you know the period, how many threads are runnable, and what happens when the budget runs out. That last part is what usually surprises people: a cgroup that exceeds its quota is not slowed down proportionally. It is stopped until the next period starts.

cgroup v2's CPU controller has three separate mechanisms that answer three different questions. Mixing them up is why CPU limits feel unpredictable.

Three questions, three knobs

What you actually want systemd setting Kernel interface
"This unit must never exceed X CPU" CPUQuota= cpu.max
"Under contention, this unit should win" CPUWeight= cpu.weight
"This unit may only run on these cores" AllowedCPUs= cpuset.cpus
"Only use CPU nobody else wants" CPUWeight=idle cpu.idle

They compose, but they are not interchangeable. A quota is enforced by throttling. A weight is enforced by the scheduler's fairness math and does exactly nothing when the machine is idle. A cpuset changes placement, not capacity.

The arithmetic of cpu.max

The file is two numbers:

$ cat /sys/fs/cgroup/system.slice/ffmpeg-worker.service/cpu.max
200000 100000
Enter fullscreen mode Exit fullscreen mode

The format is $MAX $PERIOD, both in microseconds. This group may consume up to 200 ms of CPU time in every 100 ms period, which is two CPUs' worth. The default is max 100000: unlimited quota, 100 ms period. Writing a single number updates only $MAX; writing max removes the limit.

So the conversion is:

CPUs = MAX / PERIOD
Enter fullscreen mode Exit fullscreen mode
cpu.max Meaning
max 100000 Unlimited
100000 100000 1 CPU
25000 100000 25% of one CPU
200000 100000 2 CPUs
50000 50000 1 CPU, shorter 50 ms period

systemd's CPUQuota= is a percentage of one CPU, which is why the man page tells you to use values above 100% for more than one CPU. CPUQuota=150% becomes 150000 100000. CPUQuotaPeriodSec= changes the second field; it defaults to 100 ms and is clamped to the range the kernel supports, 1 ms to 1000 ms.

The kernel's own floor is 1 ms for quota or period, and the period cannot exceed 1 s. Those limits come from Documentation/scheduler/sched-bwc.rst.

Why a quota feels like stuttering, not slowing

CFS bandwidth control does not ration CPU smoothly. Within a period, the group's quota is handed out to per-CPU run queues in slices as threads become runnable. When the budget is gone, every thread in the group is throttled until the next period refills it.

Picture CPUQuota=25% on a machine with free cores. The group gets 25 ms of runtime per 100 ms. A single busy thread runs for 25 ms, then sits parked for 75 ms. Average utilisation is 25%, but the shape is a 25 ms burst followed by a 75 ms stall. If that thread is serving requests, p99 latency now includes the stall.

I ran exactly this on a 3-vCPU Debian 13 host (kernel 6.12, systemd 257):

systemd-run --user --unit=cpu-lab --property=CPUQuota=25% \
  /bin/bash -c 'while :; do :; done'
Enter fullscreen mode Exit fullscreen mode 
systemctl --user show cpu-lab -p CPUQuotaPerSecUSec -p ControlGroup
# CPUQuotaPerSecUSec=250ms
# ControlGroup=/user.slice/user-1000.slice/user@1000.service/app.slice/cpu-lab.service
Enter fullscreen mode Exit fullscreen mode

The kernel file matched the arithmetic:

$ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/cpu-lab.service/cpu.max
25000 100000
Enter fullscreen mode Exit fullscreen mode

Sampling CPUUsageNSec over a 4.02 s window showed 1.005 s of CPU time consumed, or 25.0% of one CPU. Meanwhile cpu.stat showed the group had been throttled in 30 of 31 periods. It was not running slowly. It was running fast, then not at all.

That distinction drives most of the practical advice:

  • Latency-sensitive services usually want a generous quota plus a weight, not a tight quota. A tight quota converts load into tail latency.
  • Batch work that must not disturb anything else is the opposite case: a tight quota is correct, and stalls do not matter.
  • If you need finer-grained enforcement, shorten the period. Below ~10 ms you pay more accounting overhead for little gain, and systemd will not go below 1 ms.

Read the counters before changing anything

cpu.stat is where the truth is:

$ cat /sys/fs/cgroup/.../cpu-lab.service/cpu.stat
usage_usec 792598
user_usec 781696
system_usec 10901
nr_periods 31
nr_throttled 30
throttled_usec 2247452
nr_bursts 0
burst_usec 0
Enter fullscreen mode Exit fullscreen mode

The first three counters always exist. nr_periods, nr_throttled, throttled_usec, nr_bursts, and burst_usec appear when the controller is enabled, and only account for fair-class scheduling.

Two warnings about interpretation:

  • These five counters are not hierarchical. They report throttling caused by this cgroup's own limit, not throttling inherited from a parent slice. For the time actually lost including ancestors, read cpu.stat.local.
  • For a fully CPU-bound group, nr_throttled roughly equals nr_periods. That is normal, not a bug: the group always uses its whole budget and always hits the wall. A ratio near zero means the quota is not binding and the limit is decoration.

A useful rule of thumb: if throttled_usec divided by wall-clock time is above roughly 20%, the quota is shaping your workload's behaviour in a way users will feel.

Burst: borrow against future quiet time

cpu.max.burst (kernel file) lets a cgroup accumulate unused quota and spend it later, up to the value you set. Default is 0; the valid range is 0 to the current $MAX. nr_bursts and burst_usec in cpu.stat count the periods where it happened.

The kernel documentation describes this as borrowing against future underrun, and it is explicitly a trade: you keep the average, accept short interference with other groups, and stop throttling on spikes that are followed by idle time. It is a good fit for request-driven services with bursty CPU and a bad fit for a hard isolation requirement.

Not every kernel exposes the file. Check with ls in the cgroup directory before you plan around it.

cpu.weight is fairness, not a cap

cpu.weight lives in the range 1 to 10000, default 100. It only distributes CPU time among siblings that have runnable tasks. A lone service with weight 100 on an idle box still gets every core. Weight changes nothing until there is contention, and then it changes everything about who waits.

The split is proportional to weight among the children of the same parent. It is not transitive: two services inside system.slice compete with each other, but their weights do not let them outrank a unit in user.slice; the slice weights decide that level.

I measured a 100 vs 300 pair on 3 vCPUs, each unit running four busy threads so both would genuinely contend:

for w in 100 300; do
  systemd-run --user --unit="demo-$w" --property=CPUWeight=$w \
    /bin/bash -c 'for i in 1 2 3 4; do while :; do :; done & done; wait'
done
Enter fullscreen mode Exit fullscreen mode

Over a 6 s window: 4.52 s of CPU time for weight 100 and 13.47 s for weight 300. That is 0.75 CPU versus 2.24 CPU, a ratio of about 1:3, adding up to almost exactly the 3 vCPUs available. Weights divided the machine in proportion to 100:300.

Two more details matter in production:

  • cpu.weight.nice is an alternative view using the familiar −20..19 range. It is a coarser mapping onto the same weight.
  • CPUWeight=idle sets cpu.idle, which makes the cgroup run only when nothing else in the same parent or its siblings wants the CPU. This is the right setting for backup indexers, thumbnailers, and anything that should consume leftovers.

cpuset: placement, partitions, and a delegation trap

cpuset.cpus restricts which CPUs a group's tasks may run on. systemd exposes it as AllowedCPUs= (since v244). The requested set is not the granted set: the effective set is the intersection with every ancestor, readable as EffectiveCPUs= in systemd and cpuset.cpus.effective in the kernel.

[Service]
AllowedCPUs=2-5
Enter fullscreen mode Exit fullscreen mode

On a 3-vCPU host, I tried AllowedCPUs=0 on a user unit and got EffectiveCPUs= empty with no cpuset.cpus file at all. The per-user manager on that host had only cpu memory pids in cgroup.controllers, so the cpuset controller was never available to apply the setting. The configuration was accepted and silently did nothing.

That is the trap: always verify placement with EffectiveCPUs, and run cpuset experiments as system units (or inside a delegated container cgroup) where the controller exists.

Beyond simple pinning, cgroup v2 supports cpuset partitions. Writing root to cpuset.cpus.partition turns a cgroup into a scheduling domain with exclusive CPUs from cpuset.cpus.exclusive. Writing isolated also removes scheduler load balancing and keeps unbound workqueues off those CPUs. Partitions can end up in an invalid state if the exclusive set overlaps a sibling's, and the file then reads back something like root invalid (<reason>) — which is your cue to fix the hierarchy rather than the workload.

Use this when you genuinely want dedicated cores: latency-critical inference, DPDK-style packet processing, real-time-ish audio. Do not use it to "reserve" capacity you are not actually using; an isolated core with no runnable task is just an idle core nobody else may borrow.

cpuset.mems is the NUMA half of the same controller. On multi-socket boxes, pinning CPUs without pinning memory can send every access across the interconnect.

Wiring it into a unit, then proving it

A drop-in keeps the distribution unit file untouched:

# /etc/systemd/system/ffmpeg-worker.service.d/20-cpu.conf
[Service]
CPUAccounting=yes
CPUQuota=200%
CPUQuotaPeriodSec=100ms
CPUWeight=200
AllowedCPUs=0-1
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl daemon-reload
sudo systemctl restart ffmpeg-worker.service
Enter fullscreen mode Exit fullscreen mode

For experiments you do not want to keep:

sudo systemctl set-property --runtime ffmpeg-worker.service CPUQuota=150%
# drop --runtime to persist the change as a drop-in
Enter fullscreen mode Exit fullscreen mode

Verify the effective configuration rather than the file you wrote:

systemctl show ffmpeg-worker.service \
  -p CPUQuotaPerSecUSec -p CPUWeight -p AllowedCPUs \
  -p EffectiveCPUs -p CPUUsageNSec
Enter fullscreen mode Exit fullscreen mode

CPUQuotaPerSecUSec=1.5s means 150% of one CPU. Empty EffectiveCPUs means your cpuset request is not being applied. And a live view of who is actually consuming:

systemd-cgtop -b -c -1 --depth=4
Enter fullscreen mode Exit fullscreen mode

Note the flags: -b is batch mode, -c orders by CPU, -1 runs a single iteration. (-m orders by memory, which is a different thing.)

A reproducible lab

This script reports a unit's achieved CPU usage as a percentage of one CPU. It uses systemctl by default; set SYSTEMCTL="systemctl --user" to sample user units.

#!/usr/bin/env bash
# cpu-usage.sh <unit> [seconds]
set -euo pipefail
unit="${1:?usage: cpu-usage.sh <unit> [seconds]}"
secs="${2:-10}"
sctl="${SYSTEMCTL:-systemctl}"

a=$($sctl show "$unit" -p CPUUsageNSec --value)
t0=$(date +%s%N)
sleep "$secs"
b=$($sctl show "$unit" -p CPUUsageNSec --value)
t1=$(date +%s%N)

awk -v a="$a" -v b="$b" -v e="$((t1 - t0))" -v u="$unit" 'BEGIN {
  printf "%s: %.1f%% of one CPU over %.2fs\n", u, (b - a) / e * 100, e / 1e9
}'
Enter fullscreen mode Exit fullscreen mode

Full quota test:

systemd-run --user --unit=cpu-lab --property=CPUQuota=25% \
  /bin/bash -c 'while :; do :; done'
sleep 3
SYSTEMCTL="systemctl --user" ./cpu-usage.sh cpu-lab 10
systemctl --user stop cpu-lab
Enter fullscreen mode Exit fullscreen mode

Expected: roughly 25%, with nr_throttled close to nr_periods. If you see 100%, the controller is not active or the quota did not land; check cpu.max directly.

Then change the quota live and watch the number follow:

systemctl --user set-property --runtime cpu-lab CPUQuota=75%
Enter fullscreen mode Exit fullscreen mode

Failure modes worth checking first

  • A quota below one request's CPU cost. The request cannot finish inside its slice, gets throttled, and latency explodes. Raise the quota, shorten the period, or accept that the service needs a weight instead.
  • "50% of the machine" thinking. CPUQuota=50% is half of one CPU, total, across all threads. On a 32-core host that is 1.5% of capacity.
  • Weights on an idle box. Nothing happens, correctly. Weight is a contention policy.
  • Weights across slices. A service cannot outrank a sibling slice by weight alone; the parent slice's weight sets the ceiling.
  • cpuset without the controller. AllowedCPUs= is accepted, EffectiveCPUs stays empty, and the service runs everywhere. Check delegation.
  • Real-time threads. cgroup v2's CPU bandwidth control does not cover real-time tasks in the way you might assume; the kernel documentation carries an explicit warning about CONFIG_RT_GROUP_SCHED and RT processes in non-root cgroups.
  • Autogrouping. sched(7) describes automatic session-based group scheduling, which behaves similarly to weights but is a different mechanism. Do not debug one while looking at the other.
  • Container runtimes writing the same file. A Kubernetes CPU limit and a systemd drop-in both end up in cpu.max. The last writer wins. When numbers disagree, cat the file.
  • Steal time on VMs. Quota is enforced against vCPU time. A guest can be far below its quota and still slow because the hypervisor is not scheduling it.

Rollback 

sudo rm /etc/systemd/system/ffmpeg-worker.service.d/20-cpu.conf
sudo systemctl daemon-reload
sudo systemctl restart ffmpeg-worker.service
# or, to drop all local overrides for the unit:
sudo systemctl revert ffmpeg-worker.service
Enter fullscreen mode Exit fullscreen mode

An empty CPUQuota= unsets the quota; cpu.max returns to max 100000. CPUWeight= unset returns the kernel default of 100.

CPU control is one of the few places where the kernel's model is simpler than the folklore. A quota is a per-period budget that ends in a hard stop. A weight is a tie-breaker that only matters when someone else wants the same cycles. A cpuset is placement. Pick the one that matches the promise you are making, then read cpu.stat and EffectiveCPUs to confirm the promise is being kept.

Sources

Top comments (0)