Sharing IO: io.max, io.weight and io.latency

When one process is choking the disk on a machine, the first reflex is always the same: "let's set a limit". Then someone writes an io.max into a cgroup directory, the backup job is capped at 50 MB/s, and everyone relaxes. A few weeks later the backup window starts overrunning — while the disk sits idle most of the time.

The reason for that story is that cgroup v2 doesn't have a single way to share IO. The kernel documentation lists four: absolute limits (blk-throttle), weight-based proportional distribution (iocost, or BFQ's own cgroup support when the BFQ scheduler is in use), latency-based protection, and IO priority assignment. This article covers the first three, because that trio drives the decision; the fourth (io.prio.class) is worth keeping in your pocket for cheap moves like putting background jobs into the idle class.

Three mechanisms, three questions: io.max says "this group must never exceed this rate", io.weight says "when the disk is contended, split the share in this proportion", and io.latency says "keep this group's latency under this target, slowing others down if necessary".

My argument: most teams set up the first one when what they need is one of the other two. A hard cap doesn't protect the system — it only punishes a group and throws idle capacity away.

First, see what you actually have

Let's start with my own server: Ubuntu 24.04, a single SSD root disk (sda, major:minor 8:0) and cgroup v2.

$ cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc

$ cat /sys/fs/cgroup/system.slice/io.weight
default 100

$ cat /sys/fs/cgroup/system.slice/io.stat
7:5 rbytes=20272128 wbytes=0 rios=758 wios=0 dbytes=0 dios=0
7:4 rbytes=54236160 wbytes=0 rios=1842 wios=0 dbytes=0 dios=0
7:3 rbytes=755712 wbytes=0 rios=107 wios=0 dbytes=0 dios=0
7:2 rbytes=20448256 wbytes=0 rios=693 wios=0 dbytes=0 dios=0
7:1 rbytes=1116160 wbytes=0 rios=59 wios=0 dbytes=0 dios=0
7:0 rbytes=8969216 wbytes=0 rios=896 wios=0 dbytes=0 dios=0
8:0 rbytes=835511135232 wbytes=1563887470592 rios=43331841 wios=133872437 dbytes=699905486848 dios=352412

$ cat /sys/fs/cgroup/system.slice/io.pressure
some avg10=0.08 avg60=0.05 avg300=0.10 total=7877797953
full avg10=0.00 avg60=0.00 avg300=0.04 total=6108319475

The first thing to learn from that output isn't policy, it's reading discipline: the first six lines are 7:x — loop devices, the mount points of snap packages. The real disk is at the bottom, 8:0. Taking a device number from the first line of io.stat is the easiest way to write a rule against the wrong device. systemd's documentation says the same thing separately: these settings should refer to the originating physical device, not to partitions or loopback devices.

The io.cost.qos and io.cost.model files come back empty — I can't show their content in this article because there is nothing to show.

What the picture says: the IO controller is enabled, accounting works, pressure is measured — but there is no policy at all. io.weight sits at the default 100.

The gap has more than one cause. Per the kernel documentation, io.cost.qos and io.cost.model exist only on the root cgroup, and the line for a device is populated on the first write to one of those files. But that write isn't always yours: since systemd 254, udev's iocost tool fills in io.cost.* at boot on block devices for which it finds a matching solution in hwdb. My virtual disk has no match — udevadm info /dev/sda | grep -i IOCOST comes back empty — which is why the files are empty. Don't be surprised if yours are populated; it may have configured itself.

Three mechanisms, three questions

io.max: the hard ceiling

(The io.low you'll see in older guides is gone; that blk-throttle experiment was removed from the kernel.)

io.max is a nested file keyed by device number, taking rbps, wbps, riops and wiops. IOs are delayed once the limit is reached, and temporary bursts are allowed. For example, capping device 8:0 at 2 MB/s reads and 120 write IOPS:

echo "8:0 rbps=2097152 wiops=120" > /sys/fs/cgroup/backup.slice/io.max

The critical property is stated plainly in systemd's own documentation: this limit is not work-conserving, and the processes are not allowed to use more even if the device has idle capacity. So at 3 a.m., with the disk completely idle, your backup still crawls at 50 MB/s.

There's also persistence: a value you write into a cgroup file by hand can be overwritten by systemd on the next daemon-reload or when the unit restarts. The durable route is systemd's own directives — IOReadBandwidthMax=, IOWriteBandwidthMax=, IOReadIOPSMax=, IOWriteIOPSMax=:

systemctl set-property backup.slice IOWriteBandwidthMax="/dev/sda 50M"

That makes io.max a safety valve rather than a performance tool: it's right where predictability matters more than speed, when you want to say "this workload must never exceed this, whatever happens". Per-tenant ceilings in a rented environment, for instance.

io.weight: fairness, but only with a cost model

What's usually wanted is this: as long as the disk isn't contended everyone runs free, and when it is, the share is split by weights. That's io.weight, and the engine behind it is iocost.

There's a setup step here that gets missed: for weights to mean anything you need one of two things. Either the device's scheduler is BFQ — in which case BFQ's own cgroup support handles the distribution — or iocost is enabled. On my server the scheduler is none:

$ cat /sys/block/sda/queue/scheduler
[none] mq-deadline

So there's no BFQ here; if I want weights, I have to enable iocost. The controller is disabled by default and enabled by setting enable to 1.

# turn on weight-based control for device 8:0
echo "8:0 enable=1 ctrl=auto" > /sys/fs/cgroup/io.cost.qos

ctrl=auto means the parameters are managed by the kernel. And when rpct and wpct are left at zero, the controller uses the device's internal saturation state to adjust the overall IO rate between the min and max percentages. If you need device-specific coefficients, tools/cgroup/iocost_coef_gen.py in the kernel tree exists for that. When you need better control quality you can specify latency QoS parameters by hand — the documentation's example reads enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=..., i.e. scaling against a target like "the 95th percentile of reads shouldn't exceed 75 ms".

The systemd equivalent is IOWeight=: it takes a value between 1 and 10000 and controls the io.weight attribute directly (which defaults to 100). The share is proportional: a unit's share is its weight divided by the sum of weights within the same slice. So IOWeight=500 means five times only if every neighbour sits at the default 100; raise the neighbours to 500 too and nothing changes.

io.latency: protection, not punishment

The third one is the least known: the most direct tool when you want to protect a single service. io.latency defines a target latency and is work conserving — as long as everybody meets their target the controller does nothing. The syntax resembles the others:

echo "8:0 target=25000" > /sys/fs/cgroup/database.slice/io.latency   # 25 ms

Once a group starts missing its target, its peers get throttled. Two details here are the key to setting it up correctly.

First: throttling is applied only at the peer level. In the documentation's example, A, B and C under the root influence each other; D and F under A influence each other; groups in different branches never touch. Translated to systemd: two services under system.slice are peers, system.slice and user.slice are peers at the root, and two services in different slices don't affect each other at all. Put the target at the wrong level and nothing happens — and nothing errors either.

Second, and less known: the throttled peers don't need targets of their own. The comment in the kernel source is explicit — groups throttled as collateral have min_lat_nsec == 0, i.e. groups where io.latency was never set. In practice: you protect one service, and its target-less neighbours back off.

The throttling takes two forms: queue depth throttling (the number of outstanding IOs a group may have, clamped down all the way to one IO at a time) and artificial delay induction. The second is interesting, because IO types that can't be throttled — swapping and metadata IO — are allowed to proceed normally but are "charged" to the originating group.

For picking a target the guidance is clear: don't set a value lower than the latency your device supports. On rotational devices use the avg_lat value in io.stat as a basis and set it 10-15% higher; on non-rotational devices no average latency is reported, so set the target from your device and verify with the missed/total fields. But run this line first, or you'll never see those fields:

echo 1 > /sys/module/blk_cgroup/parameters/blkcg_debug_stats

Those debug stats are disabled by default; when latency fields are missing from io.stat, the usual reason isn't a broken setting but a disabled measurement.

The systemd equivalent is IODeviceLatencyTargetSec=, which takes a device and a timespan, e.g. /dev/sda 25ms.

A footnote: the introductory sentence in the kernel documentation says it throttles peers with a lower target, while its own "How IO Latency Throttling Works" section says higher. The source code supports the second; if you trip over that contradiction while reading, that's why.

The layer that's easy to skip: writeback

Everything above assumes IO can actually be charged to the right cgroup. With buffered writes that assumption collapses easily.

Page cache is dirtied through buffered writes and shared mmaps, then written asynchronously by the writeback mechanism. The documentation treats this layer separately: the io controller, together with the memory controller, implements control of page cache writeback IOs — the memory controller defines the memory domain for which the dirty ratio is maintained, and the io controller defines the IO domain that writes those pages out. Both system-wide and per-cgroup dirty memory states are examined and the more restrictive of the two is enforced.

And the critical sentence: cgroup writeback requires explicit support from the underlying filesystem. The documentation lists the supported ones — ext2, ext4, btrfs, f2fs and xfs.

In practice that means: if your filesystem isn't on that list, or the memory controller isn't enabled on the relevant hierarchy, buffered writes reach the disk with unclear ownership and your IO policy is largely ineffective on the write side. A good share of "I set a limit and the disk still fills up" complaints come from exactly here.

Which one, when?

Three questions, in order:

Is this workload's ceiling defined by a contract? A per-tenant quota, IOPS sold to a customer, a "backups must never exceed this" policy — if yes, io.max. Set it knowing it can't use idle capacity.

Is your problem the split under contention? Several services sharing a disk and the sentence "the database matters more than the log shipper" — if yes, io.weight (IOWeight= on the systemd side). But turn iocost on first, or the weights are decoration.

Is there a specific latency you want to protect? A user-facing service, a p99 target, "this service stays under 25 ms whatever the background jobs do" — if yes, io.latency.

Let's be precise about combinations: io.weight and io.latency are two answers to the same question, and stacking them is pointless. Combining io.max with either of them, on the other hand, is perfectly sensible — a contractual ceiling plus behaviour under contention. Still: measure first, then start with one mechanism.

On container hosts, think about this trio alongside CPU and memory limits; the diagnostic flow in my article about spotting and limiting a resource-hog container on a VPS is the starting point. If storage latency under the virtualisation layer is in play, the questions in my piece on storage IO latency battles in legacy infrastructure apply too: the problem may not be cgroup policy but a shared queue underneath.

Measurement: where to look

• io.stat → bytes and operations read/written per group; this is where you see which device is really in use.
• io.pressure → pressure, but read it correctly. The averages (avg10/60/300) decay; an avg300 read at noon says nothing about the backup window at midnight. The right method is to take the delta of total across the window you care about. And for IO the full line is a sharper signal than some: full counts the time when every task in the system was waiting.
• If you set io.latency, the missed/total fields in io.stat → is the target actually being met?
• After the change, the workload's own metric → did the restriction protect what you wanted, or just move the problem?

As a checklist: get the device numbers ($MAJ:$MIN) right and confirm with lsblk that the device is the disk you meant; enable iocost if you're going to use weights; check the filesystem's cgroup writeback support for write-heavy workloads; and justify every policy with a pressure measurement first.

Setting a limit isn't solving the problem

Putting these three files side by side, what I see is the general state of resource management: the hard limit is the easiest tool to understand and the least useful. It's understandable because you give a number and the system obeys. It's not useful because the system's real problem isn't average throughput, it's behaviour under contention.

If there's no contention you need no policy at all — as on my server. If there is, the question isn't "how many MB/s" but "who comes before whom, and what am I protecting?" The moment you know that answer, which file to write into is already decided.

Official Sources

• Linux kernel: cgroup v2 — the IO controller, writeback and io.latency
• The source document in the kernel tree (cgroup-v2.rst)
• systemd: resource control settings (IOWeight, IODeviceLatencyTargetSec)