DEV Community

Cover image for ionice: none Ignores It, mq-deadline Reads Class, BFQ Level
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

ionice: none Ignores It, mq-deadline Reads Class, BFQ Level

#linux #kernel #ioscheduler #ionice

Three days ago a lesson landed in our knowledge base: don't run long disk scans like du -xh during a deploy, and if you must, run them under nice -n 19 ionice -c3. Sound advice; I've been writing the same line for years. Except this morning on VPS3 I re-read a line I had already printed two weeks ago in the io.max post without ever connecting it to ionice, and realised that half of the recipe does nothing on this machine:

$ cat /sys/block/sda/queue/scheduler
[none] mq-deadline
Enter fullscreen mode Exit fullscreen mode

none means exactly that: no scheduler. If nothing queues the requests, nothing orders them; if nothing orders them, nobody reads the "this request matters less" hint. All ionice -c3 does is stick a label on the process; whether anyone reads the label depends on the disk's scheduler. On VPS3 nobody does. And I'm not the only one sticking labels on: on the same machine the man-db, e2scrub_reap and systemd-tmpfiles-clean units say IOSchedulingClass=idle, while logrotate says best-effort 7, and my own disk-cleanup and runner-image-refresh units say the same. Six units, from the distro packagers down to me, all wrote the same line with the same good intentions, and all six lines quietly go nowhere.

This post follows that label: who attaches it, which paths it travels, which of the four schedulers reads it, and how deep the queue has to be for the reader to read anything at all. The answers come from kernel source and from a lab built on scsi_debug. One lab result cost me real time: under mq-deadline, the low-priority job got more IOPS than the high-priority one. The cause turned out to be the narrowness of the queue.

Where the label comes from

In the kernel an I/O priority is a single integer: class in the upper bits, level in the lower ones. There are four classes — none (0), realtime (1), best-effort (2), idle (3) — and eight levels 0-7 for realtime and best-effort, 0 being highest. You hand a process this value one of three ways: the ionice command, IOSchedulingClass=/IOSchedulingPriority= in a systemd unit, or a direct ioprio_set(2) call. All three write to the same place.

The interesting part is what happens when you write nothing. If the class stays none, the kernel derives the value at I/O time from the CPU settings; two small functions in include/linux/ioprio.h do it. The level is (nice + 20) / 5: a process you started with nice -n 19 becomes best-effort 7 on the I/O side, nice 0 becomes 4. The class comes from the CPU scheduling policy: SCHED_IDLE processes fall into idle, real-time policies into realtime, everything else into best-effort. So a process you launched as chrt --idle 0 rsync ... counts as idle I/O without ever touching ionice. The Nice=19 line in the logrotate unit already produces best-effort 7 by itself; the IOSchedulingPriority=7 beneath it says the same thing twice.

ionice -p doesn't show this derivation, because it reads the raw value on the process. Three quick runs on vps5:

$ sudo systemd-run -p IOSchedulingClass=idle -p IOSchedulingPriority=7 sleep 30
$ ionice -p $(systemctl show -p MainPID --value run-u...)
idle
$ sudo systemd-run -p IOSchedulingClass=best-effort -p IOSchedulingPriority=7 -p Nice=10 sleep 30
best-effort: prio 7
$ sudo systemd-run -p Nice=10 sleep 30
none: prio 0
Enter fullscreen mode Exit fullscreen mode

The third says "none: prio 0" but will be treated as best-effort 6 the moment it issues I/O; (10 + 20) / 5. Don't read ionice -p output as "this process has no priority"; read it as "this process never asked for one explicitly".

Who reads the label

The value is copied from the process into the bio, from there into the request, and travels all the way to the block layer's scheduler. For requests submitted via libaio, fs/aio.c takes the submitting task's priority when the iocb doesn't carry an explicit one; the loop device copies the ki_ioprio field when forwarding a request to its backing file. It doesn't get lost along the way. The problem is at the end of the road: which of the four names selectable in /sys/block/*/queue/scheduler ever looks at the field.

Diagram

none. No scheduler; requests go down to the driver in arrival order. There's no code to look, so there's no field to look at.

kyber. A latency-targeted scheduler written for fast devices. The word ioprio does not appear in block/kyber-iosched.c; it separates reads from writes, not processes from processes.

mq-deadline. Didn't look for a long time. Class support arrived in 5.14 (Bart Van Assche's patch c807ab520fc3, June 2021): a separate FIFO and a separate sort tree per class, dispatch in RT > BE > IDLE order. The 5.14 version moved on to a lower class as soon as the higher class's queue was empty; the follow-up patch that tightened the order was reverted once and settled in 5.16 together with prio_aging_expire: while a higher class has uncompleted work, a lower class is never dispatched, but a request that has waited more than 10 seconds goes down without waiting for the upper class. The level is ignored entirely; dd_rq_ioclass reads only the class bits, and the ioprio_class_to_prio table maps the none class to BE. So for mq-deadline, best-effort 0 and best-effort 7 are the same thing. The IOSchedulingPriority=7 line is empty here as well.

One more detail, and it will surface in the lab: the question "does a higher class have work pending" is answered by dd_queued, which counts inserted - completed. Requests already handed to the device but not yet completed count as pending too. While even one BE request is inside the device, the idle class waits its turn; free slots in the device aren't enough.

BFQ. In the kernel since 4.12 and, being CFQ's heir, aware of priorities from birth. Its documentation is explicit: classes are served in strict priority order, a lower class is not served as long as higher-class queues exist; only the idle class gets a "very thin" slice so it doesn't starve. The level turns into a weight: weight = (IOPRIO_BE_NR - ioprio) * 10, i.e. 80 for best-effort 0, 40 for 4, 10 for 7. Queues in the same class share bandwidth by that weight. Of the four schedulers, it is the only one that reads the level.

And there's one that no longer exists. CFQ, the scheduler the priority classes were written for back in 2.6.13, was deleted in 5.0 along with the legacy block layer (f382fb0bcef4, "block: remove legacy IO schedulers"). The ioprio_set(2) man page still says today that "as at kernel 2.6.17 the only such scheduler is the Completely Fair Queuing (CFQ) I/O scheduler"; ionice(1) notes "since 2.6.13 with the CFQ I/O scheduler" too. Both are stale. The current truth: mq-deadline reads the class, BFQ reads class and level, the other two read neither.

The lab: a 1 ms virtual disk

I didn't need a real disk to verify this; quite the opposite, I wanted to escape a real disk's noise. The scsi_debug module creates a SCSI device in RAM and delays every command by as long as you ask. The modprobe line is the very one Bart Van Assche used to test his own patch (not the rest of his test: he measured 11,000 to 40 with sequential reads, a 100-second aging expiry and cgroup io.prio.class):

$ sudo modprobe scsi_debug ndelay=1000000 max_queue=16 dev_size_mb=256
Enter fullscreen mode Exit fullscreen mode

Every command takes 1 ms and the device accepts at most 16 commands at once; theoretical ceiling 16,000 IOPS. On top, two fio jobs, each libaio, direct I/O, 4 KiB random reads, queue depth 32, 15 seconds. A single job running alone gets 14,800-15,000 IOPS and 2.1 ms latency under all four schedulers; the device is saturated. Then I placed a second job next to it with a different priority and measured the split. vps5, Ubuntu 26.04, kernel 7.0.0-31.

Scheduler BE/4 – idle BE/0 – BE/7 RT/4 – BE/4
none 7,707 – 7,398 7,588 – 7,518 7,532 – 7,563
kyber 7,441 – 7,624 7,477 – 7,581 7,555 – 7,512
mq-deadline (nr_requests=34) 3,803 – 5,557 7,489 – 7,587 3,945 – 5,358
mq-deadline (nr_requests=256) 14,871 – 5 7,521 – 7,499 14,876 – 5
bfq 14,963 – 4 8,278 – 1,952 14,795 – 9

The none and kyber rows are what I expected: a fifty-fifty split in every column, no trace of the label. So is the BFQ row: the idle job drops to 4 IOPS with a mean latency of 5.7 seconds; the BE job facing RT gets 9 IOPS. The level difference shows too, BE/0 getting 4.2 times what BE/7 gets. The weight formula promised 8 times; I attribute the gap to the low_latency=1 default, whose documentation says it automatically raises the weight of queues it deems interactive and warns to "unset this tunable if you need/want to control weights". The 40-to-10 pair you'll see below matching the formula (4 expected, 4.4 measured) weakens that explanation, I know. I didn't switch it off and re-measure; that's not this post's subject.

The problem was the third row. On the first run, mq-deadline gave the idle job more IOPS than the BE job: 5,557 to 3,803. BE beat RT too. The source said RT > BE > IDLE and the measurement showed the exact opposite. It took me half an hour to blame my own setup rather than the scheduler.

A scheduler can only order what it holds

The answer was in the nr_requests file. A scheduler can only order the requests waiting in its own queue, and that queue has a bounded depth. The default lives in block/blk-mq.h: twice the hardware depth, capped at 256. My virtual device holds 17 commands; the scheduler queue holds 34. The two fio jobs want to have 64 requests outstanding. Once all 34 tags are taken, the 35th request waits at tag allocation without ever reaching the scheduler, and the tag wait queue has no notion of priority; whoever wakes first gets it. The scheduler orders the 34 it holds correctly, but has no say over which 34 it gets. Two cells in the table say it isn't a plain lottery either: the total fell from 15,000 to 9,360, and idle came out ahead both times. My reading: the tags taken by idle requests sit hostage in the scheduler, because dd_queued also counts BE's requests inside the device and refuses to dispatch idle; BE is squeezed into the two or three tags left, and half the device runs empty. The moment BE's in-device work hits zero, the waiting idle pile goes down in one salvo. Low priority, given a narrow queue, chokes high priority.

I confirmed it two ways. First I lowered the per-job depth to 8, a total of 16, below the queue: BE 7,342 IOPS, idle 74. Then I kept the depths at 32 and widened the queue:

$ echo 256 | sudo tee /sys/block/sdb/queue/nr_requests
Enter fullscreen mode Exit fullscreen mode

BE 14,871, idle 5; the idle job's mean latency 6.8 seconds, its 99th percentile 11.9. Those five IOPS come from prio_aging_expire: requests that hit 10 seconds go down without waiting for the upper class, and that's the leak. The RT-BE pair gave the same picture, while the BE/0-BE/7 pair stayed fifty-fifty; the level isn't read, and widening the queue doesn't make it read.

This setting has three more habits, and I saw all three in the lab. Under none, nr_requests equals the hardware depth (17 on my device) and you can't go above it; echo 256 comes back with "Invalid argument" — no one to sort, no extra slots. Switching schedulers resets the value to the default: I wrote 256, switched to kyber, got 34, switched back to mq-deadline, still 34. If you want it to stick, your udev rule has to write both the scheduler and the depth. One more distinction: the value you see under none is the controller's tag pool (16 commands plus 1 reserved on my device), while VPS3's queue_depth of 128 is the per-LUN depth; two different counters, so VPS3's nr_requests of 256 is no contradiction. And derived classes obey the same rule: a fio started under nice -n 19 got exactly the same 7,500 IOPS as the normal job on the wide queue under mq-deadline (level 7, class still BE), while one started under chrt --idle 0 dropped to 10 IOPS. On BFQ, nice -n 19 alone produced a 12,021 to 2,753 split; a 40-to-10 weight predicts 4 times, measured 4.4. You're writing a priority even without ionice, as long as someone reads it.

Why the default is none

The kernel picks the default in block/elevator.c: if the device has one hardware queue, mq-deadline; more than one, none; the one exception is controllers whose queues share a single tag pool (blk_mq_is_shared_tags, some RAID HBAs), which get mq-deadline even with multiple queues. The comment says no more than that; the "multi-queue devices are fast, don't put a single-lock scheduler in front of them" rationale is blk-mq's well-known design preference, not something written in the code. Nobody asks whether the disk spins. vps5's QEMU HARDDISK reports rotational=1, has six virtual queues and its scheduler is none; a device that presents itself as a spinning disk gets no ordering from the kernel at all. VPS3's virtio_scsi disk has 18 queues, a queue_depth of 128, nr_requests 256; also none. On both machines, the units that say IOSchedulingClass= have been talking to none since install day.

The menu is short too. On VPS3 the scheduler file lists only none and mq-deadline; no bfq, no kyber, because modinfo bfq says "Module bfq not found". Ubuntu 24.04 ships those two modules in the linux-modules-extra package, and VPS3 was installed with linux-image-virtual, which doesn't pull it in. I downloaded the package and looked with dpkg -c: kernel/block/bfq.ko.zst and kyber-iosched.ko.zst are in there. In Ubuntu 26.04 both moved into the main linux-modules package; on vps5, modprobe bfq loaded without a question. So "let me switch to BFQ" starts with an apt install.

What to do on VPS3

Six units and one knowledge-base recipe; none of them works, none of them does harm either. The nice -n 19 half still holds on the CPU side, and the real cost of scan jobs is usually CPU and page cache anyway. The real question is: do I have a background job that saturates the disk? If so, under none there are two ways to slow it down: slow the process itself (rsync --bwlimit, pv -L in the pipeline, or plain sleep between steps) or move it into a cgroup. On the cgroup side, the io.prio.class file can push all of a group's requests down to the idle class or cap them at BE with restrict-to-be; but it writes the same label, and with no scheduler reading it, it's just as empty. The brake that works independently of the scheduler is io.max; I covered it in the io.max, io.weight and io.latency post, and io.weight's own iocost-or-BFQ requirement is written up there too.

There's one more limit, and my lab doesn't see it: I measured direct reads. Writes that go through the page cache are sent to the disk by the kernel's flusher threads, not by the process, and wbc_init_bio attaches only the cgroup to the bio, not the process's ioprio. So ionice -c3 tar ... stays unlabelled on the write side even under BFQ; tar's reads carry the label, the archive it writes doesn't. Put the same job in a cgroup and write io.prio.class, and the label survives writeback, because what travels there is the cgroup. For the du recipe the difference is moot, du only reads; for backup scripts it matters.

My own decision: I'm seriously considering moving VPS3's sda to mq-deadline, because the real scheduling beneath the virtual disk is done by the hypervisor anyway, and the scheduler inside the guest is the only place where the guest's own priorities can be expressed at all. The price is 18 queues collapsing behind a single spinlock; I doubt that's measurable for VPS3's I/O load, so I'll measure. I didn't touch the live server for this post; the lab is done, the change is a separate day, in front of iostat. If I do it, the udev rule will look like this, with the nr_requests line included; on VPS3 the default would land on 256 anyway, so the line is harmless there, but having watched the depth reset on a scheduler switch once, I'd rather write the rule generically:

ACTION=="add|change", KERNEL=="sda", ATTR{queue/scheduler}="mq-deadline", ATTR{queue/nr_requests}="256"
Enter fullscreen mode Exit fullscreen mode

Checklist

  • cat /sys/block/<dev>/queue/scheduler: if the bracketed name is none or kyber, then ionice, IOSchedulingClass= and io.prio.class do nothing on that device.
  • If it's mq-deadline, only the class counts: idle and realtime are meaningful, best-effort 0-7 makes no difference. You needn't delete the IOSchedulingPriority= line, just know it's inert.
  • If you want levels, BFQ; on Ubuntu 24.04 ask for the linux-modules-extra package, linux-image-virtual installs don't bring it.
  • nr_requests must cover the total concurrent requests; the moment tags run out, priority becomes a lottery. When you persist the scheduler through udev, write the depth too.
  • If ionice -p says "none: prio 0", the process isn't priority-less; its priority is derived from nice and the scheduling policy: a SCHED_IDLE process is idle, a nice 19 process is BE 7.
  • Priority changes anything only when the device is saturated; on an idle disk the idle job runs at full speed too. If what you want to throttle is total I/O rather than contention, the tool is io.max.

I started with one line: half the recipe does nothing on this machine. What was wrong wasn't the recipe but my forgetting that the recipe has a precondition. ionice is a request; the scheduler that would honour it is chosen per disk, and on every multi-queue virtual disk the default answer is "nobody's honouring it". Sticking labels on is easy. Asking once who reads the label is cheaper than the silence of six units and one recipe.

Official Sources

Top comments (0)