Linux has two settings separated by a single letter: irq_affinity and rq_affinity. I wrote about the first one back in April, in a softirq saturation and IRQ affinity runbook — that piece was about network cards and about which core an interrupt lands on. The second one lives a layer down: it decides, after the interrupt has landed, whose ledger the block layer writes the request into. I had never measured that one.
My assumption was this: rq_affinity=1 — which is what devices ship with — moves completion work to the CPU that submitted the request. The documentation says roughly that too. When I ran it, the default moved nothing at all. The only value that moved anything was 2, and between 0 and 1 I could not find a single difference.
The lab is the usual place: Docker Desktop's linuxkit virtual machine, kernel 6.10.14, aarch64, 14 vCPUs. A privileged container can write to /sys/block/*/queue. A warning up front — when you write there you affect the whole VM, not the container. Don't try it on a live server. That is exactly why I picked a disposable VM.
Three levels, and one sentence left over from 2008
rq_affinity takes three values — but its documentation acknowledges only two. The text describing the levels sits in Documentation/ABI/stable/sysfs-block, dated September 2008, and it covers 1 and 2; 0 never appears. Zero exists only as a branch in blk-sysfs.c. The entry is — five years older than the multi-queue block layer (blk-mq, 2013) it now describes.
In short, it says this: if the value is 1, the block layer migrates completions to the "group" of the CPU that originally submitted the request; if you need to maximise distribution, 2 bypasses the group aggregation logic and runs the completion on the requesting CPU itself.
The load-bearing word here is "group", and the quotation marks are the documentation's own — whoever wrote it knew in 2008 that the word was fuzzy. Without measuring how wide that group is today, you cannot know what the setting will do. Worth noting where the file lives, too: Documentation/block/queue-sysfs.rst, which hosted these knobs for years, is gone. It is still there at v5.16 and gone at v5.17. The queue sysfs documentation moved into the ABI file.
My first measurement read zero bytes
When I first set the experiment up, I wrote this line: taskset -c 7 dd if=/dev/vda of=/dev/null bs=4k count=20000 iflag=direct 2>/dev/null. Results came back: a BLOCK softirq delta of between 1 and 7. Seven completions for twenty thousand requests. I had started building a hypothesis — "so completions never reach softirq at all" — when I stopped and added a positive control: did the device actually get read?
The first field of /sys/block/vda/stat (completed reads) had not budged. Delta zero. Then I unmuted stderr:
sh: taskset: not found
exit=127
Alpine's busybox has no taskset. dd had never run, and I had spent four minutes measuring background noise and theorising about a mechanism. That innocent-looking 2>/dev/null was what hid the error message from me.
That is why my measurement scripts now always carry a proof counter: one line that confirms what the script thinks it saw against an independent counter. Here that line was the read counter in /sys/block/vda/stat, and it killed a wrong hypothesis before it was born. After apk add util-linux, the counter showed 20,000.
I verified the pinning worked, too: taskset -c 7 awk '{print $39}' /proc/self/stat → 7. Field 39 of /proc/self/stat is the CPU the process last ran on.
The shape of the device: one hardware queue, one interrupt core
Before measuring I mapped the device topology, because the answer for this setting depends entirely on topology.
vda is a virtio-blk disk with a single hardware queue: under /sys/block/vda/mq there is only a 0 directory. And that single queue's cpu_list is the whole machine: 0, 1, 2, ... 13. Fourteen software contexts are mapped onto one hardware queue.
The interrupts land in one place as well. In /proc/interrupts, every column except CPU0 is zero on all the virtio rows; the busiest of them, virtio2, shows 7.7 million interrupts, and those are all on CPU0 too. I did not separately map which virtio device corresponds to which row — what matters for this experiment is that every one of them lands on a single core. The BLOCK softirq counter accumulated since VM boot told the same story: 7,626,231 on CPU0 and exactly zero on the other thirteen cores.
The cache topology is the crux — not the disk's, the CPUs':
index0: L1 Data shared_cpu_list=0-13
index1: L1 Instruction shared_cpu_list=0-13
index2: L2 Unified shared_cpu_list=0-13
The virtual machine reports that all fourteen vCPUs share even the L1 data cache. No real hardware works like that; this is a declaration that falls out of the hypervisor flattening the topology. cpu_capacity is 1024 on all three I checked (0, 7, 13) — symmetric.
One more note: in my first probe I had written 2 into vda's rq_affinity, so the value I read back from that device was not the default, it was mine. I looked at devices I had not touched — vdb, loop0, nbd0 — and all three read 1. The kernel source confirms it: QUEUE_FLAG_MQ_DEFAULT contains QUEUE_FLAG_SAME_COMP and not SAME_FORCE. The default is 1.
The measurement
For each level, 20,000 4 KiB O_DIRECT reads pinned to CPU7, straight off raw /dev/vda. Because O_DIRECT disables the page cache and readahead, each read corresponds to its own request. I deliberately ordered the runs 1 → 0 → 2 → 1, with 1 repeated at the end, because I needed to know whether the first run was a warm-up.
| rq_affinity | BLOCK softirq → CPU0 | BLOCK softirq → CPU7 | proof (stat read delta) | duration |
|---|---|---|---|---|
| 1 (first run, cold) | 24,393 | 0 | 20,671 | 4.66 s |
| 1 (last run, warm) | 20,003 | 0 | 20,007 | 0.97 s |
| 0 | 20,063 | 0 | 20,260 | 0.95 s |
| 2 | 2 | 20,000 | 20,164 | 0.96 s |
Before reading the table, let me be precise about what the counter counts: the BLOCK row counts softirq raise events, not completions. Both blk_mq_raise_softirq() and blk_mq_complete_send_ipi() push the request onto a list with llist_add() and only act if the return value is true — that is, if the list was empty — which llist_add's own documentation states plainly. Consecutive completions can therefore coalesce into a single raise. The only reason the count lines up with the request count here is that dd is strictly serial: with a queue depth of one, each completion drains before the next submission. Under a deep-queue workload that one-to-one correspondence breaks.
The table says two things. First, the opening run is an outlier: 4.66 seconds and 24,393 softirqs, while the following three runs all land around 0.96 seconds and ~20,000 softirqs. Without that repeated 1 at the end I might have mistaken the gap for an effect of the setting. It was warm-up.
Second, and this is the real point: 0 and 1 are indistinguishable. Under both, every completion lands on CPU0 and the submitting CPU7 gets exactly zero. When I wrote 2, 20,000 completions moved to CPU7 and two were left on CPU0. In other words, on this machine there is no difference whatsoever between turning the setting off and leaving it at its default.
The code's two gates
Why it behaves this way is written in two functions in block/blk-mq.c. First the call path:
The critical gate is box E, the condition inside blk_mq_complete_need_ipi(): if the submitting CPU is the current CPU, or if SAME_FORCE is not set and the submitter and the current CPU share a cache domain and have equal capacity, the function says "no IPI needed".
Look at the two helpers. cpus_share_cache() compares the last-level cache domain id (sd_llc_id) of two CPUs. That id comes from the scheduler's own domain tree and is never exposed under /sys; the shared_cpu_list output above is a proxy you can consult in its place, not its source. I inferred that all fourteen vCPUs sit in one domain from that proxy and from the fact that at level 1 no completion ever left CPU0 — I could not read it directly. cpus_equal_capacity() is even more of a shortcut: if asymmetric capacity is not active it returns true without comparing anything at all, so on a symmetric machine that gate is already open.
The upshot: with rq_affinity=1, all three conditions hold, so need_ipi returns false and the completion is not redirected. Then, because nr_hw_queues == 1, a local softirq is raised — on CPU0, where the interrupt landed. With rq_affinity=0, SAME_COMP is never set, so the function returns false on its very first line and the path ends up at that same local softirq. Two different reasons, one destination; that is why the measurement cannot tell them apart.
When I write 2, SAME_FORCE gets set and the whole parenthesis drops out. All that remains is "is the submitting CPU the current CPU?" — and since I submit from CPU7 and complete on CPU0, the answer is no, so smp_call_function_single_async() pushes the completion onto CPU7's queue.
There is a small oddity, too. In block/blk-sysfs.c, the line that shows the value is queue_var_show(set << force, page). The number you read is the two flags combined by a shift — with set=1, force=1 you get 1 << 1 = 2. On the write side there are branches only for 0, 1 and 2, and since queue_var_store() returns the byte count for any valid number, echo 3 > rq_affinity returns success and changes nothing. I did not measure that; I derived it from the code — my lab died in the meantime, and I explain why below.
So where is 1 useful?
Saying the setting is useless would be wrong; it is useless on my machine. The code's two gates close in exactly these cases, and then 1 really does move the completion:
-
When cache domains are separate. On multi-socket servers, different sockets have different
sd_llc_ids. The same is true on AMD's multi-CCX chips, where each CCX has its own last-level cache; the domain changes even within one package. If the interrupt lands on a core in one domain and the request was submitted by a core in another,1performs the redirect — this is where the documentation's "group" really is smaller than the machine. -
When asymmetric capacity is active. On big.LITTLE-style ARM systems, cores can share a cache and still differ in capacity;
cpus_equal_capacity()then actually compares, and can return false.
Conversely, two situations switch 2 off as well. If force_irqthreads() is true — that is, if the kernel was booted with threadirqs — the function returns false without ever looking at cache or capacity, because raising a softirq from an SMP call means waking ksoftirqd, which can cost more than completing in a different cache domain. My /proc/cmdline has no threadirqs; I checked. Second, if the submitting CPU is the same as the interrupt CPU, the first condition fires and even 2 performs no redirect. I did not measure that case; I derived it from the code.
So what about multi-queue NVMe? This needs care, because NVMe goes through the very same gate: drivers/nvme/host/nvme.h hands the completion to blk_mq_complete_request_remote(). When there really is a hardware queue per CPU (nr_ctx == 1), the first gate picks local completion and the setting goes idle. But when there are fewer queues than CPUs — hardly exotic, since the queue count is capped by the MSI-X vectors available, and eight queues on a sixty-four-core box is ordinary — several CPUs map onto one hardware queue, nr_ctx exceeds one, and that first gate closes. At that point 0 and 1 are still indistinguishable, but 2 still issues its IPI. So a single queue kills only the nr_hw_queues == 1 softirq branch, not the setting. The device under test here is a single-queue virtio disk — that is, a typical virtual machine guest.
What I did not measure
I am giving this its own heading, because the limits of a measurement matter as much as its result.
The IPI counter did not separate the levels. I watched the IPI1 (function call) row in /proc/interrupts across all three levels, and CPU7 received ~20,000 in each: 20,000 at 0, 20,005 at 1, 20,022 at 2. The totals were similarly close, in the 22–23 thousand band. I expected a clear rise at 2; I did not see one. I later found the reason in the scheduler's code: ttwu_queue_cond() in kernel/sched/core.c hands a task to the target CPU's wakelist when that CPU is idle, and announces it with a function-call IPI. Since dd sits blocked on CPU7, CPU7's runqueue is empty, so every completion — at any level — sends CPU7 a wakeup IPI. That counter lumps completion IPIs together with task wakeups in one row, and at queue depth one the latter completely masks the former. I derived this from the code rather than measuring it separately; but one practical conclusion is firm: don't use IPI1 to measure the completion IPI.
I am claiming no throughput difference. The three warm runs came in at 80.4 / 82.2 / 81.7 MB/s. That spread is about two per cent and I have exactly one warm sample per level. Calling one level faster on that data would be presenting noise as effect. The effect visible in these numbers is where the work lands, not how fast it finishes.
One machine, one device type, one architecture. A single virtual machine on aarch64, one single-queue virtio-blk disk. I did not measure 1 actually migrating on multi-socket or asymmetric-capacity hardware; I derived those two bullet points by reading the code.
Kernel version. My lab runs 6.10.14, while the stable release kernel.org lists as I write this is 7.2.10. I downloaded both functions (blk_mq_complete_need_ipi and blk_mq_complete_request_remote) from the v6.10 and v7.2 tags separately and compared them: byte for byte identical. So the behaviour in the table above is not a quirk of an old kernel. The contents of QUEUE_FLAG_MQ_DEFAULT did change — three flags in v6.10, only SAME_COMP in v7.2 — but the part that matters here, the default being 1, is the same in both.
One honest admission to close: I could not run the two extra experiments I had planned (pinning the submitter to CPU0 to show 2 becoming ineffective, and writing echo 3 to confirm it is silently swallowed). The Mac's data volume hit one hundred per cent full, containerd could no longer write, and every image started returning input/output error. Rather than trying to rescue a lab with three gigabytes of headroom left on disk, I drew a line between the measured and the derived.
Ask your own machine, in three commands
Is your rq_affinity=1 doing anything? The answer is in your CPU topology, not your device:
# 1) How many cores share the LAST-LEVEL cache?
# CAREFUL: unioning every index misleads you — on real hardware L1/L2 are
# per-core, so you get dozens of lines even when there is a single LLC.
# Look only at each CPU's TOP index:
for c in /sys/devices/system/cpu/cpu[0-9]*; do
last=$(ls -d "$c"/cache/index* 2>/dev/null | tail -1)
[ -n "$last" ] && cat "$last/shared_cpu_list"
done | sort -u
cat /sys/devices/system/cpu/online # same as above => one domain
ls -d /sys/devices/system/node/node* | wc -l # NUMA splits the domain too
# 2) Are capacities symmetric? This file exists ONLY on arm64/riscv.
# On x86 it prints nothing — and empty output does NOT mean "symmetric".
cat /sys/devices/system/cpu/cpu*/cpu_capacity 2>/dev/null | sort -u
# 3) How many hardware queues does the device have?
ls /sys/block/<device>/mq | wc -l
If the first command's last-level list matches the online list and you have a single NUMA node, your machine is one cache domain. On the capacity side, architecture decides: on arm64/riscv, a single value from the second command means the capacity gate is open too; on x86 that file does not exist at all, and the question to ask there is whether the CPU is a hybrid (P/E-core) part. If both hold, rq_affinity=1 is inert for you too, and dropping it to 0 is the same as leaving it at 1.
If the third command says 1, every completion is piling onto whichever core takes the interrupt. The BLOCK row of /proc/softirqs tells you which core that is — but not how many completions there were; because of the coalescing described above, under high IOPS the number falls well short of the request count. Use that row to locate the work, not to count it. And if the interrupt core has become your bottleneck, the value that distributes the work is 2, not 1.
Two more caveats. First, this setting only means anything for drivers that report completion through blk_mq_complete_request(); for drivers that call blk_mq_end_request() in their own context, the value you read is decoration. loop0, which I consulted above to confirm the default, is one such case — drivers/block/loop.c finishes the request synchronously on one path. Second, a value you write to sysfs is lost on reboot; if you want it to stick, attach it to a udev rule or a unit that runs at boot.
I am not saying "everyone should write 2". 2 incurs an IPI debt on every completion, and my measurement could not produce the invoice for that debt. What I am saying is more modest: this setting ships in a position where, on a great many machines, it does nothing at all — and there is no way to know that without measuring.
The general lesson I took away
The story of this setting made me think something about the older knobs under /sys and /proc: a setting's documentation can be older than the subsystem beneath it. When someone said "CPU group" in 2008, the thing they meant really was smaller than the machine on that era's hardware. In the eighteen years since, cache topologies changed and hypervisors started flattening topology — on many guests today, "group" and "whole machine" are the same thing. The documentation did not change, because the documentation is not wrong; it simply describes an empty set now.
Black boxes make their quietest mistakes while their sentences remain true. The description of rq_affinity still holds: 1 migrates the completion to the group of the submitting CPU. The sentence saying there is no distance left to migrate when the group is the entire machine is written nowhere. You only learn that by looking at /proc/softirqs.
Official Sources
- Documentation/ABI/stable/sysfs-block — the rq_affinity entry (September 2008) and the queue sysfs reference
- block/blk-mq.c — blk_mq_complete_need_ipi, blk_mq_complete_request_remote, blk_mq_complete_send_ipi, blk_mq_raise_softirq
- block/blk-sysfs.c — the shifted read in queue_rq_affinity_show and the store's 0/1/2 branches
- include/linux/blkdev.h — QUEUE_FLAG_SAME_COMP, QUEUE_FLAG_SAME_FORCE and QUEUE_FLAG_MQ_DEFAULT
- drivers/nvme/host/nvme.h — NVMe hands the completion to blk_mq_complete_request_remote
- include/linux/llist.h — llist_add returns true only when the list was empty (why softirq raises coalesce)
- drivers/block/loop.c — the path that finishes the request with blk_mq_end_request in its own context
- kernel/sched/core.c — the cpus_share_cache and cpus_equal_capacity implementations
- Multi-Queue Block IO Queueing Mechanism (blk-mq) — software/hardware queue mapping
- SMP IRQ affinity — distributing interrupts across cores
Top comments (0)