Mustafa ERBAY

Posted on Sep 23 Originally published at mustafaerbay.com.tr

Why Is read_ahead_kb 8192? Linux 6.17 Believed the Disk

#linux #kernel #readahead #blocklayer

I was doing a routine sweep across the fleet: line up the block queue settings on every server and look for outliers. On the three machines I had upgraded to 26.04, /sys/block/sda/queue/read_ahead_kb said 8192. On the four still running 24.04 and 25.04, the same file said 128. Same provider, same virtual disk, same image family.

My first instinct was to blame myself: I must have run blockdev --setra somewhere and forgotten about it. But /etc/sysctl.d, /etc/udev/rules.d, tuned, the provisioning scripts — none of them mentioned read_ahead at all. Nobody had typed 8192. The kernel had written it, without asking me, and based on a lie the disk tells about itself.

This post is the record of chasing that number down. At the end of it sit three separate kernel patches, plus one old habit of virtualization. The amplification I measured in the lab was 970x. I wrote about a similar amplification on the anonymous-memory side in the THP and madvise piece; this time it is entirely the file side — the page cache and readahead.

Seven servers, two numbers

Here is the raw table I collected from the fleet on 22 September 2026:

Server	Ubuntu	Kernel	Device	`rotational`	`max_sectors_kb`	`read_ahead_kb`
vps1	25.04	6.14.0-37	sda	1	1280	128
vps2	25.04	6.14.0-37	sda	1	1280	128
vps3	24.04	6.8.0-139	sda	0	1280	128
vps4	24.04	6.8.0-139	vda	1	1280	128
vps5	26.04	7.0.0-31	sda	1	4096	8192
vps6	26.04	7.0.0-31	sda	1	4096	8192
vps7	26.04	7.0.0-31	sda	1	4096	8192

Two things change at once in that table: max_sectors_kb went from 1280 to 4096, and read_ahead_kb went from 128 to 8192. There is exactly a factor-of-two relationship between them (4096 × 2 = 8192), and that is not a coincidence. Note something else, too: vps1 and vps2 also believe they are spinning disks, but because they are on 6.14 they stayed at 128. So this is not only about what the disk claims — it is about when the kernel started taking that claim seriously.

There is also the classic unit trap. blockdev --getra /dev/sda returns 16384 here while sysfs says 8192. No contradiction: blockdev counts 512-byte sectors, sysfs counts kilobytes. Same number, two costumes. I have seen plenty of people mix these up and report that they "halved readahead".

The line that wrote 8192

The readahead size lives in sysfs, but sysfs does not produce it. The value is the ra_pages field of the block device's backing_dev_info structure, recalculated every time the device's queue limits are updated. The whole computation sits in one function in block/blk-settings.c. Here is how it looks in v6.17:

void blk_apply_bdi_limits(struct backing_dev_info *bdi,
        struct queue_limits *lim)
{
    u64 io_opt = lim->io_opt;

    if (!io_opt && (lim->features & BLK_FEAT_ROTATIONAL))
        io_opt = (u64)lim->max_sectors << SECTOR_SHIFT;

    bdi->ra_pages = max3(bdi->ra_pages,
                io_opt * 2 >> PAGE_SHIFT,
                VM_READAHEAD_PAGES);
    bdi->io_pages = lim->max_sectors >> PAGE_SECTORS_SHIFT;
}

The 128 KB everyone has memorized is right there at the bottom, as the floor: in include/linux/pagemap.h, VM_READAHEAD_PAGES is defined as exactly (SZ_128K / PAGE_SIZE). So 128 KB is not a default, it is a lower bound. The function picks the largest of three candidates: the device's current value, twice the optimal I/O size, and that 128 KB floor.

The interesting term is the middle one. My disk reports optimal_io_size as 0 — virtual disks usually do not report it, and neither do ATA disks. Normally that means the second candidate is zero and the 128 KB floor wins. But the if line intervenes: if the device is flagged as rotational and reports no optimal size, the kernel substitutes max_sectors instead. On my boxes the arithmetic is max_sectors_kb 4096, doubled to 8192. That is where the number comes from. Nobody typed it; the disk said two things about itself and the kernel multiplied the rest.

Looking at the 24.04 code makes the difference obvious. In v6.8 the function is a single statement:

disk->bdi->ra_pages =
    max(queue_io_opt(q) * 2 / PAGE_SIZE, VM_READAHEAD_PAGES);

No rotational exception, and no third term preserving the current value. If optimal_io_size is zero the answer is always 128 KB. That is the entire reason vps3 and vps4 stayed at 128.

Three patches, stacked

The gap is not caused by a single change; three patches landed in sequence, all pushing the same way.

The first arrived in v6.15: 7b720c720253, "block: never reduce ra_pages in blk_apply_bdi_limits". The max became a three-way one, and bdi->ra_pages — the device's current value — joined the list. The comment in the code states the reasoning plainly: there is no hardware limit on readahead size, the user may have raised it through sysfs, so never decrease it. That sounds considerate. In a moment we will see its other face.

The second landed in v6.17: Damien Le Moal's 9b8b84879d4a raised BLK_DEF_MAX_SECTORS_CAP from 2560 sectors (1280 KiB) to 8192 sectors (4 MiB). The justification was that the 1280 KiB limit set back in 2015 was an arbitrary number chosen to "accommodate a 10-data-disk stripe write", and that in the meantime software RAID drivers learned to advertise their stripe width properly. The 1280 → 4096 jump in my table is this patch.

The third is also v6.17 and also Le Moal: 459779d04ae8, "block: Improve read ahead size for rotational devices" — the patch that added the if line above. Its commit message is direct about the problem: ATA defines no concept of an optimal I/O size, so every ATA device is stuck with 128 KB readahead, and in multi-context buffered reads the seek overhead destroys throughput. The gains it measured are substantial: on XFS, with 8 readers doing 1 MB buffered reads from a SATA HDD, throughput rose from 122 MB/s to 167 MB/s (+36%); with 128 KB buffered reads, from 57 MB/s to 165 MB/s (+189%). For a SCSI disk it points at 2560 KB as the new default.

I have no quarrel with the patch itself; on a genuinely spinning disk that gain is real. The problem is that the BLK_FEAT_ROTATIONAL flag does not actually mean "this disk spins" — it means "this block device advertises itself as rotational". And in the virtualization layer those two drift apart constantly. My disks report the model QEMU HARDDISK with rotational set to 1; the provider does not say what actually spins underneath, and I do not know — though I have no reason to assume a cloud provider is genuinely turning platters in 2026. A fresh loop device leaves no room for doubt at all: in the lab below, the device I created with losetup came up rotational=1 with max_sectors_kb 1280, and read_ahead_kb read exactly 2560, just as the formula predicts. A loop device over a file has no seek penalty at all, but the kernel grants it the same discount.

That 1280 may have caught your eye: I just said the cap rose to 4 MiB, so why is loop still at 1280? Because drivers/block/loop.c pins the loop device's hardware limit at 2560 sectors; the comment in the code describes it as a random number picked from the historic block max_sectors cap. The new cap never reaches loop at all. That is why two block devices on the same kernel land on different numbers.

The first place I saw this cost real money was not my fleet. The RavenDB team published a dedicated advisory for 6.17: because Azure presents Standard SSD and Standard HDD disks to the VM as rotational, read_ahead_kb jumps from 128 to 4096, and a database doing random access over memory-mapped files fills its page cache with data it never asked for until I/O wait approaches 100%. On the same kernel version, AWS machines are fine — because there the disks do not advertise themselves as rotational. I find it striking that a single flag can sharpen the difference between two clouds this much.

I asked for 200 pages and got 758 MiB

Understanding the mechanism is one thing; seeing the bill is another. I built a controlled lab on vps5: a loop device over a 2 GB file, ext4 on top of it, and inside it a 1 GiB file filled from /dev/urandom. Then I used fio to do memory-mapped (mmap) random 4 KiB reads — the exact access pattern RavenDB was complaining about. I dropped the caches before each round, generated exactly 200 page faults, and read the actual number of sectors fetched from the device out of the loop device's /sys/block/loop0/stat counter.

`read_ahead_kb`	Requested	Read from device	Amplification
128	0.78 MiB	24.4 MiB	31x
2560	0.78 MiB	370.7 MiB	474x
8192	0.78 MiB	758.1 MiB	970x

I asked for 800 kilobytes and the disk delivered 758 megabytes. The reason the last row stops at 970x rather than 2000x is that the file is only 1 GiB; the readahead windows overlap each other and run into the end of the file. In other words, that number is the ceiling-clipped version.

This is so brutal because of the "mmap read-around" block in mm/filemap.c. For a mapping that has not been marked as either sequential or random, the code does this:

ra->start = max_t(long, 0, vmf->pgoff - ra->ra_pages / 2);
ra->size = ra->ra_pages;
ra->async_size = ra->ra_pages / 4;

It opens a window exactly ra_pages wide around the faulting page and centres it there. At a setting of 8192 KB, that means reading 8 MB to service a single 4 KiB page fault. On the read() path things are gentler, because there readahead ramps up gradually from the requested size; with mmap there is no ramp, the full window opens immediately.

It is not entirely without brakes. The same function defines MMAP_LOTSAMISS as 100: if a file misses far more than it hits, the kernel gives up on readahead because "it will only hurt". But the brake never engaged in my measurement, because the wide window turns some of the following faults into hits and keeps the counter below the threshold. The brake fires when readahead is useless — not when it is partly useful and wasting memory.

I should admit that I got this measurement wrong the first time, because the mistake turned out to be instructive. In the first round all three settings showed 1x amplification; the knob appeared to do nothing at all. The cause was that fio passes the MADV_RANDOM hint by default in random-read mode. In the source, that corresponds to this line:

if ((vm_flags & (VM_RAND_READ | VM_EXEC)) == VM_RAND_READ)
    return fpin;

The moment the application says "I will be accessing this randomly", readahead shuts off completely. Turning the hint off with fio --fadvise_hint=0 produced the table above; with the hint on, amplification is 1x even at 8192. So the last word on this knob belongs to the application, not the kernel. If your database passes MADV_RANDOM, fiddling with read_ahead_kb changes nothing — and if it does not, that is the place worth fixing.

What you raise sticks, what you lower does not

We have seen the considerate face of the "never decrease" rule inside max3. Its other face is this: the rule only protects increases. Nothing protects your decrease. I worked through it step by step on the loop device:

I pulled read_ahead_kb down to 128, then wrote the value already sitting in max_sectors_kb (1280) straight back, unchanged — purely to trigger a recalculation. Readahead jumped back to 2560. A single write that changed nothing and looks unrelated to readahead erased my setting.
I pulled it to 128 and set rotational to 0: it stayed at 128. As expected — the rotational exception no longer applies.
I set rotational back to 1: it leapt to 2560 immediately.
This time I raised the value to 16384, then set rotational to 0: it stayed at 16384. Setting it back to 1: still 16384. And while it sat there, I lowered max_sectors_kb to 512 — readahead still stayed at 16384. An increase is never undone.
I set the same experiment up the other way round: first I dropped read_ahead_kb to 128, then lowered max_sectors_kb to 256. This time readahead settled at 512, twice the new limit. Only one thing differs: while the value is small the formula can pull it up, while it is large the formula cannot touch it.

In short: the value you raise is permanent, the value you lower is provisional. Any event that touches the queue limits — a device rescan, a driver parameter change, a stacked device recomputing its limits — can silently undo your reduction. And when it does, nothing appears in the logs.

I discovered how stubborn that permanence is by accident. After the experiments I detached the loop device with losetup -d and reattached it to a completely different file; expecting a clean slate, I found read_ahead_kb still holding the raised value from the previous round. It makes sense: the backing_dev_info belongs to the device, not to the file behind it, and no path ever pulls that field down. What you take for a fresh device may be carrying your old setting.

That is why I do not trust the sentence "I ran echo 128 > ... once, it is handled." I also tested the udev rule usually recommended as the permanent fix:

ACTION=="add|change", KERNEL=="sd*", ATTR{queue/read_ahead_kb}="128"

The rule works: after udevadm trigger --action=change the value drops to 128. But when I then wrote to max_sectors_kb, it climbed back to 2560; triggering udev again brought it back down to 128. So the udev rule is not a lock, it is a corrector: it only applies when a uevent arrives. In practice little touches the queue limits after boot, so it is good enough on most systems — but it does not deserve to be described as a guarantee.

In LVM, the same formula, a very different answer

On stacked devices things get more interesting still. I built a volume group from two loop devices and created two logical volumes: one linear, one striped across two devices with a 64 KB chunk. The result:

Device	`read_ahead_kb`
Underlying loop devices	2560
Linear LV (`dm-0`)	2560
Striped LV (`dm-1`)	256

The striped volume sits at one tenth of the disks beneath it. lvs confirms the same thing: the policy is auto on both, but the kernel value is 2.50m on the linear volume and 256.00k on the striped one. And LVM's own documentation describes auto as "use default value chosen by kernel" — meaning LVM is not writing this number by hand.

The writer is the same function again. What differs is the input: for a striped target, drivers/md/dm-stripe.c reports a genuine optimal I/O size to the queue — io_min is the chunk size and io_opt is chunk size × number of stripes. In my case 64 KB × 2 = 128 KB, and twice that is 256 KB. The formula never changed; the device simply knew its own optimal size this time, so the "does it spin" guess never came up. The moment a device tells the truth, the guess drops out.

The practical consequence: if LVM is in the picture, tuning the underlying physical disk accomplishes nothing. I tested it — I raised the underlying loop device to 4096 and both LVs stayed where they were, at 2560 and 256. Every block device has its own backing_dev_info; readahead is read from whichever device your filesystem actually sits on. If you have ever watched someone tune /dev/sdb and wait for results on /dev/mapper/vg-data, this is why.

So what should you do

Let me be clear first: 8192 is not wrong everywhere. If you are reading large files sequentially off genuinely spinning disks, running backups, streaming video, scanning log archives, the gain Le Moal measured is your gain too. The point of this post is not that the number is large — it is that it changed without anyone deciding to change it.

The order I would work through for your own setup:

cat /sys/block/*/queue/read_ahead_kb — check it once after every kernel upgrade. Between pre-6.17 and post-6.17 you may see a 64-fold difference.
Look at the rotational field. If it says 0, none of this applies to you — relax. If it says 1, nothing guarantees that flag reflects the hardware underneath — yet the readahead decision rests entirely on it.
Ask what your workload's access pattern is. A randomly accessed, memory-mapped database (RavenDB, LMDB-based stores, some embedded engines) is the worst case. Anything reading large files sequentially is the best case.
Does the application pass MADV_RANDOM/POSIX_FADV_RANDOM? If so, do not bother with the block layer; the question is already settled.
If you are going to lower it, lower it with a udev rule and verify it survives a reboot; a one-off echo is not persistent.
If there is a stack (LVM, dm-crypt, md), look at the device your filesystem sits on directly; tuning the disk underneath will not change the outcome.
When cross-checking with blockdev --getra, remember to divide by two.

One note for anyone running containers: read_ahead_kb is the host's setting. Your database may live inside a container, but this number sits under /sys/block, belongs to the host, and cannot be changed from inside. The I/O jitter you are seeing in the container may come from a file you have no permission to touch — the fix belongs on the host side.

If you are wondering whether this creates memory pressure, how the page cache gets reclaimed is in the min_free_kbytes post; who divides up I/O at the cgroup level is in the io.max and io.weight post.

What stays with me from this is not a number but a direction. For years the block layer has been moving away from fixed defaults and towards deriving settings from what the hardware says about itself: optimal I/O size, stripe width, rotational or not. On a physical machine that makes perfect sense. In a virtualized fleet, the hardware is now an imitation produced by software, and that imitation is not always honest. A virtual disk reporting rotational=1 is not trying to deceive anyone; it is just that nobody filling in that field expected someone to read it and multiply readahead by 64. We need to accept that "default" is no longer a fixed number but the result of a calculation — and that the inputs to that calculation change in places the release notes never mention.

DEV Community