Mustafa ERBAY

Posted on Oct 21 Originally published at mustafaerbay.com.tr

statx Said 512; the Kernel Only Went Direct at 4096

#linux #kernel #storage #filesystem

If you have ever written code that uses O_DIRECT, you know the folklore: align your buffer to 512 bytes, make the offset a multiple of 512, and the length too, or you get EINVAL. The flag landed in the kernel in 2.4.10, back in 2001; the rule of thumb has travelled with it ever since, and it has been incomplete the whole time, because "512" is not a property of a filesystem — it is the memorised logical sector size of the block device underneath. Linux 6.1 offered an alternative to the memorising: you can now ask the file itself. Pass the STATX_DIOALIGN mask to statx() and the kernel hands back two numbers, the alignment for your memory buffer and the alignment for the file offset.

The first time I made the call on my own machine I got exactly the numbers I expected. Then I started wondering: if I obey that number, am I really writing directly? And if I disobey it, do I really get an error? I put both questions to the lab. Both answers came back as "it depends", and there are more such cases than I thought.

The three shapes of the answer

The subtle part of STATX_DIOALIGN is that the question does not have a single answer. The kernel can say three different things, and two of them are easy to miss:

It does not return the mask bit at all. If the field you asked for is absent from stx_mask, the kernel has said "I am not giving you this information for this file". You learned nothing.
It returns the bit with zero values. This means "there is no direct I/O on this file". The man page is explicit: the field is 0 if direct I/O is not supported on this file.
It returns the bit with non-zero values. This is the answer you were hoping for: align the buffer to this, the offset to that.

Code that does not distinguish the first from the second — code that writes if (stx.stx_dio_mem_align == 0) and never checks the mask bit — puts "I did not answer" and "there is no direct I/O here" in the same box. The right response to each is the exact opposite of the other: in the first case you have to be conservative and measure for yourself, in the second you have to give up on O_DIRECT entirely.

The coverage is limited too, and it matters to read that from the documentation rather than from memory: the man page says STATX_DIOALIGN works on block devices since 6.1, and on regular files it is supported by ext4, f2fs and xfs since 6.1. It is the kind of list where you can look and say "my filesystem isn't on it". There is also a layer that is not on the list yet answers: the overlayfs row in the table reports 512/512, because ovl_getattr() hands the question down to the real inode underneath — the answer actually comes from the ext4 below it. The list counts the filesystems that implement it; the layers in between carry up the answer of whatever is below. Four years later, in today's mainline (7.3), btrfs has not been added; btrfs_getattr() fills in the birth time, the subvolume number and the compression flag, but not the alignment.

There is one more field: stx_dio_read_offset_align. It was added in 6.14 because the alignment for writes can differ from the alignment for reads, and according to the man page only xfs fills it in. Why it is needed is coming up shortly; the answer landed in my lap in the lab.

The probe: what statx says versus what the kernel does

To measure this I wrote a 139-line C program. It does three things: it fetches the alignment report with statx(), it tries nine different O_DIRECT requests (aligned reads and writes, a buffer shifted by one byte, a 511-byte length, offset 511), and then it measures the interesting part — whether the write was actually direct.

I do that with mincore(). The logic: a successful O_DIRECT write bypasses the page cache, so the page you wrote should not appear in the cache. The probe first drops the file's cache with POSIX_FADV_DONTNEED, confirms "0 pages before" with mincore(), then writes with O_DIRECT and looks again. Still 0 afterwards means the write went direct. 1 afterwards means the write travelled through the page cache — the call returned successfully, but the O_DIRECT promise was not kept. The same probe showed up while hunting for the counter MADV_FREE moves: mincore() answers "is this page in memory right now", not "who put it there", and here that is exactly the question.

The lab is the Docker Desktop linuxkit virtual machine on my own Mac, kernel 6.10.14. Inside a privileged container I set up ext4, xfs and btrfs on loop devices; tmpfs and the container's own overlayfs are on the list too. Each filesystem lives on a 512 MiB image file, and every loop device underneath reports a logical sector size of 512 and a dma_alignment of 511.

On the first round I had not noticed that I was running the probe against a mounted device, and the program wrote 4 KiB of 0x5a over the place where the ext4 superblock lives. The next attempt to create a file after mount said "Structure needs cleaning". Destroying your own lab with your own probe is an educational experience; that accident produced the most surprising finding in this article, which we will get to.

Twelve cases, one table

Here is the summary. The third and fourth rows are two regions of the same device; why they diverge comes at the end of the article. The "direct" and "buffered" columns are the result of the mincore() probe; the "misaligned request" column is the return value of a 4 KiB request whose buffer was shifted by one byte.

Case	statx DIOALIGN	512B write	4096B write	misaligned request
overlayfs file (container root)	512 / 512	direct	direct	`EINVAL`
raw loop device (not mounted)	512 / 512	direct	direct	`EINVAL`
mounted device, offset 0	512 / 512	—	buffered	write OK, read `EINVAL`
mounted device, distant offset	512 / 512	—	direct	`EINVAL`
ext4 plain file (4 KiB block)	512 / 512	direct	direct	`EINVAL`
ext4, `inline_data` small file	0 / 0	—	—	all OK
ext4, `-o data=journal`	0 / 0	buffered	buffered	all OK
ext4, `-o nodelalloc` + `chattr +j`	0 / 0	buffered	buffered	all OK
xfs, unshared file	512 / 512	direct	direct	`EINVAL`
xfs, reflinked file	512 / 512	buffered	direct	`EINVAL`
btrfs file	no bit	buffered	direct	all OK
tmpfs file	no bit	everything is cached	everything is cached	all OK

The "—" cells in the table are not missing measurements but invalid ones: on the inline_data file the probe's own 4 KiB write pulls the file out of inline storage (the next statx says 512/512), and on the mounted device the "zero pages cached beforehand" precondition cannot be met. In both, the cache probe has nothing to say.

The first row hides a small but pleasing detail: even on an ext4 formatted with a 4 KiB block size, the offset alignment is 512, not 4096. The iomap defaults use the device's logical sector size rather than the filesystem's block size; inside ext4_getattr() this is literally two lines — dio_mem_align is the device's DMA alignment plus one, dio_offset_align is the device's logical block size. Alongside them, stx_blksize reports 4096: "efficient I/O size" and "mandatory alignment" are two separate fields of the same call, and they are very easy to conflate.

Zero is not an error code, it is an answer

The function that answers the alignment question in ext4 is a dozen lines long: ext4_dio_alignment(). It returns 0 if fsverity is active, 0 if the inode uses journalled data, and 0 if the data is stored inside the inode (inline_data); for an encrypted inode it returns the filesystem's block size (on my kernel, only if fscrypt supports direct I/O for it), and in every other case it returns 1, meaning "use the iomap defaults".

The practical consequence: two files sitting side by side under the same mount can have different alignment requirements. On an ext4 mounted with -o nodelalloc I set chattr +j on one file and left its neighbour alone. The neighbour reported 512/512; the +j file reported 0/0. Same filesystem, same block device, same second.

More interesting still, opening the file that reports 0 with O_DIRECT does not fail. open() succeeds and even a 511-byte write returns OK; on the data=journal and +j files the mincore() probe shows the page ending up in the cache. So after answering "there is no direct I/O on this file", the kernel meets the application's insistence not with an error but with silence: it routes the request down the buffered path. The kernel says as much in its own log, too; when I mounted with data=journal, dmesg printed this:

EXT4-fs: Warning: mounting with data=journal disables delayed allocation,
dioread_nolock, O_DIRECT and fast_commit support!

If the word fast_commit in that warning rings a bell, the article where I enabled fast_commit and it never ran across 4,000 fsyncs was another member of the same family: the distance in ext4 between a feature being "on" and the feature running.

That produced one more trap. On the first round I had mounted ext4 with default options and set chattr +j on the file, and statx returned 512 rather than 0. lsattr shows the flag; the mode does not engage. The reason sits in ext4_inode_journal_mode() along with a comment in the source: per-inode data journalling only kicks in when delayed allocation is off, and ext4 mounts with delalloc by default. In other words, the j in your lsattr output means nothing without looking at the mount options. statx was telling the truth; the flag I was reading was lying.

The two that did not answer

btrfs and tmpfs never returned the mask bit — but not for the same reason.

btrfs does support O_DIRECT; the probe's 4 KiB write bypassed the page cache. Yet a 512-byte write, a 511-byte write, a buffer shifted by one byte, a read from offset 511 — all of them returned success instead of EINVAL, and left a page in the cache. So btrfs's real alignment requirement is 4096, and the penalty for violating it is not an error but buffered I/O. The requirement has a name and a place: check_direct_IO() checks both the offset and the buffer against sectorsize, and btrfs_direct_write() does goto buffered when that check returns -EINVAL. So even without a STATX_DIOALIGN field you can read the requirement — statfs()'s f_bsize reports the sector size, and so does the blksize=4096 in my own output. What you cannot read is whether you complied: no error code says so, and you need a probe like mincore() or /proc/diskstats.

tmpfs is odd from the start. On a filesystem whose files live entirely in RAM, "bypass the page cache" is a meaningless request; even so, open(O_DIRECT) does not fail. The reason is a single line in mm/shmem.c: shmem_file_open() sets FMODE_CAN_ODIRECT on the file, so it passes the VFS check for "this file does not accept O_DIRECT". The probe confirms everything is cached — mincore() reports one page even before the write. The honest party here is statx: it does not answer the alignment question, because there is no answer to give.

The right number, an insufficient answer

My favourite measurement is on xfs. On a reflink-enabled xfs I created a 4 MiB file, A.bin. The probe reported 512/512, and a 512-byte O_DIRECT write went direct; no page in the cache. Then I ran cp --reflink=always A.bin B.bin — without touching a single byte of A.bin — and ran the same probe against the same file again:

=== 1) A.bin fresh (unshared)
    extent flags: found last,eof
      cache probe   512B write: before=0 after=0
      cache probe  4096B write: before=0 after=0
=== 2) A.bin, sharing extents with B (A WAS NOT TOUCHED)
    extent flags: found last,shared,eof
      cache probe   512B write: before=0 after=1
      cache probe  4096B write: before=0 after=0

statx still says 512. The 512-byte write still returns successfully. But it is no longer direct: the page is in the cache. The 4096-byte write stays direct. So this file's real write-side alignment is not the 512 statx reported but 4096, the filesystem's allocation unit. I repeated the shared side over three rounds. The unshared control came from a separate run with cp --reflink=never, because in my first three-round script the control file was the source of the reflink: since the flag lands on both inodes, the file I was measuring as "unshared" was shared too, and all six probes were measuring the same state. My own thesis had broken my own control group.

This crack does not open on every device: it needs the device's logical sector to be smaller than the filesystem's allocation unit. That holds on disks reporting a 512-byte logical sector — most SSDs on the market — while on a 4Kn device statx already says 4096 and the trap closes.

The reason is written in the xfs source, mechanism and all. On my kernel the write path runs into this gate inside xfs_file_dio_write(): right under the comment "We can't properly handle unaligned direct I/O to reflink files yet, as we can't unshare a partial block", if xfs_is_cow_inode(ip) holds, the kernel leaves a trace_xfs_reflink_bounce_dio_write trace and returns -ENOTBLK; xfs_file_write_iter() then falls back to the buffered write on that error code alone (if (ret != -ENOTBLK)). The reason behind the bounce is the read-modify-write cycle: allocating a new location, reading the old block, writing over it. The mincore() probe sees the consequence of that escape; on xfs you can also watch the escape itself by enabling that tracepoint, no probe needed. The comment on xfs_report_dioalign(), added in 6.14, explains this first-hand and states the fix: for COW inodes, dio_offset_align is reported as the allocation unit while dio_read_offset_align is reported as the device's logical block size. My kernel is 6.10, so what I captured here is the pre-fix behaviour; the fix itself comes from reading the source, not from a measurement. On 6.14 and later, statx should report 4096 on the write side and 512 on the read side for the same file.

I probed the read half of that asymmetry too, because measuring half a claim and writing all of it is not a thing:

A.bin unshared          read   512B: before=0 after=0 -> DIRECT
A.bin after reflink     read   512B: before=0 after=0 -> DIRECT
A.bin after reflink     write  512B: before=0 after=1 -> BUFFERED

On the shared file a 512-byte read leaves no page in the cache, so it stays direct; a write of the same size on the same file is buffered. One inode, two directions, two different alignment requirements: that is precisely why STATX_DIO_READ_ALIGN exists, and it matches what is measurable on 6.10 exactly. The probe asks for the STATX_DIO_READ_ALIGN bit on every file and 6.10 never returned it — the cheapest possible evidence that the field arrived in 6.14.

The sticky part is more annoying. I deleted B.bin and did a umount/mount cycle; filefrag no longer shows the shared flag, so the sharing is over. A.bin's 512-byte write is still buffered. Because at the moment of sharing, xfs sets a flag on the inode itself (XFS_DIFLAG2_REFLINK) and sets it on both the source and the destination. Two things clear it, and neither happens by itself during ordinary use: an explicit unshare (xfs_reflink_unshare() → xfs_reflink_try_clear_inode_flag(), which per its own comment only fires if the sharing is gone and the size has not changed), or truncating the file to zero (xfs_itruncate_clear_reflink_flags(), once the data and COW forks are both empty). An rm plus a umount/mount is neither. Translated into practice: if you took a snapshot of a database file with cp --reflink, that file's small O_DIRECT writes keep going down the buffered path even after you delete the copy. A fresh file on the same filesystem (step 5 of the probe) writes direct without trouble — so the condition belongs to that inode, not to the filesystem.

That also gives you the remedy. You can see the sharing with filefrag -v (the shared flag); to drop the inode flag, run xfs_io -c "unshare 0 <len>" file (it maps to FALLOC_FL_UNSHARE_RANGE in the kernel), or move the file to a new inode with cp --reflink=never. Whether the write path actually recovered is something you confirm with the mincore() probe, or with xfs's own tracepoint.

Same number, two outcomes: the block device

Now back to the accident that trashed my superblock on the first round. statx reports an alignment for a block device as well (512/512) and a misaligned request is rejected with EINVAL — but not always. I ran the probe against four states separately, and statx answers identically in all of them:

E1) clean device, cache empty                   -> write 4096B buffer+1: Invalid argument
E2) same device, cache full but pages clean     -> write 4096B buffer+1: Invalid argument
F ) device with ext4 MOUNTED on it              -> write 4096B buffer+1: OK (4096 bytes)
F2) same device, after unmounting               -> write 4096B buffer+1: Invalid argument

The trigger is not a populated page cache — the second line rules that out; I warmed the device with a buffered read and still got EINVAL. The trigger is the pages being uninvalidatable. The path can be traced step by step in block/fops.c: blkdev_direct_write() tries to invalidate the pages in the range before writing; if that call returns -EBUSY the function returns not an error but 0, and blkdev_write_iter() sees "zero bytes written but the request is still pending" and routes the request down the buffered path. On a mounted device the filesystem holds its own references to those pages, invalidation fails, and the request goes buffered without its alignment ever being examined. The read side has no such escape hatch; that is why a misaligned read on the same device is EINVAL while a misaligned write is OK.

"The device is mounted" is not a precise enough rule, though, because the invalidation attempt covers only the range being written. I ran it at three different offsets on the same mounted device, this time also marking the path taken by the aligned 4 KiB write:

MOUNTED device, different offsets (baseline in all three: 0 pages cached)
  offset=0           aligned 4096B: OK, cached afterwards=1 | buffer+1: OK
  offset=64 MiB      aligned 4096B: OK, cached afterwards=0 | buffer+1: Invalid argument
  offset=400 MiB     aligned 4096B: OK, cached afterwards=0 | buffer+1: Invalid argument
same device, unmounted
  offset=0           aligned 4096B: OK, cached afterwards=0 | buffer+1: Invalid argument

At offset zero, where the filesystem's superblock and metadata buffers live, even the aligned write goes buffered — I wrote with O_DIRECT and the page appeared in the cache. Sixty-four megabytes further along, in a region the filesystem is not holding at that moment, the same device with the same statx answer behaves completely differently: the aligned write is direct and the misaligned one is EINVAL. So the rule is not "it relaxes if the device is mounted" but "it relaxes if the range you are writing holds pages the kernel cannot invalidate".

And that is exactly why my probe could overwrite the ext4 superblock on the first round: the probe was writing at offset zero. I got success where I expected an error, and the write went down the buffered path and corrupted the filesystem. The lesson is simple: for tools that write raw to a mounted device, the assumption "if I'm misaligned I'll get EINVAL, so I'm safe" is wrong — and being aligned does not save you either.

What to do in practice

If you use O_DIRECT in your own code, here is the checklist the measurement leaves behind:

Read the mask bit before you read the value. If stx_mask & STATX_DIOALIGN is absent, you learned nothing; seeing 0 and concluding "there is no direct I/O" is a wrong inference. btrfs sets exactly this trap.
If it is 0, open with O_DIRECT but do not trust it. The call will not fail and your request will go buffered. If you were using O_DIRECT to avoid polluting the cache, that benefit does not exist on that file; design accordingly.
If there is a value, comply — but measure the write side separately. On pre-6.14 xfs, or on filesystems where statx answers nothing at all, obeying the reported number is not a guarantee of direct I/O. The cheapest verification is mincore(): drop the cache, write, and see whether the page shows up.
Do not assume the number is fixed for the file's lifetime. A small file on an inline_data ext4 reports 0; after the probe wrote 4 KiB, the same file reported 512. A cp --reflink permanently changes the write alignment of an xfs file. Asking once at startup and caching the answer means working with a stale answer, at least on the write path.
If you write raw to a block device, check the mount state yourself. Do not treat EINVAL as a safety net; on a mounted device that net is not there.
Default to 4096 — but expect no magic. In none of the file cases where statx reported a non-zero number did a 4096-aligned request end up on the buffered path; 512-aligned ones did. On files that report 0/0 no alignment helps at all — 4096 goes buffered there too — and in the uninvalidatable range of a mounted device the alignment is never even read. Even so, when the logical sector is 512, pushing your alignment to 4096 costs a few wasted bytes.

We saw a similar split on the device side in the rq_affinity article: between a setting's reported value and its observed behaviour sat a condition nobody had written down in the documentation. The same thing is happening here.

An API being able to say "I don't know"

What this measurement taught me is less about O_DIRECT than about interface design. The mask mechanism in statx() exists for exactly this problem: the call is not obliged to fill in every field you ask for, and it tells you which ones it filled. That is what separates "there is no direct I/O on this file" (value 0) from "I am not providing this information" (no bit). The old stat() family had no such channel, so every field was forced to say something, and the fields with nothing to say lied — that is the story st_blksize has been telling for years.

Of course a channel existing does not mean it gets used. On the kernel side btrfs behaves correctly by not returning the bit at all, xfs added a new field in 6.14 to correct an insufficient answer, and ext4 says "none" with a zero. On the application side, every line that skips the mask bit throws that care away. My own first probe did exactly that: I was reading the value and not the bit, and on btrfs I saw 0 and noted down "so there is no direct I/O here". But btrfs does have direct I/O; what it did not have was an answer.

Measurements taken on 21 October 2026 in the Docker Desktop virtual machine on my own Mac (kernel 6.10.14-linuxkit) inside a privileged Debian trixie container; e2fsprogs 1.47, xfsprogs 6.x, btrfs-progs 6.x. The filesystems live on 512 MiB loop images (logical_block_size=512, dma_alignment=511), with xfs created using -m reflink=1. The probe calls the statx() syscall directly and verifies the struct statx layout against 256 bytes with sizeof. The two critical measurements (xfs reflink and btrfs) were repeated over three rounds each; the block-device offset comparison and the read-path measurement used separate probes (atprobe, rdprobe). The 6.14 xfs fix was not measured; it was read in the source.

Official Sources

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.