DEV Community

Cover image for lazytime Didn't Defer mtime but Took 12 Transactions to Zero on Reads
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

lazytime Didn't Defer mtime but Took 12 Transactions to Zero on Reads

The lazytime entry in the mount manual is short and confident: only update timestamps on the in-memory copy of the inode, which "significantly reduces writes to the inode table for workloads that perform frequent random writes to preallocated files". That reads like a description of a database file. This blog's content also lives in a single 58 MiB SQLite file, written into on every publish. "There is a free win sitting here," I thought.

Right underneath, the manual also puts an honest list of exceptions — the on-disk timestamps are updated only when:

  • the inode needs to be updated for some change unrelated to file timestamps,
  • the application employs fsync(2), syncfs(2), or sync(2),
  • an undeleted inode is evicted from memory,
  • more than 24 hours have passed since the inode was written to disk.

Reading that first item, what came to mind was "now and then". I underrated it; the whole story turned out to be hiding in there.

The measurement said otherwise: the win was not there. More precisely: it was not there in exactly the workload the manual gives as its example. It was somewhere else entirely, on the read side nobody advertises, and there the difference between the two mounts was total — 12 journal transactions went to zero.

The reason was hiding in a field that looks unrelated to any of this: i_version.

The rig, and the trick in the measurement

The lab this time is the blog's own server: Ubuntu 24.04.4, kernel 6.8.0-147-generic, e2fsprogs 1.47.0. The root filesystem is ext4 mounted rw,relatime,discard,errors=remount-ro,commit=30 — and there is not a single filesystem on this server mounted with lazytime; grep -c lazytime /proc/mounts returns zero. I touched no global sysctl; the experiment stayed inside its own loop devices: four 512 MiB ext4 images, four separate mount points (two more follow later on, for relatime).

Mount Options Why
nz noatime write control
lz noatime,lazytime write candidate
sa strictatime read control
salz strictatime,lazytime read candidate

The real trick is measuring "the value on disk". The first instinct is to write debugfs -R "stat <12>" /dev/loop7, but that will fool you. ext4 updates the inode table in the block device's buffer cache and leaves it dirty there; if you read the device you will take a value that has not reached the disk yet for an on-disk value. So I read not the loop device but the image file underneath it: debugfs -R "stat <12>" /var/tmp/lazylab4/lz.img. The content of the image file only changes when the filesystem actually sends a write to the loop device.

I did not assume the method worked; I proved it with a deliberate positive. On the mount without lazytime, the very same read showed the expected new timestamp on time. Had the method been blind, that line would have stayed old too.

The files are 64 MiB and fully written with dd — filefrag -v reports zero unwritten extents. I note this deliberately, because my first attempt allocated the file with fallocate and that round carried a confounding variable: on ext4, fallocate leaves unwritten extents, the first write converts one, and the conversion dirties the inode for a reason that has nothing to do with timestamps — that is, the first item on the manual's own exception list. To see the mechanism cleanly I wrote the file out with dd and removed that variable. (For the curious: after two separate 60-second rounds of random writes, that file was left with 3,347 separate unwritten extents.)

First surprise: touch is never deferred

I ran the easiest test first. I wrote a 2 January timestamp onto the file in both mounts with touch -m -t 202601020304.05, waited 70 seconds, and read the images. Both had pushed the new timestamp to disk. lazytime had made no difference.

That is not a bug, it is a difference of scope. lazytime defers implicit timestamp updates — the ones produced as a side effect of a write or a read. touch calls utimensat(2); that call goes through notify_change() into the filesystem's setattr operation, and there ext4 does this explicitly:

setattr_copy(idmap, inode, attr);
mark_inode_dirty(inode);
Enter fullscreen mode Exit fullscreen mode

mark_inode_dirty() dirties the inode in full. The entrance to the deferral path is never visited. If you have a script that aligns your backup files' timestamps with touch -r, lazytime is reducing none of those writes.

So far the story agrees with the manual. The break comes in the next test.

Second surprise: write-driven mtime isn't deferred either

The real experiment is this: an in-place write that changes no size and carries no fsync. The only things that change on the file should be mtime and ctime — the textbook case for deferral.

I issued a single 4 KiB in-place write and looked three times:

-- 0 s : nz disk=09:04:32 | lz disk=09:04:32
-- 40 s: nz disk=09:04:32 | lz disk=09:04:32
-- 80 s: nz disk=09:04:37 | lz disk=09:04:37
Enter fullscreen mode Exit fullscreen mode

The in-memory timestamp was 09:04:37 on both. At the fortieth second both were old; at the eightieth both were new. The two mounts behaved identically, line for line. (I did not narrow down the exact second at which the write landed; the finding here is that there was no difference between them.)

One write is not convincing, so I moved to quantity. On both mounts, for 60 seconds, at 10 ms intervals, random 4 KiB in-place writes into the 64 MiB file — same seed, no fsync. I did not count disk sectors as the metric; data writes push the noise far above the effect being measured. Instead I looked straight at the journal: the jbd2 transaction count on the first line of /proc/fs/jbd2/<device>/info. Because data writes in data=ordered mode spawn no new metadata transaction, that counter answers almost purely the question "was the inode dirtied?"

Workload Mount Calls jbd2 transactions
60 s in-place write noatime 5,455 12
60 s in-place write noatime,lazytime 5,454 12
60 s read strictatime 5,488 12
60 s read strictatime,lazytime 5,633 0
60 s read relatime 5,529 1
60 s read relatime,lazytime 5,325 0

No difference in the write rows: 12 and 12. The 12 is no coincidence: ext4's default journal commit interval is 5 seconds, which over 60 seconds is exactly 12 commits. So under roughly 91 writes a second, the inode lands in the journal once per commit interval — with lazytime on and with it off. (I read the counters once, at the end of all the runs; ext4 produces no transactions while idle, but the deltas come from that shared window rather than from a strict 60-second one.) Because the four workloads ran one after another, at the check that closed the measurement — about 3.5 minutes later for nz, 2.5 for lz — the on-disk mtime equalled the in-memory value on both. lazytime deferred nothing at all in the manual's own example workload.

I had run the same comparison once before in a separate round, that time with the file allocated via fallocate, and it came out 13 to 13. Two independent runs point the same way.

I will come back to the last row; first, the "why".

I asked the kernel which flag it uses

I did not want to close the claim with a guess, so I asked the kernel. The writeback tracepoints print, line by line, which flag reaches __mark_inode_dirty(). To keep the global tracer untouched, I opened a separate tracing instance.

An in-place write on the lazytime mount:

writeback_mark_inode_dirty: bdi 7:7: ino=12 state=I_DIRTY_SYNC|I_DIRTY_PAGES flags=I_DIRTY_SYNC|I_DIRTY_TIME
Enter fullscreen mode Exit fullscreen mode

A read on the lazytime mount:

writeback_mark_inode_dirty: bdi 7:9: ino=12 state=I_DIRTY_TIME flags=I_DIRTY_TIME
Enter fullscreen mode Exit fullscreen mode

The field to read is flags: the first carries I_DIRTY_SYNC, the second does not. I_DIRTY_SYNC means "write this inode out"; having it sit next to I_DIRTY_TIME is not deferral, it is the cancellation of deferral. On the write path, something kills the deferral before it is even born.

generic_update_time hands over the culprit in two lines

Kernel 6.8's fs/inode.c makes the decision in two lines:

    if (updated & (S_ATIME|S_MTIME|S_CTIME))
        dirty_flags = inode->i_sb->s_flags & SB_LAZYTIME ? I_DIRTY_TIME : I_DIRTY_SYNC;
    if (updated & S_VERSION)
        dirty_flags |= I_DIRTY_SYNC;
Enter fullscreen mode Exit fullscreen mode

The first line is what we expect: if the superblock carries SB_LAZYTIME, the flag becomes I_DIRTY_TIME. The second line puts I_DIRTY_SYNC back if S_VERSION was updated. Where does S_VERSION come from? From the function just above it:

        if (IS_I_VERSION(inode) && inode_maybe_inc_iversion(inode, updated))
            updated |= S_VERSION;
Enter fullscreen mode Exit fullscreen mode

The second argument of inode_maybe_inc_iversion() is force. Normally the i_version counter is only incremented if someone has queried it — the comment in fs/libfs.c says so outright: "If this function returns false, and no other metadata has changed, then we can avoid logging the metadata." But here updated is passed instead of a plain false, and on the write path updated already contains S_MTIME; it is non-zero; the force is on. The counter increments every time, S_VERSION gets marked, and I_DIRTY_SYNC comes back.

The only remaining question is whether IS_I_VERSION is on for ext4. fs/ext4/super.c answers in two lines, comment included:

    /* i_version is always enabled now */
    sb->s_flags |= SB_I_VERSION;
Enter fullscreen mode Exit fullscreen mode

Always. There used to be an i_version mount option; it now sits in the parser as Opt_removed. On ext4, this path closing is not a matter of choice.

Why does the read side escape? Because the i_version bump is buried inside the branch that asks for S_MTIME|S_CTIME|S_VERSION. The read path calls inode_update_time(inode, S_ATIME) through touch_atime() and never enters that branch. updated stays S_ATIME alone, S_VERSION is never marked, and the flag remains a pure I_DIRTY_TIME.

Diagram

The read side: 12 transactions, then zero

That is why the last row of the table is so sharp. On the strictatime mount, 5,488 reads in 60 seconds produced 12 jbd2 transactions. On strictatime,lazytime, 5,633 reads produced zero. This too is one run per mount; I did not repeat it. A single read shows the same thing: the mount without lazytime put the new atime on disk within 40 seconds; on the mount with it, the old value was still there after 80 seconds — and, more than that, it was still 09:04:32 seven minutes and 5,633 reads later.

The win is real, but let us not inflate its scope, because strictatime is not how most systems are configured. The default since Linux 2.6.30 is relatime, which only updates the access timestamp when the previous one is older than or equal to mtime/ctime, plus once a day. My own server's root is relatime too. Rather than leave that to inference, I brought up a fifth and sixth mount and ran the same read workload: on relatime, 5,529 reads produced 1 jbd2 transaction; on relatime,lazytime, 5,325 reads produced 0. The win survives, but its scale changes — not 12 to 0 but 1 to 0. relatime has already done most of the work, and a single write is all lazytime has left to take.

The deferral is still visible: on the relatime,lazytime mount the in-memory atime read 09:38:45 at the end of the run while the on-disk one was still 09:37:42.

Let me also say that I did not measure lazytime's waiting horizon. In fs/fs-writeback.c, dirtytime_expire_interval starts at 12 hours, and the code's own comment says that in the worst case "a few inodes might not their timestamps updated for 24 hours". On the server, vm.dirtytime_expire_seconds read exactly 43200, that is 12 hours. I did not build a measurement that waits twelve hours; that number comes from the source.

Is it broken on your kernel? A thirty-second check

Probably the most useful part of this article is that you need not consult a version table — you can see it directly on your own machine. On a filesystem mounted with lazytime, issue a single size-preserving write to a file that already exists, and watch which flag the kernel sets:

TI=/sys/kernel/tracing/instances/lz; mkdir -p $TI
echo 1 > $TI/events/writeback/writeback_mark_inode_dirty/enable
echo > $TI/trace
dd if=/dev/urandom of=/mnt/POINT/file bs=4096 count=1 conv=notrunc status=none
grep -v '^#' $TI/trace | grep "ino=$(stat -c %i /mnt/POINT/file) "
echo 0 > $TI/events/writeback/writeback_mark_inode_dirty/enable; rmdir $TI
Enter fullscreen mode Exit fullscreen mode

If the flags= field on the resulting line holds only I_DIRTY_TIME, deferral is working. If I_DIRTY_SYNC sits next to it, you are in the situation this article describes. On my 6.8 the line came out as:

writeback_mark_inode_dirty: bdi 7:7: ino=12 state=I_DIRTY_TIME flags=I_DIRTY_SYNC|I_DIRTY_TIME
Enter fullscreen mode Exit fullscreen mode

I could not measure a difference on the fdatasync side

The actual complaint in the upstream fix was about performance: the forced i_version increment sets I_DIRTY_SYNC, which then triggers expensive log flushes on subsequent fdatasync calls. I tried to measure that on my own kernel: 2,000 rounds of (4 KiB in-place write + fdatasync), two runs per mount.

nz   (noatime)            mean=4.551 ms   p50=3.531
lz   (noatime,lazytime)   mean=4.112 ms   p50=3.336
nz   2nd run              mean=3.735 ms   p50=3.084
lz   2nd run              mean=5.174 ms   p50=3.648
Enter fullscreen mode Exit fullscreen mode

The ordering swaps between runs; the gap between two runs of the same mount is larger than the gap between the mounts. I draw no direction from this: on this kernel I could not measure lazytime's effect on fdatasync latency. Which is what you would expect anyway — if I_DIRTY_SYNC is set on both mounts, fdatasync has the same work to do in both.

A side finding: lazytime belongs to the superblock, noatime to the mount

While chasing this I also learned something that does not exist. In /proc/mounts, lazytime looks like a mount option, but it is not one; in fs/proc_namespace.c it sits in the superblock flag table, while noatime and relatime sit in the mount flag table. What that means in practice:

mount --bind /mnt/lab2-lz /mnt/lab2-bind
mount -o remount,strictatime /mnt/lab2-bind   # only the bind changed
mount -o remount,nolazytime  /mnt/lab2-bind   # BOTH lost lazytime
Enter fullscreen mode Exit fullscreen mode

You can mount the same filesystem in two places and keep noatime on one and relatime on the other. lazytime has no such privilege: turning it off on one mount turns it off everywhere. An automation that believes it is tuning metadata-write behaviour per mount goes quietly wrong here. I met a similar distinction while tuning the thresholds of writeback: page-cache thresholds are system-wide, and it is easy to take those for per-filesystem as well.

The version story: two years and nine months, and no LTS

This behaviour was not always there, and it will not stay. The git history gives clean dates:

Version inode_maybe_inc_iversion() call mtime deferred?
v5.15, v6.1 (inode, false) yes (unless queried)
v6.6 → v7.0 (inode, updated) / (inode, !!dirty) no
v7.1, v7.2 (inode, dirty != I_DIRTY_TIME) yes

The forcing arrived in v6.6 through a cleanup aimed at something else entirely: 541d4c798a59 ("fs: drop the timespec64 arg from generic_update_time", August 2023) was opening the way to tracking whether ctime had been queried, and in the process the false argument became updated. The fix is 834e98acb748 ("fs: fix forced iversion increment on lazytime timestamp updates", 11 May 2026) and it landed in mainline in v7.1. The fix's Fixes tag points at a January 2026 cleanup; yet on the 6.8 kernel under measurement here, the behaviour is identical, so the window of breakage is roughly two and a half years wider than the tag suggests.

What is in the field today? While writing this I looked at the sources of the current long-term kernels: in v6.6.158, v6.12.112 and v6.18.56 the call is still (inode, updated). (v6.1 and v5.15 are clean, with (inode, false).) The fix has been backported to no LTS — and since the Fixes tag points at a commit that does not exist in 6.x, an automatic backport is not to be expected either. So on Ubuntu 24.04 (6.8) and on every distribution based on 6.6, 6.12 or 6.18, lazytime is ineffective for mtime.

The blast radius is not limited to ext4. XFS sets the flag on V5-superblock filesystems, and its own xfs_vn_update_time falls through to generic_update_time() when lazytime is on — so it meets the same forcing. The V4 format, meanwhile, is marked deprecated in the kernel documentation, with support to be removed entirely in September 2030, so this door will not close on the XFS side; the discussion behind the fix was in fact opened on the XFS list.

I had put btrfs in the same basket, and that was wrong. btrfs does set SB_I_VERSION unconditionally, but it writes its own timestamp path: .update_time = btrfs_update_time, and that function never looks at SB_LAZYTIME — if a timestamp changed it calls btrfs_dirty_inode() directly. So on btrfs lazytime does nothing on writes or on reads, and not because of the i_version chain this article traces, but because the filesystem never asks about the option. The lesson I take: a flag being set on the superblock does not mean any code reads it.

Among those that do not set SB_I_VERSION are f2fs and ext2. None of these three went through my lab; I only read the relevant lines in the source.

A decision frame

The list I drew up for my own servers:

  • If your kernel is below v7.1 and your filesystem is ext4 or XFS, do not add lazytime in order to reduce mtime writes. The measurement said 12 against 12. On btrfs the option is never read at all, so it sits outside both writes and reads.
  • If you use relatime — that is, if you are on the default — the read-side win comes down to 1 against 0; on noatime there is nothing left to defer at all.
  • If you have a workload that requires strictatime — mail servers, some backup and audit tools look at the access timestamp — lazytime genuinely works here; in my ext4 measurement it took 12 journal transactions down to zero over 60 seconds. That result comes from one run per mount; I did not repeat it.
  • If you host a database, the answer is neutral, not harmful. I could find no measurable difference in fdatasync latency between the two mounts, so on these kernels lazytime neither gains nor costs. Leaving it on is free — just do not build a capacity plan on a win that is not there.
  • Account for timestamp durability. What follows from the manual's four triggers is this: if the system dies before any of them fires, the pending timestamp is lost. The manual does not spell that out — it is my inference from the list; if a tool does incremental work based on mtime, choose this knowingly.
  • The presence of the option does not mean it is working. Seeing lazytime in /proc/mounts does not show that behaviour changed; measure it.

That last item is the lesson this run left me with. When you turn on a mount option, the system does not object, /proc/mounts shows the option, and the manual explains what it is for. All three are telling the truth, and none of them answers "is it working right now?" The only thing that answers that question is doing the same job twice and putting the numbers side by side. Until I put the numbers side by side this time, I thought I had taken the win.

Official Sources

Top comments (0)