df Said Zero, Root Kept Writing

For a long time I never stopped to ask who a filesystem is talking to when it says "100% full". In the lab I filled a 488 MiB ext4 volume to the brim as an ordinary user, df reported zero available space — and then, as root, I wrote another 15 MiB without a single complaint.

dd: error writing '/mnt/t/u/x': No space left on device

At the same moment, on the same filesystem:

15728640 bytes (16 MB, 15 MiB) copied, 0.014573 s, 1.1 GB/s

What stands between those two lines is ext4's reserved blocks. Every mke2fs call quietly sets aside five percent of the disk and hands it only to privileged processes. My blog server's root holds 183 million blocks; five percent of that is roughly 35 GiB — 35 GiB nobody asked about and you are paying for. (The reserve is computed from the total block count, not from the size df reports; df calls that disk 678 GB because it has already subtracted the metadata.) This article is the record of chasing that five percent: where it comes from, who it really protects, what shape it is in across my fleet, and what reclaiming it costs.

Where the reserve lives

The reserve is not a mount option; it is a number inside the superblock. The kernel's ext4 disk-layout documentation defines the field as s_r_blocks_count_lo, at offset 0x8, type __le32 — "This number of blocks can only be allocated by the super-user." Next to it sit s_def_resuid at 0x50 and s_def_resgid at 0x52, holding the user and group entitled to use the reserve. So the reserve does not belong to root; it belongs to whichever uid is written in the superblock. I will come back to that detail, because the most useful part of this whole story lives there.

I built the lab inside Docker Desktop's linuxkit VM: a privileged Ubuntu 24.04.5 container, kernel 6.10.14-linuxkit, e2fsprogs 1.47.0. Since you cannot play with ext4 on macOS directly, a loop device plus --privileged is the shortest path.

dd if=/dev/zero of=disk.img bs=1M count=512 status=none
mke2fs -t ext4 -q disk.img
tune2fs -l disk.img | grep -E "Block count|Reserved block count"

Block count:              131072
Reserved block count:     6553

6553 / 131072 = 4.9995%. Not exactly five, because mke2fs rounds down: 6553 blocks × 4 KiB = 25.6 MiB.

So where does the number five come from? The first place I looked was /etc/mke2fs.conf, and there is nothing there — Ubuntu 24.04's file contains no reserved_ratio line at all. The value is baked into the binary; in the e2fsprogs source, mke2fs.c does this:

reserved_ratio = get_double_from_profile(fs_types, "reserved_ratio", 5.0);

5.0 is a fallback. If you define reserved_ratio in mke2fs.conf — valid under both [defaults] and [fs_types] — every new filesystem is born with that ratio. Most distributions leave it unset, which means nobody actually chooses five percent; you inherit it as a default from the 1990s.

Who stands at the gate

The allocation decision is simple, but the details matter. The kernel recognises three separate gates: holding the CAP_SYS_RESOURCE capability, being the user in s_def_resuid, or belonging to the group in s_def_resgid. So it is not "being root" that counts but holding the capability — a distinction that matters inside containers.

To measure it I set an ordinary user named deneme loose on the 488 MiB filesystem:

Step	df size / used / available	Result
Empty filesystem	488 / 1 / 452 MiB	1%
deneme writes until it stops	488 / 452 / 0 MiB	ENOSPC
root writes 15 MiB	488 / 467 / 0 MiB	Success
deneme tries again	—	ENOSPC

The third row is the striking one: df had already said zero, yet root wrote 15 MiB more and df said zero again. The "Avail" column is statvfs's f_bavail field — free blocks minus the reserve.

This is where you have to be careful, because I got it wrong on my first pass. On the empty filesystem, 488 − 452 − 1 leaves a 35 MiB gap, and not all of it is the reserve we have been discussing. The metadata is already out of the picture: the 512 MiB image reports a df size of 488 MiB, and those missing 24 MiB are the filesystem's own bookkeeping. What remains is the sum of two separate reserves. The kernel's ext4_statfs says so plainly:

buf->f_bavail = buf->f_bfree - (ext4_r_blocks_count(es) + resv_blocks);

That second term is s_resv_clusters, the kernel's own cut. The comment in the source is terse — "By default we reserve 2% or 4096 clusters, whichever is smaller" — and its purpose is to keep operations like delalloc and punch-hole, which still need a block or two when space runs out, from failing. On my 131,072-block filesystem that is 131072 / 50 = 2621 blocks, or 10.2 MiB. Add the 6553 blocks from the superblock and you get 9174 blocks = 35.8 MiB, exactly the gap df shows.

To confirm it I looked at a filesystem created with -m 0, that is, with a zero superblock reserve:

Reserved block count:     0
total=124725 free=124719 available=122098

The difference between free and available is 2621 — the predicted number itself, to the block. So even after tune2fs -m 0, df still withholds two percent of the filesystem (capped at 4096 clusters, 16 MiB with 4 KiB blocks), and you cannot turn that second reserve off with tune2fs. On small volumes the difference is glaring; on production volumes above 100 GB it hits the cap and sits at 16 MiB, so it is negligible in practice. The usage percentage follows the same logic: a filesystem hits "100%" with 452 MiB written while 35 MiB still sits underneath.

The real lesson: df tells you the truth, but on behalf of the wrong user. The df output you read as root shows the space an unprivileged process would see, not the space root has. You can put both side by side with stat -f: %f gives free blocks, %a gives the unprivileged available blocks.

One more trap. If you run tune2fs -l on a mounted filesystem, the "Free blocks" line shows you the last value committed to the superblock. The kernel keeps the free block count in percpu counters and writes it out only at specific moments (umount, sync, periodic commit); in between, the number on disk stays frozen. On my freshly created image it read 124719 while mounted — the creation-time value — and after umount the same command said 2774. On a server that has been up for months that figure is not "creation time" either; it is some arbitrary older moment. The reserve count is accurate, the free-block count is not — never make a decision based on the free-space number in tune2fs -l output on a live system.

What the fleet actually looks like

I went to check how the theory holds up in the field, reading the ext4 volumes of all seven servers I run:

Server	Device	Mount	Block count	Reserved blocks	Ratio
vps1	sda1	/	19,370,235	0	0%
vps1	sda13	/boot	261,888	13,094	5.00%
vps2	sda1	/	25,923,835	0	0%
vps2	sda13	/boot	261,888	13,094	5.00%
vps3	sda1	/	183,238,395	0	0%
vps3	sda16	/boot	233,728	11,686	5.00%
vps4	vda2	/	104,857,083	4,980,650	4.75%
vps5	sda1	/	25,923,835	0	0%
vps5	sda13	/boot	261,888	13,094	5.00%
vps6	sda1	/	10,195,195	0	0%
vps6	sda13	/boot	261,888	13,094	5.00%
vps7	sda1	/	10,195,195	0	0%
vps7	sda13	/boot	261,888	13,094	5.00%

(vps4 has no separate /boot partition; it lives inside the root filesystem.)

Six of the servers have zero reserve on their root filesystem. I did not do that; they all come from the same cloud image, and whoever built that image ran mke2fs -m 0. Meanwhile the /boot partitions on those same machines carry exactly five percent, because those were created separately at install time and left at the default. So a single server hosts two different policies side by side, and nobody chose either one deliberately.

The lone exception is vps4, the machine where I collect logs. Its 393 GB root is 89% full, df reports 45 GB available — and the superblock holds 4,980,650 blocks, that is 19 GiB of reserve. On a log-collecting server, 19 GiB is the distance between the "disk is filling up" alarm and the moment the disk actually fills. That is not a bad thing; it is just insurance you are carrying without knowing it.

Where the 4.75% came from

The only oddity in that table is vps4's ratio. If the default is 5%, why 4.75%? I tried to reproduce it in the lab, and the answer turned out to be about growing.

First I doubled the filesystem while it was unmounted:

e2fsck -fp disk.img
dd if=/dev/zero bs=1M count=512 status=none >> disk.img
resize2fs disk.img

	Block count	Reserved	Ratio
Before	131,072	6,553	4.9995%
After offline grow	262,144	13,106	4.9995%

Flawless: userspace resize2fs preserves the ratio and doubles the reserve exactly. Then I ran the same test while mounted — that is, the way everyone does it in production, an online grow after growpart:

	Block count	Reserved	Ratio
Before	131,072	6,553	4.9995%
After online grow	262,144	11,795	4.4994%

Same operation, different outcome. In an online grow the work is done by the kernel rather than userspace, and the arithmetic in fs/ext4/resize.c looks like this:

reserved_blocks = ext4_r_blocks_count(es) * 100;
reserved_blocks = div64_u64(reserved_blocks, ext4_blocks_count(es));
reserved_blocks *= blocks_count;
do_div(reserved_blocks, 100);

Integer division. 6553 × 100 / 131072 = 4.9995 → 4. The kernel adds four percent, not five, to the newly added space, and the overall ratio slides downward. Every online grow erodes it one more notch.

Whether vps4's current 4.75% was produced exactly that way, I honestly cannot prove: the superblock keeps no resize history, the machine's journal only reaches back to 10 September, and the filesystem was created on 17 July. What I can say is that the mechanism producing this drift measurably exists and happens to everyone who grows a disk. Do not assume your ratio is "five percent" — read it with tune2fs -l.

What the reserve buys you

The tune2fs manual gives two reasons: "to avoid file system fragmentation, and to allow system daemons, such as syslogd, to continue to function correctly after non-privileged processes are prevented from writing." Fragmentation and surviving daemons. I tested both separately.

For fragmentation I built a 1 GiB filesystem created with -m 0 and wrote the same 64 MiB file after two different histories:

Filesystem history	Largest free extent	Free extent count	Extents in the 64 MiB file
Filled to 92%, then half deleted	95,780 KB	59	8
Filled to 100%, then 200 MiB deleted in scattered chunks	3,072 KB	127	18

In the second row the filesystem had 200 MiB free — but the largest contiguous chunk was 3 MB. There was no way left to write a 64 MiB file contiguously, and the same file ended up split into more than twice as many pieces. That is what the reserve really does: it is not a magic anti-fragmentation device, it is a buffer guaranteeing that the allocator's breathing room never reaches zero. Once a filesystem has hit the floor, freeing space afterwards does not bring the old contiguity back.

The second justification — "so daemons keep running" — works far less well in 2026 than it did in 1996. I counted the processes running on vps3 at that moment: of 716 processes, 237 are not root. The breakdown is instructive too: 131 processes under uid 70 (PostgreSQL inside a container), 38 under nginx, 23 CI runners, 15 under ubuntu, 5 belonging to my own bridge service. When the disk fills, these hit the wall first, and a reserve set aside for root does them no good. The reserve was designed for a world where services ran as root; today your most critical writers gave up their privileges long ago.

Reclaiming it: three commands, three different decisions

You do not need to unmount anything to change the size of the reserve. I ran tune2fs -m 0 on a mounted filesystem:

Setting reserved blocks percentage to 0% (0 blocks)

df updated in the same instant — available space went from 0 to 11 MiB (not all of it; the second reserve is still in place) and the user who had just received ENOSPC could write again. Setting -m 5 brought the number back. One thing to note: the command is reversible, but its consequences are not. Once the reserve has been eaten and the disk is genuinely full, writing -m 5 back frees nothing; it merely pins f_bavail at zero. The operation is instant, moves no data, and requires no remount; the only thing that changes is one number in the superblock. (Unlike the figure fstrim prints, this number really does change the space available right then.)

You have three tools, and they answer three different questions:

• tune2fs -m <percent> — sets a ratio. Reasonable on small volumes, dangerous on large ones: on vps3's root, "only five percent" is 35 GiB.
• tune2fs -r <blocks> — sets an absolute count. This is the right tool for large data volumes. When I ran tune2fs -r 1000, the superblock recorded exactly 1000 blocks. On a volume above 100 GB, "I want a 2 GiB buffer" is a far more meaningful statement than "I want five percent".
• tune2fs -u <user> — hands the reserve over. I measured this one separately, because it is the least known, the most useful and the most treacherous.

I compared that last one across two runs: two identical 256 MiB filesystems with a 3276-block reserve, both filled by the deneme user. I performed the handover before mounting:

Setup	How far deneme got	Free blocks left
Reserve owned by root (default)	206 MiB	4,586
After tune2fs -u deneme	219 MiB	1,310

The difference is exactly 3276 blocks — the reserve itself, to the block. On a volume where the database runs as the postgres user, handing the reserve over instead of deleting it is the clean way of saying "when the disk fills, let me be the last writer".

There is a trap here, and it bites in production

That "before mounting" is not incidental. When I ran the same command on a mounted filesystem, here is what happened: tune2fs printed "Setting reserved blocks uid to 1001", tune2fs -l showed the new uid — and nothing changed. deneme still stopped at 206 MiB, still left 4586 blocks free. A silent failure: the command succeeds, the effect does not arrive.

The reason is in the kernel source. The uid written in the superblock is read only inside ext4_fill_super(), that is, at mount time:

sbi->s_resuid = make_kuid(&init_user_ns, ext4_get_resuid(es));

From then on the allocator always consults that in-memory value. This also explains the asymmetry with -m and -r, which take effect instantly: those are read live from the superblock buffer, while the user identity is latched at mount.

A plain mount -o remount is not enough either — I tried that too, and the value did not budge. The only thing that works is passing the option explicitly:

mount -o remount,resuid=1001 /dev/vdb1 /data

After that run deneme wrote 13 MiB more and free blocks dropped from 4586 to 1310 — it really had taken over the reserve. The correct production recipe is to do both: tune2fs -u so it persists, and resuid= in fstab (or a remount) so it applies now. Doing only the first and walking away means changing nothing until the next reboot.

Reading your own volumes

I did not build the table above by hand; I ran this on every server. Its only dependencies are e2fsprogs and util-linux, and the output is a line you can act on directly:

lsblk -nro NAME,FSTYPE | awk '$2=="ext4"{print $1}' | while read -r d; do
  m=$(findmnt -nro TARGET -S "/dev/$d" | head -1)
  sudo tune2fs -l "/dev/$d" | awk -v d="$d" -v m="${m:-unmounted}" '
    /^Block count:/{b=$3} /^Reserved block count:/{r=$4}
    END{printf "%-10s %-6s blocks=%-12s reserved=%-10s ratio=%.2f%%  %.1f GiB\n", d, m, b, r, (b?r/b*100:0), r*4096/1073741824}'
done

On my log server the output is a single line:

vda2       /      blocks=104857083    reserved=4980650    ratio=4.75%  19.0 GiB

That last column is what makes this discussion concrete. "Five percent" reads like an abstract deduction; "19 GiB" reads like a budget line. Note that I assume a 4 KiB block size; if you have a volume created with a different block size, read the Block size line from tune2fs -l and fix the multiplier — on older volumes with 1 KiB blocks the same reserve count occupies a quarter of the space.

The command works for unmounted volumes too; when findmnt returns nothing the script prints "unmounted" instead of a mount point. And remember we do not trust "Free blocks" on a mounted filesystem — here we only read the block count and the reserve, both of which sit unchanged in the superblock.

A decision framework

Here is the order of questions I ask when looking at my own volumes:

1. Does this volume hold a root filesystem? Then leave a buffer — but pin it to an absolute count instead of a ratio. journald and the package manager still being able to write while df says 100% is cheap insurance.
2. Does this volume hold data only? Backups, media archives, log stores, container images — five percent here is pure waste. Leave a fixed buffer of a few GiB with tune2fs -r and take the rest back.
3. Is root the volume's actual writer? If not, either hand the reserve to that user with -u or accept that it is meaningless.
4. Is there an LVM thin pool or a thin-provisioned virtual disk underneath? The 35 GiB you reclaim with tune2fs -m 0 becomes visible at the filesystem layer but does not return to the pool; reporting those blocks downward takes a separate fstrim — and what that number actually means is another article's subject.
5. Did you adjust your alert threshold along with the reserve? The reserve is also your early-warning margin. In node_exporter, node_filesystem_avail_bytes is f_bavail and node_filesystem_free_bytes is f_bfree; once you zero the reserve the two converge, and the "disk filling up" alert lands in the same second as the actual fill. Lower the threshold before you take the reserve.
6. Have you measured your ratio? On any filesystem that has been grown, the real ratio may be smaller than you assume. Read the two lines from tune2fs -l and do the arithmetic yourself.
7. Do your alert thresholds watch df? If they do, on reserved volumes the warning arrives before the real fill, and on unreserved volumes exactly at it. Monitoring both volume types with the same threshold means being late on one of them.

My own takeaway is not numerical. My seven servers carried two different reserve policies, and I had chosen neither: one was decided by the team that built the cloud image, the other by a thirty-year-old default. To see the difference I had to read the superblock. The most expensive settings in infrastructure are the ones nobody remembers deciding — and the night the disk fills is far too late to find out what they say.

Official Sources

• e2fsprogs — tune2fs manual page source (-m, -r, -u, -g)
• e2fsprogs — mke2fs.c, the default reserved_ratio value
• Linux kernel — ext4 superblock fields (s_r_blocks_count_lo, s_def_resuid)
• Linux kernel — fs/ext4/resize.c, reserve arithmetic on online grow
• Linux kernel — fs/ext4/super.c, ext4_statfs and ext4_set_resv_clusters