Mustafa ERBAY

Posted on Oct 20 Originally published at mustafaerbay.com.tr

The File Was 99.98% Zeros and fallocate Freed Nothing

#linux #filesystem #ext4 #sysadmin

There is a file on my Mac that is one terabyte. Exactly 1,099,511,627,776 bytes. It sits there happily even though I only have 81 GiB of free disk, because it really occupies 16.6 GiB. It is Docker Desktop's virtual disk image, Docker.raw. ls calls it a terabyte, du -h calls it 17G. Both are telling the truth.

This article explains why those two numbers differ and how to turn the difference to your advantage. But the part I actually want to talk about is the opposite case: when you cannot close the gap. In the lab I ran fallocate -d against a file whose contents were 99.98% zeros and got back not a single byte. That was not a bug or a defect. It was the design.

Every measurement here was taken on 20 October 2026, inside Docker Desktop's linuxkit VM (kernel 6.10.14-linuxkit, ext4, 4096-byte blocks) and on my own blog server. I am quoting the commands verbatim so you can repeat them.

Two different questions, two different answers

There are two separate questions you can ask about a file, and most people think they are the same question:

How big is this file? What is the offset of its last byte? That is the apparent size.
How much disk does it occupy? How many blocks did the filesystem allocate to it?

For an ordinary file the two are nearly identical. For a file with holes — a sparse file — there is a chasm between them. A hole is a range of the file that no block backs: read it and you get zeros, but those zeros are not stored anywhere.

The right way to measure:

stat -c '%s bytes apparent, %b x 512 bytes on disk' file
du -B1 file                      # space actually occupied
du --apparent-size -B1 file      # apparent size

There is a classic trap here, and I fell into it once myself: du -b does not report occupied space. Its own man page says so plainly — -b is "equivalent to --apparent-size --block-size=1". In other words du -b tells you exactly what ls -l tells you. If your measurement script uses du -b, you think you are measuring sparse files while measuring nothing at all.

Here is what five commands said about the same 1 GiB fully punched file in the lab:

Command	Output (bytes)
`du -B1`	0
`du -b`	1,073,741,824
`du --apparent-size -B1`	1,073,741,824
`ls -l` (5th column)	1,073,741,824
`ls -s --block-size=1`	0

Three of the five agree, and the thing they agree on is not the thing you were looking for. The du man page frames it carefully: the apparent size "is usually smaller", but it "may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like". For a punched file you are firmly in the second case, and the gap is measured in gigabytes.

Digging the hole yourself: `fallocate -d`

If you have a file that is full of real, block-consuming zeros, you can make it sparse in place. That is precisely what util-linux's fallocate -d (--dig-holes) does; the man page says "Detect and dig holes. This makes the file sparse in-place, without using extra disk space", and describes it as doing "a cp --sparse and then renaming the destination file to the original, without the need for extra disk space". Underneath, it passes FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE to the kernel's fallocate(2). The two flags are inseparable — the man page says the first "must be ORed with FALLOC_FL_KEEP_SIZE" — and the second is exactly why the file size never moves.

Let's try it. Writing 1 GiB of zeros:

dd if=/dev/zero of=zeros.bin bs=1M count=1024 status=none
# apparent=1,073,741,824 B   on disk=1,073,745,920 B

The on-disk figure is 4096 bytes larger than the apparent size; that extra block is the file's extent tree. Now the hole:

fallocate -d zeros.bin
# elapsed=0.43 s
# apparent=1,073,741,824 B   on disk=0 B

The filesystem's used space dropped from 19,575,844,864 bytes to 18,502,098,944: exactly 1,073,745,920 bytes, the file's entire weight on disk. filefrag now reports "0 extents found". The content also did not change: md5sum is still cd573cfaace07e7949bc0c46028904ff, byte-for-byte the digest of 1 GiB of pure zeros. The fallocate(2) man page guarantees this — reads from a punched range return zeros, and the file size reported by stat does not change.

A gigabyte in half a second. Tempting. It would be very easy to stop here and say "let me run this across the whole server". Don't.

99.98% of the file is zeros, and the gain is zero

The smallest unit you can punch is a block. The one-sentence warning in fallocate(1) is this: "The minimum size of the hole depends on filesystem I/O block size (usually 4096 bytes)." fallocate(2) says the same thing from the kernel's side: within the given range, partial blocks are zeroed and only whole blocks are removed from the file.

To see what that means in practice, three 64 MiB files went onto the lab volume. All three are almost entirely zeros, but the non-zero bytes are placed differently:

g4096.bin — the first byte of every 4 KiB block is X, the rest zeros. 16,384 bytes of data in total.
g8192.bin — one X every 8 KiB, so half the blocks are completely empty. 8,192 bytes of data.
gmis.bin — two X bytes every 8 KiB: one in the last byte of a block, one in the first byte of the next. 16,384 bytes of data.

I ran fallocate -d on all three:

File	Non-zero bytes	Before (on disk)	After (on disk)	Reclaimed
`g4096.bin`	16,384	67,108,864	67,108,864	0
`g8192.bin`	8,192	67,108,864	33,660,928	33,447,936 (49.8%)
`gmis.bin`	16,384	67,108,864	67,108,864	0

g4096.bin was 99.9756% zeros — 99.98% when rounded. Nothing was reclaimed. Neither was anything from gmis.bin. The only difference: in g8192.bin half the blocks were zero from start to finish, while in the other two every block carried one byte.

That is why "how much of the file is zeros" is the wrong question. The right one is: how many whole blocks are zero end to end? In database files, logs and backup archives, zeros are usually scattered rather than clustered. A zero that does not keep company with other zeros is a zero you cannot reclaim.

Look at the 33,660,928 bytes g8192.bin kept, too: 8,192 full data blocks come to 33,554,432 bytes, and the remaining 26 blocks are the extent tree itself. The more holes you dig, the bigger the bookkeeping gets. I'll come back to that.

Copying fills the hole back in

This is the genuinely annoying part. You dug the hole, you saved the space, and then the first tool that touches the file can undo all of it. Measured with the same 1 GiB file:

Operation	Source file	Result (on disk)
`cp` (default)	zero-filled, no holes	1,073,745,920
`cp --sparse=always`	zero-filled, no holes	0
`cp --sparse=never`	zero-filled, no holes	1,073,745,920
`cp` (default)	punched	0
`cat src > dst`	punched	1,073,745,920
`tar -cf` + `tar -xf`	punched	1,073,745,920
`tar -Scf` + `tar -xf`	punched	0
`rsync -a`	punched	1,073,745,920
`rsync -aS`	punched	0

The distinction lives in one sentence of the cp man page: in the default --sparse=auto mode, "sparse SOURCE files are detected by a crude heuristic". So cp recognises and preserves existing holes; it does not try to turn genuinely written zeros into holes. For that you have to say --sparse=always out loud.

tar and rsync are harsher: neither cares about sparseness by default. Without -S, the archive itself came to 1,073,745,920 bytes; with -S, the archive of the same file was 10,240 bytes. A hundred-thousand-fold difference. If a backup script running at midnight tried to archive the Docker.raw above without -S, it would write a 1 TiB archive for a file that occupies 16.6 GiB — and restore it as a 1 TiB file.

There is one more subtlety in rsync: the man page notes that combining --sparse with --inplace may not produce sparse blocks "with some combinations of kernel version and/or filesystem type", and that versions older than 3.1.3 reject the combination outright.

Version warning: `cp`'s sparse behaviour keeps moving

I am not writing this from memory, because memory is unreliable here. The coreutils NEWS file records change after change in this area over the last five years: 9.0 (24 September 2021) moved cp onto copy_file_range and lseek+SEEK_HOLE, 9.2 (20 March 2023) made the default --sparse=auto mode use copy_file_range for sparse files as well, 9.8 (22 September 2025) fixed a bug where cp --sparse=always missed opportunities to create holes, 9.9 (10 November 2025) partly reverted that ("may again, like with versions before v9.8"), and 9.10 (4 February 2026) fixed cp, install and mv entering an infinite loop while copying sparse files with SEEK_HOLE. The current release is 9.12 (14 September 2026).

So "cp behaves like this" is a sentence with a version attached. The same test ran on two releases: coreutils 9.1 on Debian 12 and 9.7 on Debian 13, and the table above held on both — though, to be honest, neither release contains any of the 9.8-9.10 changes listed above, so none of them were actually exercised. Verify it once on yours; cp --version costs thirty seconds.

Where the real win is: the space under the image

Collecting the zeros inside an ordinary file is rarely profitable. The real win is in files that carry a filesystem inside them — virtual machine and container disk images.

This is one layer above the rig in my fstrim article, where the same loop lab ran at a smaller scale. Here it is scaled up to answer a different question: when exactly does the image shrink?

The scenario. A 2 GiB raw image with ext4 inside:

truncate -s 2G disk.img        # apparent 2 GiB, on disk 0 B
mkfs.ext4 -q -F disk.img       # on disk 68,317,184 B
mount -o loop disk.img /mnt/g
dd if=/dev/urandom of=/mnt/g/data.bin bs=1M count=1024
# on disk 1,142,095,872 B

Now I delete the file inside the guest and call sync:

rm /mnt/g/data.bin; sync
df -h /mnt/g     # used 24K, free 1.8G
# image: on disk 1,142,095,872 B  -- unchanged

The guest says it is empty and the image file did not shrink by a single byte. The deletion happened in the filesystem's own books; nobody told the block layer underneath that those blocks were no longer owned. That takes an fstrim:

fstrim -v /mnt/g
# /mnt/g: 1.9 GiB (2040348672 bytes) trimmed
# image: on disk 68,284,416 B

One command, 1,073,811,456 bytes. But put the two numbers side by side: fstrim reported 1.9 GiB (2,040,348,672 bytes) and the image shrank by 1.0 GiB. The reported figure is almost twice the real gain, because trim also counts space that was never allocated in the first place. That is precisely the thesis of my previous article, and it is no accident that it shows up again here.

Repeating the same experiment with -o discard, the image stood at 974,323,712 bytes immediately after rm and sync — continuous discard had done only 160 MiB of the work at that moment, and fstrim still had to finish the rest. Continuous discard does not mean instant and complete.

The difference here is that what sits below is not an SSD but a file, so the kernel turns the discard request into a hole punch and the gain shows up directly in stat rather than in df. The general rule follows from that: space only comes back when somebody tells the layer below that the blocks are free. fallocate -d tells the filesystem, fstrim tells the block layer, discard tells it continuously. If nobody speaks, nothing happens — which is half the story behind the Docker install that ate 56 GB in a day, too.

My Mac's Docker.raw is the live example. When the lab started, du -h reported 16G for it; by the end of the experiments it had grown to 25,440,837,632 bytes (23.69 GiB). After the experiment volume and the temporary images were removed it read 17,820,278,784 bytes, or 16.60 GiB. The apparent size never moved from 1 TiB. Docker Desktop's VM passes discard down, and APFS punches the hole.

What does it cost?

Digging holes is not free; you pay in the extent tree. I wrote one byte every 8 KiB into a 512 MiB file and punched it:

Left on disk: 269,230,080 bytes
Pure data: 65,536 blocks x 4096 = 268,435,456 bytes
Difference: 794,624 bytes — 194 blocks of pure "what lives where" bookkeeping

An overhead of 0.3% — an acceptable price, but a price you have to budget for. There is a small measurement trap here as well: for this file filefrag prints "1 extent found" in its summary line, while filefrag -v lists 65,536 extent rows (plus header and summary lines). The summary treats the file as unfragmented because the physical layout follows the logical order. If you want to know how many holes you made, count the extent rows of the verbose output, not the summary.

The real limits are these:

Not every filesystem supports it. The fallocate(2) man page says the operation works on "at least the following filesystems": XFS (since Linux 2.6.38), ext4 (3.0), tmpfs (3.5), Btrfs (3.7), gfs2 (4.16). The list is open-ended, but on a filesystem that does not support it the call returns an error.
If something else still holds the block, nothing is freed. A snapshot, an LVM thin pool, Btrfs/ZFS copy-on-write or a reflink copy referencing the same blocks will keep the number exactly where it was, whether you punch holes or run fstrim. This is the most common surprise in the field; answer "who else is looking at this block" first.
discard is not free and it leaks information. On an encrypted volume, continuous discard exposes occupancy and block patterns to the layer below; inside a VM, the guest's discard tells the hypervisor about tenant usage. Weigh that before enabling allow_discards on LUKS.
There is no fallocate on macOS. APFS does keep sparse files — the 1 GiB sparse file I created on my Mac occupied 16,384 bytes and cp preserved that — but cat src > dst inflated it to 1,073,741,824 bytes instantly. There is no equivalent of Linux's cp --sparse=always either.
lseek makes no promises. Hole scanning rests on SEEK_HOLE/SEEK_DATA, and the lseek(2) man page is explicit: a filesystem is not obliged to report holes, so "these operations are not a guaranteed mechanism for mapping the storage space actually allocated to a file".
Do not run it on a file being written. fallocate -d reads the file and removes the whole blocks it finds to be zero; if someone is writing there at the same time, nothing locks the window between the scan and the punch.

What I found on my own server

Having learned all of this, I ran it against my blog server. First the scan — the only correct thing to do before changing anything. On a busy machine it is a metadata walk over a hundred thousand files, so wrap it in nice -n19 ionice -c3:

find / -xdev -type f -size +64k -printf '%s %b\n' 2>/dev/null |
  awk '{n++; app+=$1; alloc+=$2*512; d=$1-$2*512;
        if (d>0) {h++; gap+=d}}
       END {printf "files=%d  apparent=%d  allocated=%d\n", n, app, alloc;
            printf "with_holes=%d  total_gap=%d B (%.2f GiB)\n", h, gap, gap/1073741824}'

The result:

files=117002  apparent=312179897502  allocated=312109670400
with_holes=330  total_gap=289615432 B (0.27 GiB)

And the denominator:

df -B1 --output=size,used,avail /
#          726958051328  323904327680  403036946432

Of 117 thousand files, only 330 have an apparent size larger than their allocation, and the total gap is 289,615,432 bytes. Against the 323,904,327,680 bytes in use (301.7 GiB) that is 0.089%. The largest single gap is in buildkit's metadata_v2.db: 338,894,848 bytes apparent against 322,117,632 allocated — already 95% full.

This scan has a blind spot worth naming: -xdev stays on the root filesystem. If /var/lib/docker lives on a separate disk, the scan never sees it. It is the same trap as --listed-in skipping volumes outside fstab in my fstrim article; run it once per mount point.

So there is nothing to be won by digging holes on my server. That is not a disappointment, it is the finding: the scan cancelled an hour of optimisation before it started. I never ran fallocate -d at all. If you are tight on disk, sparseness is unlikely to be your problem; the odds favour old container images and container files copied layer upon layer.

Checklist

When a disk-full alarm wakes you up, in order:

Measure, don't guess. Run the find | awk scan above. If the prize is under 1 GiB, close this road.
Don't use du -b. Use du -B1 for occupied space and du --apparent-size -B1 for apparent size.
Is it a disk image? Then it is not fallocate you want but fstrim inside the guest; for a permanent fix, mount the guest with -o discard or install a weekly fstrim.timer.
If it is an ordinary file, make sure whole blocks are zero. If they are not, fallocate -d will reclaim nothing.
Who else is holding this block? A snapshot, thin pool or reflink copy means nothing will be freed; clear those first.
Is the service writing to the file stopped? If not, leave it alone.
Audit your backup and copy paths. -S for tar, -S for rsync, --sparse=always for cp. Forgetting one of them restores the space you saved on the destination machine.
Check the versions. cp --version and rsync --version; the sparse behaviour of these tools has shifted several times in the last five years.
Monitor what you won. The image reinflates after every build. Without a weekly fstrim.timer or a simple metric tracking the image's stat -c %b, you will get the same alarm in a fortnight.

Closing

Sparse files remind me of my favourite property of filesystems: most of the time, the cheapest way to store something is not to write it at all. But that same cheapness quietly evaporates wherever two layers are not talking — the guest deleted, the host never heard; the hole was dug, the backup filled it in.

Disk space is a bookkeeping problem, not a physics problem. Whether a block is really there depends on whether somebody struck that line out of a ledger; fallocate -d, fstrim, -o discard and tar -S are the same job written into different ledgers. If you don't know which one you are writing to, you will hand back the space you thought you had won on the very next copy.

And sometimes, even when 99.98% of the file is zeros, there is nothing to win.