The data volume on this laptop is 426.33 GiB full today, with 12.54 GiB free. Ninety-eight
percent. I have learned to live with a disk like this: delete build output, collect the
worktrees, prune the Docker images, and end up in the same place a week later.
Today I asked a different question. Instead of hunting for something to delete, I wondered
how many times I had paid for the same thing. Under ~/IdeaProjects there are 861 files
larger than 8 MiB; among them I found 36 content groups that are byte-for-byte identical.
The surplus across those groups adds up to 76.20 GiB.
This is where I expected to cheer up: APFS supports clones, so maybe most of that 76 GiB is
shared blocks — present on paper, absent on disk. du cannot tell me. Neither can df. The
only way to find out was to ask the file system whether two files use the same physical
blocks — and I could not find a ready-made command that asks.
So I wrote one. The answer: 74.03 GiB of that 76.20 GiB has really been paid for. Only
2.17 GiB of it is cloned. In other words, roughly six times my remaining free space has gone
into writing bytes I already owned a second, a third, a seventy-ninth time.
Copying Has Two Prices
The first attempt to price a copy fell apart on my own disk: with the machine working, the
free-space counter moved 468 MiB in five seconds. Backups, compiler caches, the browser —
the background never stops writing. On ground like that a two-megabyte difference is not
measurable.
So the measurement moved off the machine's own volume: a separate 6 GiB APFS image, mounted,
with the source and all copies inside it. Nothing but this experiment writes there; idle,
six df samples three seconds apart drifted by at most 8 KiB. That is the resolution
from here on.
The source: 512 MiB as eight 64 MiB random files, incompressible and undeduplicable. Every
tool against the same source, three rounds. The figure is the df -k delta before and after.
| Tool | Taken from disk (KiB, median of 3 runs) | What du says |
|---|---|---|
cp -R |
524,288 | 524,288 |
cp -Rc |
0 | 524,288 |
ditto |
524,288 | 524,288 |
ditto --clone |
0 | 524,288 |
rsync -a (openrsync) |
524,292 | 524,288 |
| `tar \ | tar` | 524,288 |
The zero on the clone rows does not mean "very little": in two of three rounds the counter
did not move at all, and in the third it moved 8 KiB — the cost stayed under my measuring
resolution. Between the two prices of copying a half-gigabyte tree there is a factor of at
least 65,000.
The right-hand column is the real reason to read the table. du says the same thing on all
six rows: half a gigabyte. What the disk actually loses is either half a gigabyte or nothing.
Two small surprises turned up. ditto — Apple's own copy tool — does not clone by default;
the --clone flag is required. And the rsync in macOS 26 is openrsync (protocol 29), which
also pays full price. My habit for years has been rsync -a, because it shows progress and
resumes where it left off. That habit was quietly charging me disk.
What Exactly Does a Clone Share?
The clonefile(2) manual page explains the mechanism in two sentences: the cloned file
"shares its data blocks with the src file" but has its own copy of attributes; and
"subsequent writes to either the original or cloned file are private to the file being
modified (copy-on-write)". The call has existed since 10.12 — September 2016 — so it has been
within reach for ten years.
To price the sharing, 2,000 clones of a 1 MiB file went onto the isolated volume. Total
cost: 644 KiB, or 330 bytes per clone. Nominally that is 2 GiB of files; half a megabyte
left the disk. A clone is not the file, it is a record saying "those blocks are mine too" —
and that record is priced by file count, not by file size.
Then I asked the obvious follow-up: if I change a single byte in a cloned file, does the whole
file split? I wrote one byte into each of those 2,000 clones. Total 32,304 KiB, that is
16.15 KiB per write. The volume's allocation block is 4,096 bytes, so what splits is the
block, not the file — plus two or three blocks' worth of bookkeeping. I could not break that
overhead down with the tools I have; 15 KiB is a measured number, not a derived one.
What that means in practice is that opening a cloned 50 GiB disk image and working inside it
does not repay 50 GiB at once. You only pay for the blocks you touch. For my lab images that changes
the arithmetic completely.
No Tool Tells Me Which Price I Paid
After producing a clone I asked stat about the source and the clone. For a 64 MiB file both
report 131,072 blocks. Both are telling the truth: those blocks are allocated to the file. But
the fact that they are the same blocks appears in no field of the stat structure, so du
cannot subtract it either. du knows how to add; it does not know how to share.
df gives the right answer, because it looks at the whole volume — but only at the whole
volume. On a volume that holds clones, "how much space does this directory use" has no single
correct answer. At the end of September I
asked the same disk how much free space it had and got five different answers;
that article was about free space being undefined, this one is about used space being
undefined. One counts the empty side, the other the full one, and neither can be settled by
looking at a single file.
To be fair, I was already using clones. A month ago, while measuring git's status --ignored
behaviour, I
built the lab with cp -c -R —
copying node_modules in four seconds and almost no disk space. I knew the flag. It just
never became a habit.
The Silent Fallback in cp -c
The clone has a trap as well, found the same way: a separate 200 MiB APFS image, attached,
with a 64 MiB file copied to it using cp -c. The command returned successfully: exit code 0, no warning at all.
The file was there. But it was not a clone — it could not be, because the destination is a
different volume.
The behaviour is documented: the cp manual page says that if source and target are on
different file systems, or the target does not support cloning, cp falls back to
copyfile(2). Apple's published file_cmds source shows the same thing: with -c it calls
clonefileat(), and if the returned error is EXDEV or ENOTSUP the code continues into the
normal copy path without printing anything; any other errno produces a warning and a failure.
So -c is a request, not a guarantee. Use it knowingly and nothing goes wrong; put it in a
script and relax because "copies are free now", and the day you move the destination to another
volume the script starts paying full price with the same silence. Nobody will tell you that
you are paying.
I Wrote the Missing Tool
So how do I tell whether two files already on disk share blocks? The answer sits in
fcntl(2). F_LOG2PHYS_EXT translates a logical offset in a file into a
physical offset on the device. If the same logical position maps to the same physical offset in
two files, that block is shared.
static long long phys_at(int fd, long long off, long long len) {
struct log2phys l2p;
memset(&l2p, 0, sizeof(l2p));
l2p.l2p_contigbytes = len; /* IN: bytes to be queried */
l2p.l2p_devoffset = off; /* IN: bytes into the file */
if (fcntl(fd, F_LOG2PHYS_EXT, &l2p) < 0) return -1;
return (long long)l2p.l2p_devoffset; /* OUT: bytes into the device */
}
Two programs grew around that call. The first is a forty-line comparator: it walks two files
in 4 KiB steps, end to end, counting how many blocks land on the same physical address — the
controlled runs used that one. The second is for inventory work: walking thousands of files
end to end is expensive, so it samples a signature at eight points and buckets files by
physical identity. I did what I always do with a measuring tool: I asked it the answer I
already knew first. In the clone produced by cp -c, 16,384 of 16,384 blocks were shared
(100%); in the real copy made by cp -R, zero. ditto zero, ditto --clone one hundred
percent. The tool confirms the df delta along a completely independent path: one reads a free-space
counter, the other reads device addresses, and they say the same thing.
What I Found on My Own Disk
Then the tool went at real data. ~/IdeaProjects is 158.2 GiB; among the 861 files of 8 MiB
and over there are 78 same-size clusters, 36 of which are byte-identical in content. Within each
group I bucketed the files by physical signature; files in the same bucket are clones, separate
buckets are separately written copies.
- Surplus files: 248
- Separately written (paid for) surplus: 232 files / 74.03 GiB
- Surplus sitting free because it is cloned: 16 files / 2.17 GiB
Read honestly, the surplus is not spread across the disk; it is piled in two places. The two
most expensive groups are 450.9 MiB qcow2 images left over from zee lab runs, one with 79
members and one with 78. Those two alone account for 68.26 GiB — 89.6% of everything found.
A single image, with identical bytes, sitting in 79 separate places. The script that copied those files for every branch of an A/B scenario is my script;
when I wrote it I knew clones existed, and I typed cp.
74.03 GiB is 17.4% of the 426.33 GiB in use. It is 5.9× my free space. That number is not
"garbage I can delete" — maybe not every one of those files is needed, but that is not the
point. The point is that even the needed ones did not have to take up this much room.
Of the twelve most expensive pairs, eleven share zero blocks and one shares all of them.
That one is a 14.9 MiB System.Private.CoreLib.dll from a .NET build. Which tool cloned it
is a mystery, and nobody asked for it — so clones do appear on this disk on their own, and
nothing ever told me.
Worktrees Pay Full Price
I also measured the habit that produces duplicates most reliably. When agent sessions work in
this repository they do not touch the main working tree; they open a separate worktree based on
origin/main. A good habit — and one whose cost had never been measured.
Creating one worktree of this repository on the isolated volume took 258,392 KiB: 252
MiB. Cloning the resulting tree with cp -Rc took 2,884 KiB. Same files, same content,
90× apart. That the clone is not zero this time is informative: the tree holds thousands of
files and each wants its own record — half a kilobyte per file, the same order as the 330
bytes from the 2,000-clone run.
The reason is simple: git checkout unpacks objects and writes files from scratch; as far as
this measurement goes it never takes a cloning path. Copying the tree afterwards does.
At the end of September the accumulating worktrees filled the disk and pushed the cleanup into
a scheduled job. Right now 26 worktrees sit under eight repositories and du puts them at 5.56
GiB together. How much of that is real is no longer a mystery: node_modules went into this
article's worktree via cp -Rc — 614 MiB, 4.6 seconds, almost nothing off the disk. du
counts those 614 MiB dutifully. A clone cannot replace a worktree, since git needs its own
bookkeeping; but the price of duplicating the tree itself is now a known number.
Reclaiming It: the Recipe and Its Limits
Paid-for duplication can be reclaimed: delete the file and put a clone of the first one in its
place. Since the content is byte-identical, so is the result. Rather than trying the recipe on
real data, the test ran on the isolated volume — ten identical 50 MiB files:
# nine separate copies took 460,800 KiB from the disk
for i in $(seq 1 9); do cp a0.bin a$i.bin; done
# -p keeps mode and timestamps; the hash check proves the swap
for i in $(seq 1 9); do
rm -f "a$i.bin" && cp -cp a0.bin "a$i.bin"
[ "$(shasum -a 256 <a0.bin)" = "$(shasum -a 256 <a$i.bin)" ] || echo "MISMATCH: a$i.bin"
done
Space returned: 460,800 KiB — every byte that was paid. du goes on saying 512,000 KiB:
just as it never reported the loss, it does not report the gain. Only df sees it.
Two warnings. The recipe is not atomic: between rm and cp the file does not exist, and
a crash in that window takes the data with it — so use it only where another copy exists, and
do not skip -p or the hash check. Second, local snapshots: with Time Machine enabled, freed
blocks stay held in a snapshot for a while and df may show no gain at all. That case is
unmeasured here, because this machine has no Time Machine destination configured and
tmutil listlocalsnapshots returns empty for both volumes; on yours, the gain may have to
wait for the snapshots to age out.
The rules I wrote for myself:
- If I am multiplying something large and immutable within the same volume, use
cp -c(orditto --clone). Lab branches, image copies and "just in case" backups land exactly here. - If I am crossing a volume boundary there is no clone; since
-cwill silently pay full price, the command does not get assumed free. When it matters, it gets measured:dfbefore and after, or clone one file and compare its physical blocks. - For directory trees I use
cp -R -c: the manual page explicitly discourages applyingclonefileto directories and points atcopyfile(3)instead. - If a
dunumber is going into a report, I add a note saying the volume may contain clones. Otherwise I will have counted the same byte twice — which is precisely what I was doing. -
tar,zip, backups and moving data between volumes flatten clones. Space won by cloning comes back wherever you unpack the archive.
What I Could Not Measure
I could not produce a clone inventory for the whole disk: comparing every pair among 3.7 million
files is not feasible, so I narrowed it down to same size plus same sha256 and looked there.
On the first pass I skipped everything below 8 MiB and was ready to wave it off as "probably not
negligible"; then I realised that was measurable too. The 1–8 MiB band holds 1,975 files, 226
groups share identical content, the nominal surplus is 1.75 GiB and 1.30 GiB of it has been
paid for. A small number next to the large files — but in this band a quarter of the surplus
(168 of 669) is cloned, against 6.5% (16 of 248) among the big files. The tools that multiply small
files use cloning better than I do. Anything under 1 MiB is still outside the measurement.
That second scan taught me one more thing: on its first run it counted my own lab. I had cloned
node_modules into the worktree, remember — 614 MiB, thousands of files — and the scan quietly
filed them under "cloned duplicates". I excluded the lab directories and ran it again. A measuring
tool changing what it measures is a classic, and it catches you every single time.
The physical-signature method is not safe for every file either: in sparse or compressed files,
holes can map to the same address and produce false positives. That is why I sample the
signature at eight points along the file and only use the method on files whose content I have
already confirmed identical with a hash. Finally, how much of that 16.15 KiB write cost
is data and how much is bookkeeping remains open: the number is measured, the explanation is not.
How Many Files Does a Byte Belong To?
What I learned today is not a flag but a fact about accounting. On a file system with clones,
"how much space does this file take" cannot be answered by looking at one file; a byte can
belong to more than one file, and nobody tells you which of them paid the bill. du has no word
for this concept and stat has no field for it.
Living with a 98%-full disk, I spent years asking the wrong question: "what can I delete?" The
right question, or at least the one that should stand beside it, was: "how many times have I
written the same thing?" The answer is 74 GiB. And I wrote the scripts that wrote those 74
GiB — every one of them types cp, one letter short.
Top comments (0)