DEV Community

Cover image for Opening a File Doesn't Download It, Reading Does
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

Opening a File Doesn't Download It, Reading Does

#macos #icloud #depolama #olcum

Five days ago I wrote about my cloud quota running out. 1,958 items were stuck in the
queue, and that
article's thesis was this: the files taking up disk space are precisely the ones that
can't get to the cloud. Files whose local copy survives because the upload failed.

This week I asked the same question from the other direction. What happened to the ones
that did go up? How much room do the files that escaped the queue take on my disk?

The answer: none. My home directory holds 236,583 files that ls reports a size
for, totalling 61.84 GiB, and the space all of them occupy on disk is zero bytes.

The more uncomfortable part: those 61.84 GiB don't fit in this machine's free space.
I have 28.85 GiB available. If I asked for every file I own at once, more than half the
request — 53.3% of the bytes — could not be served.

Size is a promise, a block is a fact

macOS calls these files "dataless". The name, size, dates and permissions live on disk;
the contents don't. In Finder you see a cloud icon, but from the shell there is no icon,
just numbers. Two different numbers:

$ cd ~/Library/Mobile\ Documents/com~apple~CloudDocs
$ ls -lO _bootstrap.scss
-rw-------@ 1 mustafaerbay  staff  compressed,dataless 1340 Jun 20  2024 _bootstrap.scss
$ stat -f 'size=%z  blocks=%b' _bootstrap.scss
size=1340  blocks=0
Enter fullscreen mode Exit fullscreen mode

A file 1,340 bytes long that occupies zero blocks. The third column of ls -lO names
the flag too: dataless. This is the file system's official way of telling me "here is
this file's size, but don't ask where it is."

Apple explains the situation to users in two sentences: the files don't take up space on
your Mac, and "you can download the original files when you need them." I've just
confirmed the first half — genuinely zero blocks. The second half, "when you need them",
is what this article is actually about: which moment counts as needing, which operation
triggers the download, and who decides. Apple's support page doesn't go there. I did.

When I asked five different tools how much room was left on my disk and found a 13.5
GiB gap, this flag
wasn't in my accounting. It should have been — because one end of that gap is here.

I didn't go looking for a tool to count them, because no tool is needed. lstat(2)
returns a file's flags, and the SF_DATALESS bit is among them. So I wrote a scan that
walks my whole home directory and opens nothing:

import os
SF_DATALESS = 0x40000000
for dp, dn, fn in os.walk(os.path.expanduser('~'), followlinks=False):
    for name in fn:
        st = os.lstat(os.path.join(dp, name))
        if st.st_flags & SF_DATALESS:
            ...   # count it, add it up, but DON'T READ it
Enter fullscreen mode Exit fullscreen mode

The scan took 4 minutes 45 seconds, looked at 2,042,325 files and hit not a single
lstat error. The result:

Directory Dataless files Nominal size
~/Library/CloudStorage 71,368 57.23 GiB
~/Documents 164,204 4.39 GiB
~/Desktop 803 222.23 MiB
~/Library/Mobile Documents 208 0.50 MiB
Total 236,583 61.84 GiB

That's 11.58% of the files scanned. The space all four rows take on disk, added
together, is zero bytes. I scanned directories in a separate pass with the same method
(lstat only, this time over os.walk's directory list): there are no dataless
directories at all, 0. On this machine only file contents get emptied out; the
directory structure is entirely local.

The interesting part of the distribution: the most common extension is .smali (81,770
files) — decompiled Android sources left over from an old tablet rescue. Most of the
file count sits under ~/Documents, but 92.5% of the bytes are in
~/Library/CloudStorage: large virtual disk images and video files. Two providers, two
different characters. An inflation of small files on one side, heavy files on the other.

That there are two providers matters, because this is not an iCloud feature. Microsoft
says so plainly in its own documentation: "The system logic to decide what files count
against used disk space and what files do not is provided by the File Provider platform,
not OneDrive or Microsoft." The same page reaches for Apple's term, the word "dataless",
when describing file state. The flag belongs to the operating system and so does the
mechanism; the providers are only tenants.

I didn't set this flag, and I can't clear it

Most file flags are mine to play with. I can set nodump with chflags, lock a file
with uchg, or hide it from Finder if I'm in that kind of mood. SF_DATALESS is not
like that. The definition in the kernel source says it in one line:

#define SF_SYNTHETIC    0xc0000000      /* mask of system read-only synthetic flags */
...
#define SF_DATALESS     0x40000000     /* file is dataless object */
Enter fullscreen mode Exit fullscreen mode

The comment above it is blunter: "Synthetic flags. These are read-only. We keep them
out of SF_SUPPORTED so that attempts to set them will fail." And sure enough, the flag
falls outside the mask of superuser-changeable flags (SF_SETTABLE, 0x3fff0000).
0x40000000 isn't inside that mask, which means not even root can set this bit by hand.
I didn't test that part; I derived it from the mask. What I did test is the ordinary-user
side.

And the ordinary-user side surprised me. I tried setting SF_DATALESS on a local file:

SF_DATALESS  chflags OK  -> flags=0x00000000  (bit still set: False)
SF_ARCHIVED  chflags ERROR -> errno=1 Operation not permitted
Enter fullscreen mode Exit fullscreen mode

chflags returned success and silently swallowed the bit. Whereas a flag that
merely requires privilege — SF_ARCHIVED — failed honestly with EPERM in the same
spot. The unsupported flag fails more quietly than the supported-but-forbidden one. A
script that checks the return value and calls it done would be wrong here; I was wrong
for a moment too, until I read the flags back.

And that measurement corrects the comment I just quoted. The source says "attempts to
set them will fail"; on macOS 26.6 the attempt does not fail, it quietly does
nothing. Those aren't the same thing: one returns -1, the other returns 0. Because
the bit sits outside the mask, the kernel drops it and doesn't complain. What the
comment means — the flag can't be written — is true; the word "fail" is misleading in
the sense a shell script would understand it.

Which operation downloads?

This was the real question. What happens if I run grep -r across those 236,583 files?
If I list their names? If I back them up?

I picked nine fresh dataless files — a different file for each operation, because the
first thing to touch a file localises it and the second measurement is no longer the
same experiment. Before and after each operation I read the flag and the block count:

Operation Downloaded? Duration
lstat(2) no 0.0 ms
ls -l no 6.4 ms
stat -f %z no 3.7 ms
du -k no 3.1 ms
open(2), without reading no 0.2 ms
head -c 1 yes 1,243 ms
grep -c yes 1,177 ms
md5 yes 1,206 ms
file yes 3,869 ms ⚠

The top five work on metadata and never touch the file. du included — which makes
sense, since du counts blocks and there are no blocks here to count. (The flagged
3,869 ms was the session's very first measurement and carries the service's warm-up
cost; trust the median below, not that row.)

The bottom four read, and reading downloads. But look where open(2) sits: in the top
group. Opening the file, even read-only, pulls down nothing.

Since the article's title rests on that row, I didn't leave it at one measurement. I
opened nine fresh dataless files read-only and closed them without reading a byte:

  406 B  open  0.026 ms  dataless True->True  blocks 0->0
  ...
  450 B  open  0.018 ms  dataless True->True  blocks 0->0

open(2) RESULT: 9/9 files STAYED dataless, median 0.020 ms (min 0.016, max 0.026)
Enter fullscreen mode Exit fullscreen mode

Nine out of nine stayed dataless, and the open took a median of 0.020 milliseconds —
nothing that asks the network for anything can be that fast. The thing that downloads is
the first read. Microsoft's documentation says the file "won't download to your
device until you open it"; in everyday terms that's true, because opening a file in an
application involves reading it. At the syscall level, though, the line isn't at open,
it's at read.

I can't read why the line falls exactly there out of the kernel source, and saying so
is part of being honest about this: vfs_materialize_file(), which calls the resolver,
is open in xnu, but the code deciding which filesystem operation triggers it lives
inside APFS, and APFS isn't open source. I can measure the boundary; I can't read it.

So how long does the download take? The table's figures are single measurements; for a
number free of warm-up cost I ran a clean series across nine files, reading a single
byte each time:

  • median first read: 1,166 ms (fastest 1,092, slowest 1,441)
  • median second read of the same file: 0.208 ms

About 5,600×. A file's first read costs five thousand times what its second read
does. (That's the ratio of two medians, not the median of the per-file ratios.)

And the "single byte" doesn't stay a single byte. From a dataless file of 3,693,410
bytes I asked only for the first one:

nominal=3693410 B  before blocks=0  after blocks=7216 (=3694592 B)  dataless True->False  2063 ms for 1 byte
Enter fullscreen mode Exit fullscreen mode

I asked for one byte and 3.52 MiB arrived. No partial download happened here: the file
is either present or it isn't.

It would be wrong to pin that on the mechanism as a whole, though, because the kernel
side is open to range requests. The signature of the function that calls the resolver
looks like this:

vfs_materialize_file(
    struct vnode *vp,
    uint64_t op,
    int64_t offset,
    int64_t size)
Enter fullscreen mode Exit fullscreen mode

offset and size are right there — "fetch this range of this file" is an expressible
thing. The whole file arriving is not a kernel limit, it's a choice made by the provider
at the end of this path. What the measurement shows is that a one-byte request ended
with the whole file local; which link in the chain declined to use the range, I can't
see from here.

A practical consequence follows, and I don't like it: a grep -r hunting for content in
my home directory would try to pull down 61.84 GiB. With 28.85 GiB free, it would fill
the disk short of halfway. Microsoft notes on that same page that Spotlight doesn't do
this: it won't make dataless files locally available while indexing. Sensible. The
question is how Spotlight manages not to.

Who decides: the kernel's answer

In the kernel source the mechanism is called the "namespace resolver". A comment block
lays out how it works: when I/O is performed on a dataless object, the file system makes
an upcall to filecoordinationd, the calling thread goes into an interruptible sleep,
and when the work is done the service wakes it by writing to the vfs.nspace.complete
node. Only a process registered as the resolver may write to that node, and the source
notes: "The system filecoordinationd runs as uid == 0." On my machine, there it is:

$ ps -Ao pid,user,comm | grep -E 'filecoordinationd|fileproviderd|bird'
  715 mustafaerbay     /System/Library/PrivateFrameworks/FileProvider.framework/Support/fileproviderd
  732 root             /usr/sbin/filecoordinationd
Enter fullscreen mode Exit fullscreen mode

The resolver runs as root, the File Provider side as my user. (iCloud's own bird
process isn't in this list; it comes and goes on demand — it showed up in the first
round of measurements, not this one.)

What struck me came a few hundred lines further down the same file:

/*
 * The default behavior is to not materialize dataless files;
 * return to the caller that deadlock was detected.
 */
return EDEADLK;
Enter fullscreen mode Exit fullscreen mode

The default behaviour is not to download. Even though every read in my shell
downloads, the system's documented default is "don't fetch it, return an error." The
getiopolicy_np(3) man page says the same thing: when the default policy is selected at
process scope, the system default applies, and that is
IOPOL_MATERIALIZE_DATALESS_FILES_OFF.

So why does my shell download? Because the policy is per-process and inherited. The man
page puts it in one sentence: "New processes inherit the policy of their parent
process." The numbers:

this process (PROCESS): (2, 'ON(2)')
this thread  (THREAD):  (0, 'DEFAULT(0)')
child process PROCESS:  2
Enter fullscreen mode Exit fullscreen mode

My shell is ON. My shell's child is ON too. But if the system default is OFF, where did
this ON come from? I took the same reading in a process started directly by launchd:

pid=97432 ppid=1 comm=/sbin/launchd
sysctl prevent_materialization=1
getiopolicy PROCESS = 1
Enter fullscreen mode Exit fullscreen mode

OFF. Exactly as documented. So on the same machine, running as the same user, one of two
processes can read the file and the other cannot — and what makes the difference isn't
the file, it's whose child you are in the process tree. My shell runs underneath a
desktop application; I inherited the ON from somewhere in that chain. From here I can't
see which link set it, and I'm not going to invent one.

When I turned the policy off in my own process and tried the same read, I got the
kernel's answer first-hand:

setiopolicy_np(OFF) -> 0 | now: (1, 'OFF(1)')
READ ERROR: errno=11 (EDEADLK) -> Resource deadlock avoided   0.0 ms
file: before dataless=True blocks=0 | after dataless=True blocks=0
Enter fullscreen mode Exit fullscreen mode

Zero milliseconds, "Resource deadlock avoided", and an untouched file.

That also answers the Spotlight question: it needs to do nothing special to avoid
downloading — leaving the policy at its default is enough. I couldn't confirm this by
measurement. getiopolicy_np reports only the calling process's policy; there is no
interface for asking about another process (the same is true of the sysctl node). So
this is an inference: Microsoft's sentence and the kernel's default together imply the
indexer stays at the default, but I can't put the question to mdworker and get an
answer.

To my mind that error message is the worst part of the whole mechanism. It's technically
defensible — the kernel is refusing to wait on a service that would have to answer it,
which is a deadlock guard. But for someone staring at "resource deadlock avoided" in a
script's log, the last explanation that will come to mind is that the file is sitting in
somebody else's data centre.

Diagram

Everything so far was one provider

Every download measurement above was taken on iCloud Drive files. But 92.5% of my
dataless bytes aren't there — they're with the second provider, under
~/Library/CloudStorage, 57.23 GiB of them. Saying "the mechanism works like this"
without putting the question to that side too would be treating the half I didn't
measure like the half I did.

So I asked. Five files, all dataless, between 1 KB and 100 KB, one byte from each:

    1184 B -> errno=60 Operation timed out    719.6 ms  dataless=True blocks=0
    1231 B -> errno=60 Operation timed out    653.4 ms  dataless=True blocks=0
    1763 B -> errno=60 Operation timed out    679.2 ms  dataless=True blocks=0
    1950 B -> errno=60 Operation timed out    708.1 ms  dataless=True blocks=0
  101240 B -> errno=60 Operation timed out   2308.0 ms  dataless=True blocks=0
Enter fullscreen mode Exit fullscreen mode

Five out of five ETIMEDOUT. Not one arrived. The provider's File Provider extension
was running on the machine at the time — it's right there in the process list — so
this isn't a case to wave away with "the app was closed." I did not diagnose why it
times out: the session may have expired, something may be wrong on the account side, it
may be something else. I'm not writing it down, because I don't know.

Even undiagnosed, this stands: 57.23 GiB that Finder shows with sizes cannot be
fetched on this machine right now. Not slow — not arriving. And the brctl you'll see
in the next section is no help on this side; that's iCloud's tool.

So the "median 1,166 ms", the "asked for one byte, got the whole file" and the "5,600×"
in this article are all statements about iCloud Drive. For the other provider, the only
measurement I have is five timeouts out of five attempts.

I got it wrong twice while measuring

The article would look tidier without this section. I'm keeping it, because both traps
are ones somebody else will fall into.

The first. To prove the policy difference I thought I had set up an elegant
experiment: run the same reader script by hand and through launchd. The one run by
hand downloaded; the launchd one failed with EPERM. Just as I was about to call that
proof, something nagged at me — the error I expected was EDEADLK, and EPERM came
back. So I gave the same script one more target: a file in the same folder that is fully
local and not dataless.

dataless iCloud file  -> errno=1 EPERM (Operation not permitted)   46.1 ms
LOCAL iCloud file     -> errno=1 EPERM (Operation not permitted)    0.0 ms
scratchpad local file -> READ OK                                    0.0 ms
Enter fullscreen mode Exit fullscreen mode

The launchd-started process can't read the local file either. So the obstacle isn't the
dataless policy, it's the privacy layer: a process with no identity has no access to user
folders. The experiment said nothing whatsoever about materialisation. The policy
difference is still real — getiopolicy_np reports it directly — but that read attempt
is not the evidence for it. Without drawing the distinction I would have reached a correct
conclusion on false evidence. A number that supports your case deserves to be questioned
twice; the convenient number is the least questioned one.

The second. vfs.nspace.prevent_materialization is a sysctl node. Say sysctl and
people assume a system-wide setting; the name reads that way too. It isn't. I read it from
three places and got three answers: 0 in my shell, 1 in the process where I had turned
the policy off, 1 in the launchd process. The node reports the state of whoever asks.
A monitoring script that reports this as a machine's "setting" will produce different
truths depending on which process it runs from.

Counting is easy, putting it back needs a tool

As soon as the measurements were done I evicted the files I'd downloaded: brctl evict,
11 out of 11 successful. Cleanup complete — or so I thought.

While finishing the article I counted again, since this piece's whole thesis is "don't
assume, count blocks." Good thing I did. My cleanup script only held the list of the 11
files from the timing series; the run that produced the operation table was a different
script, and it never wrote its 4 downloaded files to that list. On top of that, the
target file from the EDEADLK test had been evicted once and then re-downloaded on a later
round. Re-scanning the ~/Library/Mobile Documents tree said so plainly: dataless was
203, not 208.

I evicted the missing five and counted once more:

FINAL STATE Mobile Documents: total 223 dataless 208  (first scan: 208)
Enter fullscreen mode Exit fullscreen mode

Sixteen files downloaded in total: 11 in the timing series, 4 in the operation table, and
one of them twice across two rounds. All evicted, and the count is back to the 208 it
started at. Nothing came down at the second provider — nothing could — so there was
nothing to clean up there.

The difference between saying "I cleaned up" and having cleaned up was exactly the
difference between reading a brctl evict exit status and counting the tree again.

Doing that walked me into a practical gap in the mechanism. brctl is iCloud Drive's
tool only: its man page lists download <path> and evict <path>, and both do their
job. But 92.5% of my dataless bytes aren't in iCloud, they're with another provider
running over File Provider. What's available for that one? fileproviderctl's command
list offers dump, diagnose, evaluate, check | repair and obfuscate — no
per-file download or eviction. A deliberate eviction from the command line is documented
only on the iCloud side; on the other side you fall back to Finder's context menu or the
provider's own interface.

Querying state, though, is provider-independent. On 9 November I wrote that there was
no command to ask about a single file's status, then found what I was after in brctl's
own help text. This time the answer is cleaner, and a little humbling: no command is
needed. The flag is on the file itself. And lstat tells the same truth for both
providers, because neither of them sets that flag — the kernel does.

Turning the policy off myself

If the kernel hands you a door, use it. I wrote a ten-line wrapper that calls
setiopolicy_np and then execs whatever command I give it:

#!/usr/bin/env python3
"""nofetch.py <command> [arg...] — runs it with materialization off"""
import ctypes, os, sys
IOPOL_TYPE_VFS_MATERIALIZE_DATALESS_FILES = 3
IOPOL_SCOPE_PROCESS = 0
IOPOL_MATERIALIZE_DATALESS_FILES_OFF = 1
libc = ctypes.CDLL(None, use_errno=True)
if libc.setiopolicy_np(IOPOL_TYPE_VFS_MATERIALIZE_DATALESS_FILES,
                       IOPOL_SCOPE_PROCESS,
                       IOPOL_MATERIALIZE_DATALESS_FILES_OFF) != 0:
    sys.exit('setiopolicy_np failed: %d' % ctypes.get_errno())
os.execvp(sys.argv[1], sys.argv[1:])
Enter fullscreen mode Exit fullscreen mode

Because the policy is inherited, it still holds after the exec. I pointed grep at
three dataless files:

$ ./nofetch.py grep -l color <three-dataless-files>
grep: .../_bootstrap-dark.scss: Resource deadlock avoided
grep: .../_bootstrap-extended-dark.scss: Resource deadlock avoided
grep: .../_bootstrap-extended.scss: Resource deadlock avoided
$ echo $?
2
Enter fullscreen mode Exit fullscreen mode

All three stayed dataless, block count zero. This was the thing I actually wanted to
know: grep doesn't skip the file quietly. It writes a line to stderr for each one and
exits 2. A scan run through this wrapper tells you which files it couldn't touch.
No silent omission — which matters more than blocking the download does.

I'm not offering this as a general backup recipe; I tried it with grep on my own
machine, and I haven't measured how tar or anything else reacts to that error.

What I'm going to do

My list for myself is short:

  • Stop counting the second provider's 57.23 GiB as a backup. Five attempts, five timeouts. If I can't read a file I can't back it up either, and Finder showing me a size promises me nothing.
  • Put content searches through the wrapper. A bare grep -r in my home directory is a download command as far as I'm concerned now. Run through nofetch, the worst case is that it counts the files it couldn't touch.
  • Ask "what's on disk" in blocks. ls tells you the size; st_blocks tells you whether it occupies anything. On this machine the gap between the two is 61.84 GiB.
  • Pin what matters. If a file matters enough that I want it to open when I have no connection, it's too important to leave to the cloud's idea of freeing up space.

This is the exact inverse of the accounting error in the piece about
APFS clones making me pay twice for the same bytes.
There, one copy of a byte was counted in two places; here, bytes that don't exist are
counted in one. Both lie to the same question in different directions: how much room
does this file take?

One closing note, in cloud storage's favour rather than against it: the iCloud side
genuinely works well. I can wander around a five-year archive as if I were carrying it on
a 460 GiB disk, and most days I don't feel the difference. What bothers me isn't the
mechanism but its invisibility: the file system shows me not what I have, but what I
think I have.

A file being "here" is no longer a binary state. Its name is here, its size is here, its
date is here; its contents depend on a policy. I didn't write that policy, I mostly
haven't seen it, and until today I had never once checked it. There is 61.84 GiB of
promise on my disk — and today I learned that 57.23 GiB of it doesn't answer when asked.

Official Sources

Top comments (0)