DEV Community: Mustafa ERBAY

The 204,800 Everyone Copies Costs 295 MiB of Kernel Memory

Mustafa ERBAY — Sun, 25 Oct 2026 14:30:41 +0000

When a file watcher dies with "No space left on device", the internet's answer is one line: set fs.inotify.max_user_watches to 204800. Syncthing's own documentation recommends that number. You paste the line, the watcher starts, the case closes.

What nobody mentions is that the number is a memory budget. The cost per watch, on the bench: around 1.5 KiB on ext4, 2.45 KiB on a container's own layer. Actually installing 204,800 watches therefore means 295 MiB of kernel memory — and drop_caches will not give it back. Raising the limit is free; what is free is the permission, not the bill.

Error messages that lie are well-trodden ground on this blog: df reported zero while root kept writing, and a single fork error message hid four separate ceilings. Both inotify limits belong to that family, so I will pass through them quickly and get to the real question: if you use what you were permitted, what do you pay?

The rig

Measurements come from Docker Desktop's Linux virtual machine on my own Mac: kernel 6.10.14-linuxkit, 14 vCPUs, MemTotal: 5036592 kB (4.8 GiB). Because the experiments write to host sysctls and call drop_caches, no customer server and not this blog's own server took part.

The containers ran like this — budget tests in an ordinary container, memory tests --privileged, since they write to drop_caches:

# budget test: ordinary container
docker run --rm -v "$LAB":/lab:ro python:3.12-slim python3 /lab/watchbomb.py watches 20000 /tmp/w
# memory test: privileged, on a real ext4 volume
docker run --rm --privileged -v inolab:/ext4lab -v "$LAB":/lab:ro python:3.12-slim \
  python3 -u /lab/bellek_ext4.py 150000

There is no library on the watcher side; ctypes calls libc directly, because I want to see which errno stopped me without asking any wrapper:

libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
fd = libc.inotify_init1(0)
wd = libc.inotify_add_watch(fd, path.encode(), IN_ATTRIB)
if wd < 0:
    err = ctypes.get_errno()      # raw errno, no interpretation

Every watch went on a separate inode. Add a second watch to the same inode and the kernel allocates nothing new, it updates the existing record (inotify_update_watch() tries inotify_update_existing_watch() first and falls through to inotify_new_watch() on -ENOENT). What drains the budget is not the number of files but the number of distinct inodes — for a tool that watches directories, that means the directory count.

The lab scripts print Turkish keys; each output block below is glossed in the line that follows it.

Two limits, two borrowed errnos

For the budget tests I lowered the limit to the classic field value, 8192. As root in a container, watching 20,000 files:

eklenen_watch=8172
durma_errno=28 (No space left on device)

(eklenen_watch = watches added; durma_errno = the errno that stopped it.) At that moment the container's disk — the VM's thin-provisioned virtual disk, so not the Mac's real free space, but what df sees:

overlay        1007G   21G  936G   3% /

The second limit is less well known: how many inotify instances (descriptors) you may open. This ran under a separate uid (1002) so the bucket would be clean:

ulimit_n=1048576
acilan_instance=8192
durma_errno=24 (Too many open files)

(acilan_instance = instances opened.) EMFILE, "too many open files" — yet the process holds 8,192 descriptors and the descriptor limit is 1,048,576. The reflex here is to raise ulimit -n; nothing changes, because the wall is fs.inotify.max_user_instances. (That 1,048,576 happening to equal the ceiling of the watch limit is a coincidence; they are separate resources.) The source is explicit about both:

if (!inc_inotify_watches(group->inotify_data.ucounts)) {
    ret = -ENOSPC;
...
group->inotify_data.ucounts = inc_ucount(current_user_ns(),
                                         current_euid(),
                                         UCOUNT_INOTIFY_INSTANCES);
if (!group->inotify_data.ucounts) {
    return ERR_PTR(-EMFILE);

The kernel's own default for the instance limit is 128; my VM reports 8192, so somebody raised it.

The bucket is not on the "real user", it is on the euid

The man page uses the same phrase for both limits: per real user ID. The source says something else — the second argument in the call above is current_euid(), the effective user ID. The watch side uses the same bucket, because inc_inotify_watches() counts against the ucounts record frozen at inotify_init time.

That gap between documentation and code does not stay on paper. The classic privilege-dropping service pattern — start as root, call inotify_init(), then setuid() down — charges its watches to the old euid's bucket, and the user you dropped to is never blamed. For the same reason, the advice to "run the watcher under its own uid" only works if the uid change happens before the inotify descriptor is opened.

In my own runs the real and effective uids were identical, so the bench does not separate the two; this claim rests on the source, not on a measurement.

What the per-user bucket means in practice, though, did get measured. While the first watcher held its 8,172 watches, a completely separate container started next to it — separate PID, mount and network namespaces, nothing running inside:

eklenen_watch=0
durma_errno=28 (No space left on device)

Zero. A spotless container, unable to watch a single file; the only thing they share is uid 0. The pattern itself is not new here — when a neighbouring container shrank my pipes eightfold the same thing happened to pipe buffers, and there no error came back at all.

A user namespace does not rescue you either: after filling uid 1000's bucket, becoming root in a fresh namespace via unshare -U -r still returned zero. inc_ucount() walks up the chain and hits the parent's ceiling; the child namespace's own ceiling is INT_MAX, so the refusal comes from the parent, not the child. One caveat: unshare -r maps the inner 0 to the same outer uid, which is why no fresh bucket appears. A rootless setup that maps into a genuinely different range via /etc/subuid (say podman --userns=auto) lands on a different kuid and does get a fresh bucket.

Who spent the budget before me

That 8,172 is the interesting part, because the limit was 8192. Counting the missing 20 across the whole VM, grouped by uid:

uid=0 pid=84337 comm=python3          izleme=8172
uid=0 pid=362   comm=dockerd          izleme=1
uid=0 pid=361   comm=rpc.idmapd       izleme=1
uid=0 pid=24277 comm=containerd-shim  izleme=2
uid=0 pid=30789 comm=containerd-shim  izleme=2
uid=0 pid=797   comm=containerd-shim  izleme=2
uid=0 pid=800   comm=containerd-shim  izleme=2
uid=0 pid=803   comm=containerd-shim  izleme=2
uid=0 pid=804   comm=containerd-shim  izleme=2
uid=0 pid=83354 comm=containerd-shim  izleme=2
uid=0 pid=84318 comm=containerd-shim  izleme=2
uid=0 pid=96409 comm=containerd-shim  izleme=2
TOPLAM_IZLEME=8192

(izleme = watches held by that descriptor; TOPLAM_IZLEME = total watches.) Nine containerd-shim processes at two each, dockerd and rpc.idmapd at one each: 20. Add my 8,172 and the total is exactly 8192. The container infrastructure eats from the same bucket. At a limit of 8192, twenty watches are noise; a capacity plan that budgets 650 watches for a 650-directory tree fails by a hair.

Here is the counting script, with two caveats:

tot=0
for f in /proc/[0-9]*/fdinfo/*; do
  [ -r "$f" ] || continue
  n=$(awk '/^inotify /{c++} END{print c+0}' "$f" 2>/dev/null)
  [ -z "$n" ] && n=0            # the process may vanish mid-loop
  [ "$n" -gt 0 ] || continue
  pid=$(echo "$f" | cut -d/ -f3)
  uid=$(awk '/^Uid:/{print $2}' "/proc/$pid/status" 2>/dev/null)
  [ -z "$uid" ] && continue
  echo "uid=$uid pid=$pid comm=$(cat /proc/$pid/comm 2>/dev/null) watches=$n"
  tot=$((tot+n))
done
echo "TOTAL_WATCHES=$tot"

First: because the budget is per user, the lines have to be summed per uid; a raw total mixes every user on the machine. Second: after a fork the same fsnotify_group shows up under two descriptors — the kernel counts it once, this loop counts it twice. On a machine running containers the loop belongs on the host; from inside a container you cannot see the neighbours you share the bucket with.

What one watch costs

This was the real question. First the watch record itself: create the files, stat them all (so inodes and dentries are already cached), read Slab, then add the watches and read again. For 150,000 watches the increase was 15.8 MiB, or 110.3 bytes per watch — and almost all of it in SUnreclaim, the class that cannot be reclaimed under memory pressure. Noise stayed at 20 kB. The kernel's own object size for that record agrees:

$ cat /sys/kernel/slab/inotify_inode_mark/object_size
80

An 80-byte object against 110 bytes on the bench; the gap is the connector structure allocated per watched inode, plus slab overhead.

In real life, though, the inodes of the files you watch are not sitting ready. The second rig creates the files, flushes the cache with drop_caches, takes a cold baseline, and only then adds the watches. On a genuine ext4 volume, 150,000 watches, three runs:

Run	Slab increase	Per watch
1	244.5 MiB	1,709.5 bytes
2	215.1 MiB	1,503.8 bytes
3	216.1 MiB	1,510.8 bytes (median)

Runs two and three land within 0.5% of each other; the first is the outlier. The number to work with is ~1.5 KiB per watch. On the container's own writable layer (overlay) the same bench gave 2,450.6 and 2,458.4 bytes across two runs — a watch there is roughly 1.6× more expensive.

Where does the difference come from? A watch holds on to the inode it watches. The kernel's own comment says so: "An inotify watch requires allocating an inotify_inode_mark structure as well as pinning the watched inode." The object sizes on this machine complete the arithmetic: ext4_inode_cache 1,168 bytes, dentry 192, inotify_inode_mark 80.

So is that memory truly held, or would pressure reclaim it, as its place in KReclaimable suggests? Calling drop_caches with the watches still in hand, across three runs:

Run	Increase	`drop_caches` returned	On closing the fd
1	250,412 kB	48,076 kB (19.2%)	all returned
2	220,288 kB	0 (Slab grew by 3,972 kB)	all returned
3	221,304 kB	28,564 kB (12.9%)	all returned

At best the kernel handed back a fifth; in one run it handed back nothing. The remainder stayed for as long as the watcher lived, and in all three runs the whole lot was freed the instant the descriptor closed. The optimism in the KReclaimable column does not survive contact with a live watch.

One caveat: 1.5 KiB is the upper bound, for a watcher that never touches the files. In a tool that opens and reads what it watches — a compiler's watch mode, a log shipper — those inodes would be resident anyway, and the marginal cost of a watch approaches 110 bytes.

What to do about the limit

Now the multiplication from the opening. At the median unit cost, 204,800 watches come to 295.1 MiB (333.9 MiB with the outlier run; roughly 480 MiB on a container layer). Filling the 1,048,576 my VM reports would be 1.48 GiB — 30.7% of a 4.8 GiB machine.

That 1,048,576 is a story of its own. The kernel computes it at boot from RAM: 1% of memory, divided by INOTIFY_WATCH_COST, clamped to [8192, 1048576]. The cost constant in the source is one record plus two inodes, which on this machine's object sizes is 80 + 2×624 = 1,328 bytes. Since 1% of 4.8 GiB is 51,572,736 bytes, the number the kernel would arrive at is 38,834. The reported value is twenty-seven times that, so it did not come from the arithmetic — it was written. I checked /etc/sysctl.conf, /etc/sysctl.d/ and /lib/sysctl.d/ in the VM (/run, /usr/lib and /usr/local/lib have no such directory in this image) and found no inotify line. The author stays unidentified — though a privileged container overwriting a host sysctl has happened to me on this very machine before: when I went looking for who changed fs.file-max, the trail led into a container.

My decision framework:

Count before you raise. The loop above gives current consumption broken down by uid. A watcher that dies at 8192 rarely needs 204,800; the number it needs is usually your tree's directory count plus the few dozen the infrastructure eats.
Do the multiplication. Target watches × ~1.5 KiB (≈2.45 KiB on a container layer) = kernel memory that will not be reclaimed while the watcher lives. Decide there whether that is acceptable against the machine's RAM.
You cannot give a container its own budget. fs.inotify.* is not namespaced: /proc/sys is read-only inside the container, and docker run --sysctl fs.inotify.max_user_watches=... refuses outright ("sysctl ... is not allowed"). The only addresses are the node's own sysctl, or a different uid.
A separate uid, but before the descriptor. Because the bucket is charged to the euid at inotify_init time, ordering matters in privilege-dropping services.
If the tree is large, change the instrument. inotify is the wrong one for hundreds of thousands of directories; fanotify watches an entire mount point with a single mark, at the price of CAP_SYS_ADMIN.

What I did not prove

The measurements come from a single kernel (6.10.14-linuxkit) on a single machine. Since the size of struct inode varies with build options, both the 1.5 KiB figure and the 1,328-byte cost constant belong to this machine and should be re-read elsewhere via /sys/kernel/slab/.../object_size. The 1.48 GiB figure was extrapolated linearly from 150,000 watches; a million were never actually installed. The 110-byte figure rests on a single run. The max_queued_events limit and the IN_Q_OVERFLOW event that arrives on overflow went untested. The euid claim rests on the source; no experiment here separates real from effective uid. And one more: my first "ext4" bench was silently running on overlay because a bind mount had failed, and I was about to publish the claim that both filesystems cost the same — repeating it with a named volume is what exposed the difference.

Conclusion

inotify's limits are not badly documented; what is bad is mistaking them for a capacity measurement. The number in max_user_watches is not how many watches your machine can carry, it is a permission somebody granted — and filling that permission has a price measured in gigabytes, one the kernel will not hand back under pressure.

The habit worth taking from this: before raising a sysctl, measure its unit cost and multiply by your target. It is two lines of work, and more often than not the answer comes back "don't raise it, lower it".

Official Sources

fs/notify/inotify/inotify_user.c (Linux, the commit behind tag v6.10) — where ENOSPC/EMFILE are returned, current_euid(), INOTIFY_WATCH_COST, and how the defaults are computed
kernel/ucount.c (Linux, the commit behind tag v6.10) — inc_ucount() walking the namespace chain, and its rollback loop
Syncthing FAQ — inotify limits — the 8192 limit in the field and the 204800 recommendation

My Runs Took 19 Hours, My Measurement Counted 58,530

Mustafa ERBAY — Sun, 25 Oct 2026 11:31:16 +0000

Two weeks ago I audited the claims in this blog's agent instructions one by one
against the code. The one that mattered most in that audit was deploy duration:
the docs said "about 12 minutes" in five separate places, and measuring twenty-four
runs gave a median of 3.3 minutes. That was one workflow. This morning the question
was wider: how much machine time do all of the repository's workflows consume?

Measuring looked easy. The GitHub Actions API gives timestamps for every run;
subtract created_at from updated_at and you have the duration. One line of
Python, one coffee. I pulled the last thousand runs, summed the differences, and the
screen told me: 58,530 hours. Six and a half years. Across fifteen days.

The number was so absurd that it saved me. I wish every bad measurement were wrong
that loudly; most of them look plausible, settle into a dashboard and quietly bend
decisions. The uncomfortable part wasn't that the measurement was wrong — it was that
I turned out to be the culprit.

A thousand runs — but why a thousand?

I didn't choose the window; my query did. While pulling the runs I had set a
created=>=2026-09-25 filter, and when the API returned nothing after the tenth page
I assumed that was all of it. It wasn't. The repository's run list reports a
total_count of 4,240, and when I removed the filter and paged again, all
forty-three pages came back full — every one of the 4,240 runs, reaching back to
14 September.

The ceiling is real, just not where I thought it was: GitHub documents that this
endpoint returns at most 1,000 results for searches that use filters such as
actor, branch, event, status or creation date. An unfiltered listing has no such
limit. So what cut my dataset down to 1,000 runs wasn't the repository's history. It
was my query.

I kept the measurement on those thousand runs anyway — 10 October 08:45 UTC through
25 October 08:09 UTC, fifteen days. But I'm now writing that down as a choice rather
than a finding. The rest of this article is about a bigger version of the same
mistake: the shape of the query decides the shape of the answer.

The second surprise came from inside the dataset. I also pulled the job records for
each of those thousand runs, because I wanted the real duration from there. Only
801 of the 1,000 yield a real duration; the remaining 199 consist entirely of
skipped jobs, and curiously all of them belong to a single workflow (I'll come back
to that). A skipped job has a completed_at, and its started_at doesn't mark a
real beginning, so a naive "end minus start" produces negative durations on those
rows. My first pass produced exactly that. This is where I always lose the most time
when measuring: the data contains rows that have nothing to do with the question, and
averaging them without filtering makes you look foolish.

So now I have two durations for 801 runs: the naive one from the API field, and
the real one from the jobs' own timestamps.

Measurement	Total	Median
`updated_at` − `created_at` (naive)	58,530 hours	87.4 hours
Job timestamps (real)	19.2 hours	73 seconds

Three thousand and fifty times off. In the median, the gap between 87 hours and 73
seconds is the difference between "it ran for four days" and "it ran for a minute and
a half."

If the error depends on age, the calendar is the suspect

My first instinct was to blame the API. Then I sorted the rows by date and the
pattern stared back at me: for today's and yesterday's runs, the naive duration is
correct. Across the 147 intact runs the median difference between naive and real
is 4 seconds — the method works. The error lives only in rows older than three
days, and there it has no exceptions: 654 of the 801 runs (81.6%) show a naive
duration more than an hour above reality.

An age-dependent error doesn't tell you the arithmetic is broken. It tells you the
data was modified after the fact. If a run's duration grew days after the run
finished, somebody touched the record and walked away.

The age at corruption came out of the same data: of the 654 broken rows, the fifth
percentile is 3.30 days, the median is 3.75 days, and the highest is 4.23 days.
Every record rots about three and a half to four days after birth. That was oddly
close to a number I had written myself.

The culprit's signature: 05:43

In the repository's .github/workflows/ directory sits a workflow called
cleanup-actions-storage.yml. I wrote it. Its purpose is clear: delete old run logs
every day so the Actions storage quota never fills. Its default keep_days is 3.

I tied it down with two measurements. First: on the morning of 25 October the cleanup
run started at 05:41:56 and finished at 05:45:02. The new timestamps on the records
that rotted that morning are spread across four separate seconds, from 05:43:05 to
05:43:08 — the middle of the three minutes the janitor was awake.

The second is more decisive. I checked whether the logs of the broken rows still
exist, probing the log endpoint for twenty runs spread across the window:

Run age	Log HTTP	Naive duration	State
0–3 days (4 runs)	200	0.00–0.35 hours	intact
3+ days (16 runs)	404	78–101 hours	broken

Twenty runs, twenty hits. Not one record with surviving logs has an inflated
duration; not one record with deleted logs has a correct one. The script's artifact
and cache deletion steps aren't filtered by age — whatever they delete, they delete
every night — so they can't produce a signature that lands on a three-day threshold.
That leaves the one step that does look at age: log deletion.

There's also this: the cleanup job's own run records meet the same fate. The record
of the janitor that ran on the morning of 21 October was overwritten at 05:43:08 on
25 October — by the colleague that came four days later. I watched an automation eat
its own tail and, I'll admit, I rather enjoyed it.

My intent wasn't careless; I even wrote the comment at the top of the workflow
myself: run records (success/failure status) are preserved, only the detailed log
files are deleted. The sentence is true. The run record is still there, its status is
there, its commit is there. The one thing that never crossed my mind was that the
record's timestamp is data too. The log deletion call pulls updated_at to the
current hour, and the record looks like it was breathing for four days.

One field does survive, and the data shows it: in all 654 broken rows
run_started_at still matches created_at to the second. The janitor doesn't touch
it.

Every sweep touches exactly one day

A small detail, but it sits at the heart of the mistake I repeat most often in my
career: keep_days says 3 and the measurement says 3.75 days. Two reasons.

First, the cleanup script builds its cutoff at day resolution. The line is right
there in the 25 October run's own log: Cutoff date: 2026-10-22, with the query
created=<2026-10-22. That sweep therefore excludes all of 22 October and touches
only runs created on the 21st. The data says the same: all 37 records overwritten
that morning were created on 21 October, aged between 3.30 and 4.17 days. A job that
ran just after midnight sees four days; one that ran in the late afternoon sees three
and a half. A single calendar day becomes an age band a day wide.

Second, and sneakier: the cron time doesn't hold. The workflow says 0 1 * * *,
i.e. 01:00 UTC. Across the last thirty scheduled runs the median delay is 16
minutes and the worst is 286 minutes. The distribution splits in two: runs up
to 13 October started 9–16 minutes late, and every run after 14 October was 259–286
minutes late. That looks less like load jitter than a regime change; GitHub's docs
say the schedule event can be delayed under load, but I didn't measure the cause of
this step and I don't know it. The practical effect: my "runs at one in the morning"
cleanup actually runs around 05:30, pushing the age band a day and a half forward.

In fairness: 2 of the 654 broken rows don't fit this pattern. Both were updated
75–79 minutes after their run, on the evening of 22 October, and both are CodeQL
analysis runs. It's plausible that code-scanning results touched the record late, but
I couldn't verify it. The arithmetic works for 652 of them and not for two; I'm
writing that down rather than deleting it, because a table that hides the limits of
its measurement lies to the reader.

Hunting for the right field: defined and undefined

So which field should I trust? I downloaded GitHub's own OpenAPI description and
looked at the workflow-run schema. The schema has 36 fields, and 15 of them
carry no description at all — including created_at and updated_at. Both
fields my metric rests on are typed but undefined: "string, date-time", with nothing
said about which moment they capture.

One field does carry a definition, and it's precisely the one I needed:
run_started_at is documented as "The start time of the latest run. Resets on
re-run." One sentence, with a warning inside it — it resets on a re-run. My window
contains no re-runs (every run_attempt is 1), so I can use it; in another
repository I couldn't use it so comfortably.

That leaves two ways to get the real duration:

The /timing endpoint — it returns run_duration_ms per run. On my sample run it reported 224,000 ms, while the value I computed from job timestamps was 204 seconds. The two sources disagree by 10%, and both say "three and a half minutes." I don't know which is more accurate, and closing that gap isn't this article's job.
The /jobs endpoint — the jobs' own start and finish timestamps. It survives log deletion, which makes it the one duration source the janitor doesn't touch.

There's an irony here. /timing carries this warning in GitHub's documentation:
"This endpoint is in the process of closing down." The endpoint that gives the right
number is closing; the field that feeds the wrong arithmetic stays put. And the
warning lives in prose only: the machine-readable deprecated flag in the schema is
not set. If you scan your endpoints programmatically for deprecation, you won't see
this one — I wouldn't have either, had I not read the page with my eyes.

My decision: build the duration metric on /jobs, and read updated_at only as
"when was this record last touched." The field's name was saying that all along. I
was the one asking it a different question.

The real numbers

Once the measurement was fixed, the table turned boring — in the good sense. In
fifteen days, 801 measurable runs, 19.2 hours of work in total. Median 73
seconds, mean 86 seconds, longest run 23.4 minutes.

Workflow	Total runs	Measurable	Real median
Workflow Failure Alert	327	128	7 s
Social Media Post	184	184	76 s
Daily publish pace check	127	127	7 s
Code Quality: Push on main	91	91	84 s
Push on main	91	91	197 s
Veo Prompt (Daily Short)	49	49	68 s
Deploy Only (Code, No Content)	44	44	172 s
FAQ Backfill (TR + EN)	30	30	70 s

The first row explains the 199 runs I mentioned earlier: every run made up
entirely of skipped jobs belongs to Workflow Failure Alert. That sentry ran 327 times
in the window, and in 199 of them no job ran at all — it looked at whether something
had broken, said no, and closed.

Two weeks ago I counted
what my watchman workflows give and what they cost:
two sentries that produce nothing were eating three out of every five runs. That
article asked "how many times did it run?", and the answer is the Total runs
column: 327 and 127. This table puts duration next to them: 7 seconds. Their bill is
heavy in run count and almost nothing in machine time. Reading the same data with two
different questions produces two different decisions — and both are correct.

Two wrong numbers, two different causes

Back to where I started. For the Deploy Only workflow, the median across 44 runs is
2.9 minutes, the 90th percentile is 5.1 minutes (by linear interpolation), and
the worst is 23.4 minutes. Two weeks ago the same workflow showed a median of 3.3
minutes over twenty-four runs; the two measurements agree. The number in the
docs is still ~12 min.

In that earlier article I also found where the 12 came from: it lives in two code
comments, left over from a period when the build genuinely took 8–12 minutes. It was
correct the day it was written. The build got faster; the number stayed.

That's the interesting pair. The number that went wrong by never being updated is
four times reality; the number I found by measuring is three thousand times reality.
The bad measurement produced a worse outcome than the stale guess — because the guess
carries an "approximately" next to it and binds nobody, whereas nobody would have
questioned the measurement had I put it in a table. Attach a method to a number and
the right of objection closes. That, I think, is the most expensive side effect of
measurement culture: we grant trust to the costume rather than to the figure.

Five questions I now ask before choosing a field

I wrote myself a short list after this run. Before standing up a new metric:

Does the field have a definition? If I'm deriving meaning from a field with no description in the schema, I may have invented that meaning. (15 of 36 fields were undefined.)
Who else modifies this record? Cleanup jobs, retention sweeps, re-runs. My own automations belong on that list too.
Is the error age-dependent? Run the same query separately against today's rows, yesterday's, and rows from a week back. Inconsistency means the data changes after the fact.
Is my query truncating the data? The silent ceiling on a filtered search had cut my 4,240 runs down to 1,000, and it took me half a day to notice.
How long will the correct source live? If I'm wiring a metric to an endpoint that is closing down, I need to note that today — otherwise in six months I'm left with a dashboard that quietly returns nothing.

The second item gave me the most trouble, because it required writing my own name in
there. The same thing happened when I wrote about
five TODOs that had been sitting for months:
the hardest component in my inventory keeps turning out to be me.

So what do I do about the janitor?

Closing the article on "lesson learned" would be easy, but it leaves the reader
empty-handed. I have three options, and I'm weighing them against the data.

The first is to stop deleting logs altogether and hand retention to the repository's
own setting. GitHub lets you set log and artifact retention between 1 and 90 days in
repository settings; that turns my job into changing a number rather than maintaining
a workflow. The open end: I don't know whether the platform's own sweep overwrites
updated_at — I didn't measure it, and I won't write what I haven't measured.

The second is to keep the cleanup and make the duration durable instead: write each
run's real duration into my own ledger when it finishes. Then an overwritten stamp
stops mattering.

The third is to leave the script alone and move the metric to /jobs — which is what
I did today. I also noted something on the side: every night the script re-sends the
DELETE call for hundreds of runs whose logs are already gone (its log says "100 runs
to clean" on all five pages). The wasted API calls cost close to nothing, but that
behaviour is exactly what keeps overwriting the stamps night after night.

Measuring is not an innocent act

For a long time I thought of measuring as passive work: the system runs, I watch from
the side, I write the number down. This run showed me the opposite. In my own
infrastructure, the ground my measurement stood on was being erased by another
automation of mine. I wrote both, both were doing their jobs correctly, and the place
where they intersect had no owner.

If nobody re-derives a metric, it rots slowly. And you only notice the rot on the day
it hands you an absurd result — I was lucky enough to get a result as absurd as six
and a half years. A milder error, say twice reality, would have sat on a dashboard
for months and produced a meeting about how "the deploy is getting slower."

What I did today isn't even two lines of code: leave one field for another, question
the filter on a query, note the side effect of a cleanup job. But the real gain was a
single sentence — I have to count my own automations as part of the system I measure,
not outside it.

Official Sources

REST API endpoints for workflow runs — GitHub Docs: the fields of the run object, the 1,000-result limit on filtered searches, the closing-down notice on the /timing endpoint, and the definition of the log-deletion call.
Events that trigger workflows — GitHub Docs: that the schedule event can be delayed under load, and that cron runs in UTC by default.
github/rest-api-description: the OpenAPI description of GitHub's REST API; the field descriptions in the workflow-run schema (and the fields that have none) can be verified here.
Managing GitHub Actions settings for a repository — GitHub Docs: retention settings for run logs and artifacts.

My Notification List Has 94 Rows, 25 Have No App Path

Mustafa ERBAY — Sat, 24 Oct 2026 14:13:46 +0000

Turning a notification off in the settings pane is easy. What I actually
wondered was something else: when I turn it off, does the thing leave the list,
or does it just go quiet? The question looks innocent, but the answer isn't in
the pane, because the pane shows you the switch, not the ledger.

So I opened the ledger. On my laptop running macOS 26.6.2, notification
permissions live in ~/Library/Preferences/com.apple.ncprefs.plist. I found 94
rows in it. My first reaction on seeing that number was "so I've granted
permission to 94 apps." Wrong reaction. Twenty-five of those rows don't even
carry an app path, and three point to a file that no longer exists on disk.

The real finding isn't in the numbers. It's that this ledger is an
append-only one. Everything that asks for permission leaves a row behind;
nothing ever removes one.

Counting the file is not counting the pane

First I asked the same question two different ways, because on macOS
preferences aren't written straight to disk — a daemon called cfprefsd sits
in between, and the file can lag behind the in-memory state. If the two
answers disagreed, I needed to know which one to trust.

# (1) via cfprefsd: the state the operating system sees
defaults export com.apple.ncprefs - > /tmp/nc.plist

# (2) the file on disk, directly
python3 -c "import plistlib,os;d=plistlib.load(open(os.path.expanduser(
'~/Library/Preferences/com.apple.ncprefs.plist'),'rb'));print(len(d['apps']))"

Both said 94. Better yet, the sets of bundle-id values from the two sources
matched exactly: nothing extra, nothing missing. So the file on disk isn't
stale; this is the living state.

Then I looked at the file itself, and that's where it got interesting:

stat -f "mtime=%Sm size=%z" -t "%Y-%m-%d %H:%M:%S" \
  ~/Library/Preferences/com.apple.ncprefs.plist
# mtime=2025-09-23 16:22:05 size=30324

This file hasn't been rewritten in thirteen months. Ask file what it is and
it answers "Apple binary property list": thirty kilobytes of binary record,
sitting perfectly still for thirteen months, and still today's effective
setting. Black boxes tend to spend their quietest stretches without telling
anybody.

25 of the 94 rows have no app path

When I split the rows by the prefix on their bundle-id, the picture came out
like this:

Row type	Count	What it means
`_SYSTEM_CENTER_:`	24	The system's own alert sources
`_WEB_CENTER_:`	1	A permission granted to a website
Ordinary `bundle-id`	69	Actual applications

None of the twenty-four _SYSTEM_CENTER_ rows has a path field. Their names
give them away:
com.apple.diskspaced (the disk-is-filling-up warning),
com.apple.SoftwareUpdateNotification, com.apple.BluetoothUIServer,
com.apple.security.keychain-circle-notification,
com.apple.DiskArbitration.DiskArbitrationAgent (the warning you get for
yanking a drive without ejecting it), com.apple.EscrowSecurityAlert. Four of
them are purely Wi‑Fi business: diagnostics, joining a network, personal
hotspot, peer-to-peer.

A distinction is needed here: "no path" and "no code behind it" are not the
same thing. Twenty of these twenty-four rows carry the signature field I'll get
to shortly, and it names a real signed binary. One of them even corresponds to
an app outright: _SYSTEM_CENTER_:com.apple.FaceTime. FaceTime is recorded
twice in the ledger, once as a system alert source with no path, once as an app
with the path /System/Applications/FaceTime.app. The measurement here is the
absence of a path; saying "no app" would claim more than the data shows.

These rows still aren't permissions I granted: there's no app doing the asking,
and you wouldn't expect a keychain alert to petition you. But in the ledger they sit in exactly the same shape as the apps,
with the same fields. This is where reading the row count as "I granted
permission to 94 apps" starts going wrong.

Of the remaining 69 rows, 67 have a path. Two don't: com.apple.iChat and
com.apple.TelephonyUtilities. The first is the Messages app's old identifier, a name abandoned
years ago, still holding its own row in the ledger. With no path, I couldn't
tell at first glance what these two rows belong to, but it can be worked out:
the signature field I'll open shortly names them both. The com.apple.iChat
row points at com.apple.identityservicesd, and the
com.apple.TelephonyUtilities row at com.apple.imagent. Both are system
services, so these two rows belong not to an app but to the plumbing underneath
Messages.

Fourteen rows also carry a universal-app-id field, the one that unifies the
iPhone and Mac versions of the same app under a single identity. So the ledger
tracks not just "which app" but "which app family."

The spread of the paths says something too:

41 rows point under /System/ (the apps that ship with the operating system),
25 rows under /Applications/,
1 row at an updater under /Library/Application Support/.

A notification permission granted to localhost

The single _WEB_CENTER_ row stopped me. It's a notification permission
granted to a website, and the site's address is web.<project>.localhost, a
development server running on my own machine. The row is the record of my
having tried notifications at that address once: the browser asked, permission
was given.

That address doesn't lead anywhere now. The name doesn't resolve, the request
goes unanswered, nothing on the machine is listening:

host web.<project>.localhost
# Host web.<project>.localhost not found: 3(NXDOMAIN)

curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://web.<project>.localhost/
# 000

The server is gone. The permission stayed.

There's an interesting asymmetry here. On the web side, permission is granted
per origin. MDN's page for Notification.requestPermission() states
plainly that permission is requested "for the current origin." The W3C Push API
specification goes further. When permission is revoked, the user agent must
deactivate the affected subscriptions and delete the stored details. So the
web's own model has a revocation mechanism defined.

But when that origin is a localhost address, there's no site owner to revoke
anything and no app to uninstall. The row outlives the thing it points at.

I also found a push section on eight rows: the ones registered for remote
notifications. One of them held a real 32-byte device token. I'm not printing
its value here, but I am printing where it lives: the file you assumed held
permissions also holds tokens.

Three paths that aren't on disk

I checked how many of the 67 paths actually point at a file. Three don't:

python3 - <<'PY'
import plistlib, os
p = os.path.expanduser('~/Library/Preferences/com.apple.ncprefs.plist')
for a in plistlib.load(open(p, 'rb'))['apps']:
    if 'path' in a and not os.path.exists(a['path']):
        print(a['bundle-id'], '->', a['path'])
PY

All three end up in the same place, but they got there by different routes, and
the difference matters.

Minecraft is the simple case: /Applications/Minecraft.app isn't there.
The app is gone from the machine; the row stayed.

Chrome and Opera are more roundabout. Both applications are installed and
working on my machine. But the notification row doesn't point at the app — it
points at a helper program inside the framework, with the version number
baked into the path:

Browser	Version in the row	Versions on disk
Chrome	`140.0.7339.185`	`154.0.8037.98`, `155.0.8059.85`, `155.0.8059.86`
Opera	`120.0.5543.161`	`136.0.6008.80`, `137.0.6036.63`

Chrome's own version today is 155.0.8059.86. So my notification ledger knows
Chrome fifteen major versions back and Opera seventeen. Each update creates a
new version directory and removes the old one; the path in the ledger points
into empty space.

And now the file's frozen timestamp means something: the row was written while
Chrome was at 140, and nobody has touched it since September 2025. The browser
advanced fifteen major versions. The ledger was never written again.

Which raises a question. If the permission still works, and there's no sign
that it broke, then the path in the ledger can't be what makes it work. So
what carries the identity?

The row is bound to a signature, not a path

The answer is in another field. 88 of the 94 rows have a field called src,
holding 89 blobs in total. Every single one of them starts with the same four
bytes: fade0c00.

python3 - <<'PY'
import plistlib, os, collections
p = os.path.expanduser('~/Library/Preferences/com.apple.ncprefs.plist')
m = collections.Counter()
for a in plistlib.load(open(p, 'rb'))['apps']:
    for s in a.get('src', []):
        if 'req' in s:
            m[s['req'][:4].hex()] += 1
print(dict(m))   # {'fade0c00': 89}
PY

I found where that number comes from in Apple's own open-source Security
distribution. In requirement.h there's this line:
class Requirement: public Blob<Requirement, 0xfade0c00>. So these blobs are
code signing requirement blobs.

Searching inside them for readable strings confirmed it: in 40 of the 89 blobs,
the row's own bundle-id appears as plain text. The blob on the Mail row holds
a single string: com.apple.mail.

Chrome confirms it exactly. The identity in the row is
com.google.Chrome.framework.AlertNotificationService; when I open today's
version of the helper and read its identifier, the same string comes back:

defaults read "/Applications/Google Chrome.app/Contents/Frameworks/Google Chrome \
Framework.framework/Versions/155.0.8059.86/Helpers/Google Chrome Helper (Alerts).app/Contents/Info.plist" \
  CFBundleIdentifier
# com.google.Chrome.framework.AlertNotificationService

The path is stale, the identity is live. That is precisely why the row kept
working across fifteen versions.

So the ledger's logic is this: the permission is bound to a signed code
identity, not to a path on disk. The path field is a convenience, a hint,
maybe a way to find the icon for the pane. The thing that carries identity is
the signing requirement.

That's news in two directions. The good direction: because it doesn't work by
path, another program can't take over /Applications/Minecraft.app and inherit
Minecraft's permission; if the signature doesn't match, the row is useless to
it. The bad direction: nothing checks whether the thing a row points at still
exists, because looking at the path was never needed in the first place.
Nobody comes to repair what isn't broken.

I can't read the state in the file

Here's a boundary I need to be honest about. Every row has a flags field,
most have auth, all have content_visibility. What the numeric values of
those fields mean isn't documented by Apple, and I'm not going to guess.

What I can say is that the values don't fit the documented model. Apple's
UNAuthorizationStatus documentation defines exactly five constants for
permission state: notDetermined, denied, authorized, provisional,
ephemeral. The auth field in my file contains these values: 0, 1, 4, 6, 7,
38, 47, 54, 71, 102, 263, 271, 326, 327, 775, 791. Twenty-four rows have no
auth field at all.

Those twenty-four are not the same set as the twenty-four system rows from
earlier. Five come from elsewhere: FaceTime, iChat, TelephonyUtilities,
Freeform and that localhost row. In exchange, five system rows do have an
auth field (the software-update notification and four Wi‑Fi rows). The two
numbers matching here is a coincidence; they aren't counting the same thing.

263 or 791 cannot be a value of a five-case enumeration. The field is probably
a bit set packing the state together with the options, but "probably" is no
basis for a technical claim. The practical upshot: the file is not an audit
surface. I can count the rows and verify the paths; I cannot read the state.

Let me say why I'm drawing that line so firmly. If I set out to guess the bit
meanings, I could look at a number and write "this app is allowed to show
notifications on the lock screen." It would sound competent. And if it were
wrong, nobody would catch it; no reader can read their own auth value and
correct me, because there's no document to check it against. That's exactly why
I'm not writing it: an unverifiable claim looks more convincing than a
verifiable one.

Instead of filling in an undocumented field with guesses, I go back to the
measurable question: what does this row point at, and does that thing exist?
The file answers both plainly.

The content_visibility field reads 0 on 63 rows and 3 on 31. The grouping
field is 0 on all 94. I don't know which pane option those numbers correspond
to either.

The only documented surface is the pane itself. Apple's macOS user guide lists
what's on offer for an app: "Allow notifications," alert style, critical alerts,
time-sensitive alerts, badge, sound, previews, grouping. Eight controls. The one thing missing from that list is the thing I was
looking for: there is no option that removes a row. The guide describes no
such control, because there isn't one.

The other direction: 44 of my 64 installed apps aren't listed

Seeing this many accumulated rows, you might assume the ledger covers "every
app." The opposite is true. I have 64 application bundles under
/Applications; only 20 of them appear in the notification ledger. Forty-four
don't appear at all.

Once it clicks, it's simple: this isn't an inventory of what's installed, it's
a record of requests. Only the things that asked once get in. Forty-four
apps never asked: either they don't send notifications, or they haven't tried
yet.

Which is why reading the list as "what's on my machine" gives you a wrong
answer in both directions: things that aren't there are listed, and most of the
things that are there aren't. The two sets aren't even nested; they just
overlap.

One more detail: of the 67 rows carrying a path, 41 point under /System/, the
apps that ship with the operating system. Only 25 sit under /Applications/,
among the apps I installed. So 41 of the 67 rows carrying a path, more than
three-fifths, aren't my decisions at all; they're the machine's own defaults. Most of what I thought of as "my permission
list" was never mine. That's also the discovery that most quickly drains your
enthusiasm for reviewing it.

Running the same measurement on your own machine

Four commands are enough. None of them changes anything; they all just read.

# 1) How many rows?
defaults export com.apple.ncprefs - | plutil -extract apps raw -

# 2) Split the rows by type
python3 - <<'PY'
import plistlib, os, collections
p = os.path.expanduser('~/Library/Preferences/com.apple.ncprefs.plist')
apps = plistlib.load(open(p, 'rb'))['apps']
t = collections.Counter(
    '_SYSTEM_CENTER_' if a['bundle-id'].startswith('_SYSTEM_CENTER_:')
    else '_WEB_CENTER_' if a['bundle-id'].startswith('_WEB_CENTER_:')
    else 'app' for a in apps)
print(len(apps), dict(t))
PY

# 3) Paths that aren't on disk
python3 - <<'PY'
import plistlib, os
p = os.path.expanduser('~/Library/Preferences/com.apple.ncprefs.plist')
for a in plistlib.load(open(p, 'rb'))['apps']:
    if 'path' in a and not os.path.exists(a['path']):
        print('ORPHAN:', a['bundle-id'], '->', a['path'])
PY

# 4) When was the ledger last written?
stat -f "%Sm" -t "%Y-%m-%d %H:%M" ~/Library/Preferences/com.apple.ncprefs.plist

The fourth command's output tells you the most, I think. If the date is recent,
your machine is still accumulating new permissions. If it's very old, thirteen
months ago like mine, the ledger has frozen, but it's still carrying your old
decisions where it froze.

A warning: this file is not a documented interface, it's an implementation
detail. Its structure can change in any macOS release. Reading it is safe, but
don't write to it by hand; cfprefsd will overwrite from its in-memory
state and you'll be left with an inconsistent setting.

What I'm going to do about it

The honest answer: very little, and I'm choosing that deliberately.

There's nothing to be done about the three unremovable rows: the pane offers no
delete, and touching the file by hand isn't a good idea. The Minecraft row will
stay where it is. It does no harm either: with no app behind it, it can't
produce a notification. With one caveat: since the permission is bound to a
signature, if I ever reinstall a signed Minecraft I won't be asked again. That
row isn't harmless, it's dormant.

Two things are real work for me. The first is that localhost permission: next
time I'm testing notifications on a development server, I'll revoke the
permission from the browser's site settings when the test is done. The
revocation mechanism exists there, the Push API defines it; I just never used
it. The second is that device token: knowing that tokens sit next to permissions
is reason enough for me to think about where this file goes in my backups.

Beyond that, this measurement left me with a habit. When I grant permission I
think "I'll turn this off later" — but turning it off doesn't delete the row. I
ran into the same thing with data left on disk after
deleting apps:
I removed eighteen apps and 661 MiB of data stayed behind. Background jobs look
similar:
launchd has been retrying for years
and I never once watched it.

Same pattern in all three: removal removes the visible part and leaves the
record.

It would be easy to present that as a defect, but I don't think it is one.
Binding the row to a signature is the right design decision, and not checking
the path is the consistent consequence of it. Nobody wrote a garbage collector
that sweeps up notification rows for deleted apps, because that row harms
nobody. From the engineering side there's nothing to object to.

What bothers me isn't the engineering, it's the bookkeeping. When I flip a
switch off in the pane, I think "right, that's settled." It isn't: the decision
is recorded, the record is permanent, and the record itself is never shown to
me. Using a machine for years means carrying an archive of decisions you don't
remember making. Nobody shows you the archive; nobody promised to, either.

The lesson I'm taking: being careful when you grant is cheaper than revoking
later. The revocation mechanism often doesn't exist, and when it does, as with
web permissions, using it is left to you. The ledger itself is thirty
kilobytes, written thirteen months ago, still in force. And it'll stay that way until
something writes it again.

Official Sources

54 of My Apps Carry Intel Code; Two of Them Run It

Mustafa ERBAY — Sat, 24 Oct 2026 11:38:43 +0000

"Is this app Intel?" has a ready answer in macOS: select the app in Finder, hit
Get Info, read the "Kind" line. It will say "Application (Universal)", or
"Application (Intel)", or "Application (Apple silicon)". A good enough answer
for a user. My question was a little different: how many bytes of Intel code
sit inside that bundle? The Get Info window doesn't say.

So I sat down to count. Across the 64 app bundles under /Applications I walked
601,358 files, 6,149 of which turned out to be Mach-O binaries, and I read the
header of every one of them byte by byte. Here is the tally: 54 app bundles
contain x86_64 code, 6.95 GiB of it in total. This machine is an Apple M4
Pro, so it cannot execute Intel code natively. With two exceptions.

I didn't start out to reclaim disk space, but by the end I had something more
interesting than a cleanup plan: the thing we call "the transition is done" is
not done inside the binaries. And the largest share of it isn't where I can
delete it.

Anatomy of a fat file

A universal binary is one file that holds the complete code of more than one
architecture. Apple's transition guide says so plainly, and it asks developers
to turn not only apps but "all of your compiled code" universal: plug-ins,
custom frameworks, dynamic libraries, command-line tools, daemons. That is why
2,286 of the 2,533 fat files on my disk hold both an Intel and an ARM slice.
(In that count ARM means arm64 and arm64e only; two old Xcode archives that
carry arm plus i386 plus x86_64 fall outside it.)

The format is defined in fat.h inside cctools. The file opens with the magic
number 0xcafebabe, followed by a count of slices, then one record per slice
with its architecture code, its offset in the file and its length. One note in
that header matters: these structures are always written big-endian, so if
you read them in your machine's own byte order you get nonsense.

The same header describes a second format: when a slice is larger than 4 MB
or a slice's offset within the file passes 4 MB, the 64-bit layout with
0xcafebabf takes over. My script recognizes both magic numbers. It found not a
single one of the second kind: all 2,533 fat files on this disk are
0xcafebabe.

Why that is, fat.h says in its own words — the 64-bit format is flagged as "a
work in progress and not yet fully supported in all the Apple Developer Tools".
The condition is written down and not applied. The counter-example is in this
article's own diagram:

Those numbers come from Google Chrome's framework file. Of 519,576,720 bytes,
271,164,912 — 52.2 percent — is Intel. The second slice starts at byte
271,187,968, so both the size and the offset are sixty times the threshold, and
the layout is still 32-bit. More than half of Chrome's framework is code that
will never run on this machine.

The inspection command in Apple's documentation is this one:

% lipo -archs /System/Applications/Mail.app/Contents/MacOS/Mail
x86_64 arm64

Running the same command on this machine today gives a slightly different
answer:

% lipo -archs /System/Applications/Mail.app/Contents/MacOS/Mail
x86_64 arm64e

arm64e is the ARM slice with pointer authentication enabled; Apple compiles
its own system binaries that way. The doc prints arm64, the machine says
arm64e. Same mechanism, mismatched label. I can't claim the document was once
right and went stale — no record of mine shows that; all I can say is that today
the example and the machine disagree. Which is a fair warning about every sample
output you copy and paste.

The tally

I added up the slices of all 6,149 Mach-O files by architecture. The slices come
to 23.78 GiB:

Architecture	Uncompressed	Compressed	Total
arm64	12,242.8 MiB	4,750.6 MiB	16,993.4 MiB
x86_64	5,769.6 MiB	1,349.3 MiB	7,118.9 MiB
arm64e	0.1 MiB	175.5 MiB	175.6 MiB
arm64_32	0.0 MiB	62.5 MiB	62.5 MiB
arm	0.0 MiB	2.2 MiB	2.2 MiB
i386	0.0 MiB	0.45 MiB	0.45 MiB

One row is missing from that table: a 0.9 MiB entry whose architecture number my
script did not recognize. It gets its own section below. The arm64_32 row was
in the same state — the script printed cpu33554444 and I named it by hand: the
watchOS architecture, and all 62.5 MiB of it comes from inside Xcode. I own no
watch, but I do own a watch development environment. arm is 32-bit ARM: 2.2 MiB
of an older world, still here.

The Intel share (x86_64 plus i386) is 6.95 GiB, 29.2 percent of all slice bytes.
Here is the denominator too: the whole /Applications folder is 54.29 GiB, so
Intel code is 12.8 percent of my applications. One byte in eight.

Split by main executable, the picture is this: of 64 apps, 24 are ARM-only, 38
carry both ARM and Intel, and 2 are Intel-only. Look at the whole bundle and the
number climbs to 54, because an app with a clean main binary can still ship a fat
helper library inside. Only 10 apps are entirely free of Intel code.

The eight heaviest Intel loads:

App	Intel slices
Google Chrome	965.9 MiB
OneDrive	685.0 MiB
Microsoft Outlook	510.7 MiB
Microsoft Teams	499.7 MiB
Xcode	448.9 MiB
Microsoft Word	441.1 MiB
Microsoft PowerPoint	427.9 MiB
Microsoft Excel	392.4 MiB

Chrome's figure is inflated by one more thing: three separate versions of its
framework sit on disk (154.0.8037.98, 155.0.8059.85, 155.0.8059.86), each around
495 MiB. It is reasonable for a browser update not to delete the old version
immediately — you may need to roll back. Still, three versions times half-Intel
gives you pause.

First correction: what the file says is not what the disk holds

At this point it was tempting to write "6.95 GiB wasted" and move on. I'm glad I
ran one more check: a file's logical size and the blocks actually allocated to it
are different numbers. The logical total of those 6,149 Mach-O files is 23.84
GiB; the allocated blocks add up to 19.93 GiB. (The 56 MiB gap against the slice
total is fat headers plus the alignment padding between slices; fat.h notes
that padding may be stored as holes where the filesystem supports them.) So
where did the missing 3.9 GiB go?

Into filesystem compression. 2,333 files have the UF_COMPRESSED flag set, and
their 6.21 GiB of logical content occupies 2.29 GiB of blocks on disk. So the
accurate sentence is this: 5.63 GiB of the Intel slices live in uncompressed
files, byte for byte on disk; the remaining 1.32 GiB sit inside compressed files
and take up less room there. Applying the compression ratio from above to that
1.32 GiB gives roughly 0.5 GiB — a derived figure, not a counted one, and I'm
flagging it as such.

I've fallen into this trap before: asking "how much free space is there" five
different ways on the same disk produced a
13.5 GiB spread.
When someone quotes me a single number about a machine now, my reflex is to ask
what that number is counting.

Second correction: cafebabe doesn't belong to Mach-O alone

The table also had a row labelled cputype 16777239 — 0.93 MiB, apparently
irrelevant. I chased it anyway, because writing off something your tool doesn't
understand means writing off the other places where the tool is wrong.

The files turned out to be .metallib bundles inside Xcode: GPU shader
libraries. And here is the interesting part — lipo refuses to call them
Mach-O, while file does not.

% xxd -l 16 .../CoreImage.metallib
00000000: cafe babe 0000 0002 0100 0017 0000 0008

% lipo -archs .../CoreImage.metallib
lipo: not a mach-o '.../CoreImage.metallib'

% file .../CoreImage.metallib      # shortened
Mach-O universal binary with 2 architectures: [air64_v24:...] [air64:...]

Same file, same magic number, two tools with two answers. The fat container is
not reserved for CPU code; Apple uses it for GPU code too, and air64 slices
live under the same header. My byte-level script counted a file the tool had
rejected. The error is 0.93 MiB, so the table survives — but it taught me not to
see 0xcafebabe and conclude "Mach-O" on the next pass. That same magic number
is also the signature of Java class files; OpenJDK's class file parser
identifies a file by those exact four bytes. I found no .class file in these
bundles, so that wasn't what polluted my table this time. It could have been.

Apple's own share, and a surprise of scale

Everything so far might read like a choice third-party developers made. So I ran
the same count under /System/Applications, across Apple's own 46 apps. 11,221
files, 201 binaries, 0.63 GiB of slices:

arm64e: 330.2 MiB (51.1 percent)
x86_64: 315.4 MiB (48.9 percent)

45 of the 46 apps carry Intel code. Mail, Notes, Music, Preview, Terminal — all
of them x86_64 arm64e. The heaviest are Books (46.3 MiB of Intel), Freeform
(45.5 MiB) and Music (36.7 MiB). Which also means the 54 in the title is scoped
to /Applications: the number of apps on this machine carrying Intel code is
really 99.

But calling 315 MiB "Apple's share" got the scale badly wrong, because system
apps are thin shells: the code mass lives in the shared cache. So I looked
there:

File set	Logical	Blocks
`dyld_shared_cache_x86_64*` (9 files)	5.35 GiB	2.08 GiB
`dyld_shared_cache_arm64e*` (15 files)	5.42 GiB	1.91 GiB
`aot_shared_cache.0..6`	5.73 GiB	2.17 GiB

That directory shows up under two paths, but stat reports the same device and
inode for both: one copy, two doors. And the root volume is sealed —
diskutil apfs list says Sealed: Yes, and mount reports /dev/disk3s1s1 on / (apfs, sealed, local, read-only, journaled).

Add it up and the picture inverts. Intel code I could delete: 6.95 GiB. Intel
code I cannot: 315.4 MiB plus 5.35 GiB, or 5.66 GiB logical, around 2.4 GiB
in blocks. The two are nearly the same weight. I came into this with a thesis
about third-party developers bloating my disk; the count talked me out of it.
This isn't one developer being careless, it's the platform's distribution model.

Can it be removed? An experiment

Curiosity turned practical: what happens to a bundle's signature if I thin it?
Apple's ditto has a documented flag for exactly this, and the man page puts it
in one line: "Thin Universal binaries to the specified architecture."

I ran the experiment on a small notarized app (Shottr, 6,780 KiB):

% ditto --arch arm64 /Applications/Shottr.app ./Shottr-thin.app
% du -sk ./Shottr-thin.app
3852    ./Shottr-thin.app

% lipo -archs ./Shottr-thin.app/Contents/MacOS/Shottr
arm64

% codesign -v ./Shottr-thin.app ; echo $?
0

% spctl -a -vv ./Shottr-thin.app
./Shottr-thin.app: accepted
source=Notarized Developer ID

The bundle went from 6,780 KiB to 3,852 KiB, 43.2 percent smaller. The signature
verified, Gatekeeper accepted it, notarization held. So each architecture slice
carries its own signature, and dropping one does not invalidate the other.

Let me state the limits of that experiment, because it is a narrow one: I never
launched the thinned copy, I ran codesign without --deep --strict, and I
never tried a large bundle with frameworks of its own inside. What it proves is
that one notarized bundle's signature chain survived thinning — not that this
works everywhere.

Beyond that, I'm not putting it forward as a way to reclaim disk space, and I
haven't kept it in place. The first update brings the universal bundle right
back, so the space you won is rented. What an app's own updater does with a
thinned bundle is a separate question mark — a tool that ships delta updates may
not recognize it. And the thinnable part isn't actually 6.95 GiB: 6,673.1 MiB of
those Intel bytes live in fat files, but 446.2 MiB sit in 533 single-architecture
files, where dropping the code means dropping the app. The two biggest of those
are the two Intel-only apps themselves.

In short, on a 460 GiB disk with 125 GiB free, the prize doesn't earn a weekend.
Counting was instructive; cleaning wouldn't have been.

The date that actually matters

I said "with two exceptions" at the top. Those two — an old build of a media
player (VLC 3.0.23) and a serial-port tool — carry Intel code only, so when they
launch they run inside the Rosetta translation environment.

Let me be honest here: I never watched a running process. My evidence is a
property of the binaries — these two bundles have no ARM slice, so they cannot
run without translation. And "two" isn't a property of the machine, only the
count under the defaults: per Apple's documentation a user can force any
universal app to open under Rosetta from the Get Info window, and a developer can
reorder the preference with the LSArchitecturePriority key. The Intel slice in
those 38 universal apps isn't dead; it's asleep.

Rosetta is installed (/Library/Apple/usr/libexec/oah is in place) and pkgutil
dates the package's install time to 1 September 2026 — an OS update can refresh
that stamp too, so I'm not reading it as "the day I installed it by hand".

How the translation actually works is in Apple's platform security
documentation, and reading it made me correct my first draft. There are two
paths. At runtime, the kernel spots an x86_64 Mach object and hands control to a
Rosetta translation stub rather than to the dynamic linker, and pages are
translated inside the process. On the ahead-of-time path, every Mach object goes
through the AOT pipeline while macOS is being built — but what lands in the trust
cache is not translated code, it is a hash: three code directory hashes per
object (arm64, x86_64, and the AOT translation of x86_64). Apple states outright
that "the actual translated products don't ship with the operating system";
translation is deterministic, so the products are reconstituted on demand, signed
with a device-specific key managed by the Secure Enclave and kept in a vault only
the Rosetta service can reach.

The table above has an aot_shared_cache row: 5.73 GiB logical, 2.17 GiB in
blocks, with file dates matching the OS build. From the outside I can't tell
whether it arrived with the installer or was produced during installation; what I
can tell is that it is on my disk right now.

And this is where the calendar arrives. Apple announced on 1 September 2026:

macOS 27: Final release to support Rosetta — Intel-only apps will no longer
run on Mac computers with Apple silicon after this update.

The support page says the same thing in user language: Rosetta is available on
macOS 27 and earlier; from the next major release it remains only for certain
older, unmaintained games that rely on Intel-based frameworks. Note the timing:
macOS 27 Golden Gate shipped on 14 September 2026 and supports Apple silicon Macs
only. So the last release to carry Rosetta is already out; this machine is on
26.6.2, one major release behind. My two Intel-only apps don't have a distant
expiry date — it is one release away.

What strikes me is that the two numbers pull in opposite directions. The code
under calendar pressure amounts to 2 apps. The code taking up disk space
lives inside 54 apps and weighs 6.95 GiB. The second never runs, so Rosetta's
retirement won't touch it; it will simply stay there. The troublesome part is
small, and the heavy part travels quietly.

A natural question follows, and I don't know the answer: when Rosetta goes, do
the x86_64 slices go too? Will macOS 28 drop the x86_64 shared cache from the
sealed volume, will developers stop building universal, or will this 6.95 GiB be
dragged along for years? All Apple has said so far is that the translation ends;
nothing has been promised about what the binaries contain. The carve-out for
games even suggests some x86_64 plumbing stays in place.

If you want to count your own machine

The Get Info window is a fine start, but it says nothing about the helper
binaries inside a bundle. If you plan to go deeper, mind four things:

Don't stop at the main binary. The file under Contents/MacOS/ may be clean; the weight sits in frameworks, plug-ins and helper processes. In my count the gap was between 38 and 54.
Separate logical size from block count. stat hands you both numbers; if you don't know which one you're reporting, a 16 percent error follows you around.
Don't treat every 0xcafebabe file as Mach-O. Compare lipo -archs against file; when one refuses, look at what the other says.
Don't stop at the applications folder. The real mass is in the shared cache and that volume is sealed; a number without its scope points at the wrong culprit.

Finally, ask this: is anything actually running this code? For me the answer was
two. What sits in the other fifty-four bundles isn't load, it's luggage.

What's left

On the user's side the Apple silicon transition started in 2020 and looks long
finished; apps open, they run fast, nobody asks about architecture. Look inside
the binaries, though, and the transition hasn't ended, it has only gone
invisible. Carrying two architectures in one file is a clever compatibility
answer; the bill for that answer is split across every user's disk, and nobody
itemizes it.

The thing that stayed with me isn't the 6.95 GiB. It's how many times I reached
that number from the wrong place: the first count gave what the file says, the
second what the disk holds, the third what the tool got wrong, and the fourth
showed the real mass was never where I was looking. Someone quoting a single
number about their own machine usually doesn't know which question that number
answers. When I counted
the root certificate store
I ended up in the same place: once you start measuring, the work stops being
about finding the number and becomes about finding out what the number counts.

Official Sources

My Path Check Caught All Six Escapes, Then Let 24,378 Through

Mustafa ERBAY — Sat, 24 Oct 2026 06:14:27 +0000

Taking a filename from a user and opening it inside your own directory is one of the oldest jobs in software. Archive extractors do it, file servers do it, even template engines do it. And nearly everyone writes the same two lines: join the path onto the root, resolve it with realpath, verify the result starts with the root. If it doesn't, reject.

In mid-October I went through the behaviour of openat2 itself in a separate article, and I ran this race there too: the classic pattern that calls fstatat and then openat returned the attacker's file on 30.7% of its successful opens inside a 50 µs window. So I already know "check then open" is broken — that part is not new.

This article asks something else: which check actually gets written in the field, and what does it do? Few people write the fstatat pattern by hand; what everyone writes is realpath plus a prefix comparison. And that check looks spotless in a static test — which is probably why nobody replaces it.

I tried six defenses against nine paths. In the static table realpath came out spotless — it stopped all six escapes. Then I ran the same check while the directory structure moved underneath it: on 24,378 of 200,000 turns it read the file outside the cage. In that same race, RESOLVE_IN_ROOT returned zero.

The setup

Measurements were taken on this blog's own server: Ubuntu 24.04, kernel 6.8.0-147-generic, 18 vCPUs, gcc 13.3.0. All of it as an unprivileged user inside a temporary directory from mktemp -d; nothing persistent on the server was touched.

The cage: kok/ is the area we permit, disari/ is what must not be read.

kok/ok.txt          -> "KAFES-ICI"
kok/sub/ok2.txt     -> "KAFES-ICI2"
kok/ici      -> sub/ok2.txt        # symlink that stays inside (legitimate)
kok/mutlak   -> /etc/hostname      # absolute symlink
kok/goreli   -> ../disari/sir.txt  # relative escape
kok/atla     -> ../disari          # escape to a directory
kok/sihir    -> /proc/self/root    # magic link
disari/sir.txt      -> "DISARIDAKI-SIR"

Three of the nine paths are legitimate — they have to be accepted. I put that distinction in deliberately, because how many escapes a defense stops says nothing on its own; a function that rejects everything also stops all six. The real question is whether it does both jobs.

The legitimate three: ok.txt (plain file), sub/../ok.txt (a .. that turns back inside), ici (a symlink that stays in the cage). The six escapes: ../disari/sir.txt, /etc/hostname, mutlak, goreli, atla/sir.txt, sihir/etc/hostname.

Six defenses

First, the filter everyone writes once:

/* D1: reject if it contains ".." */
if (strstr(path, "..")) { errno = EACCES; return -1; }
return openat(dfd, path, O_RDONLY);

Second, the check serious code writes — the real defendant here:

/* D2: realpath + prefix comparison */
snprintf(joined, sizeof joined, "%s/%s", ROOT, path);
if (!realpath(joined, resolved)) return -1;
size_t rl = strlen(ROOT);
if (strncmp(resolved, ROOT, rl) != 0 ||
    (resolved[rl] != '/' && resolved[rl] != '\0')) {
    errno = EACCES; return -1;
}
return open(resolved, O_RDONLY);

The resolved[rl] test matters: without it, /srv/data-gizli passes a /srv/data prefix. A common bug, but not this article's subject; the version above is the correct one.

Third, the route taken by people who say "I turned symlinks off": openat(dfd, path, O_RDONLY | O_NOFOLLOW). The remaining three use openat2: RESOLVE_BENEATH, RESOLVE_IN_ROOT, and RESOLVE_IN_ROOT | RESOLVE_NO_MAGICLINKS | RESOLVE_NO_XDEV.

The static table

With the directory structure held still, 6 × 9 = 54 opens:

Defense	Escapes blocked	Legitimate opened
D1 `..` filter	1/6	2/3
D2 realpath + prefix	6/6	3/3
D3 `openat` + `O_NOFOLLOW`	2/6	2/3
D4 `RESOLVE_BENEATH`	6/6	3/3
D5 `RESOLVE_IN_ROOT`	6/6	3/3
D6 `IN_ROOT + NOMAGIC + NOXDEV`	6/6	3/3

Since a summary can mislead, here is the raw table too — which error code came from where is about to matter:

Defense	`ok.txt`	`sub/../ok.txt`	`ici`	`../disari/sir.txt`	`/etc/hostname`	`mutlak`	`goreli`	`atla/sir.txt`	`sihir/...`
D1 `..` filter	OPEN	`EACCES`	OPEN	`EACCES`	OPEN	OPEN	OPEN	OPEN	OPEN
D2 realpath + prefix	OPEN	OPEN	OPEN	`EACCES`	`EACCES`	`EACCES`	`EACCES`	`EACCES`	`EACCES`
D3 `O_NOFOLLOW`	OPEN	OPEN	`ELOOP`	OPEN	OPEN	`ELOOP`	`ELOOP`	OPEN	OPEN
D4 `BENEATH`	OPEN	OPEN	OPEN	`EXDEV`	`EXDEV`	`EXDEV`	`EXDEV`	`EXDEV`	`EXDEV`
D5 `IN_ROOT`	OPEN	OPEN	OPEN	`ENOENT`	`ENOENT`	`ENOENT`	`ENOENT`	`ENOENT`	`ENOENT`
D6 `IN_ROOT+NOMAGIC+NOXDEV`	OPEN	OPEN	OPEN	`ENOENT`	`ENOENT`	`ENOENT`	`ENOENT`	`ENOENT`	`ENOENT`

Every bold OPEN is a leak: in that cell the program read the contents of a file outside the cage.

The .. filter loses at both ends. It stops only one of six escapes — because there is no .. inside a symlink, and none in the absolute path /etc/hostname either. Meanwhile it rejects the legitimate sub/../ok.txt with EACCES. So it lets the attacker through and annoys the user. Managing that combination is genuinely hard.

O_NOFOLLOW keeps its promise, but the promise is small: it protects only the final component. mutlak and goreli are symlinks themselves, so they get ELOOP — but atla/sir.txt, where atla is a symlink to a directory, opens cleanly and reads the secret outside. O_NOFOLLOW is not a scoping flag; once again, something the manual never hides, yet it keeps getting used as scoping in real code. On top of that it drops the legitimate ici link with ELOOP.

That the last two rows look alike but return different error codes is no accident either — BENEATH says EXDEV, IN_ROOT says ENOENT. I will come back to that difference shortly, because it is the most misleading part of the table.

And here is the discomfort in the middle of it: D2 produces exactly the same result as the openat2 flags. Six escapes closed, three legitimate paths open. Anyone reading this far would say "so realpath is enough," and they would not be unreasonable. The table says they are right.

Because the table is incomplete.

The real test

All 54 opens above happened in a directory tree where nothing moved. In production, directory trees move. Your archive extractor gets another request into the same directory while writing a file; a backup process relocates something; or none of that happens and the person on the other side moves the directory on purpose.

realpath and open are two separate calls. There is a moment between them.

To measure that window I wrote a second probe and a separate layout:

kok/alan/hedef.txt  -> "KAFES-ICI-HEDEF"
kok/.yem     -> ../disari       # the symlink to be swapped in
disari/hedef.txt    -> "DISARIDAKI-SIR"

The main thread keeps trying to open the same path; a side thread keeps atomically swapping the alan directory with a symlink pointing outside, using renameat2(RENAME_EXCHANGE). The atomic swap is critical here: there is no half-finished state, so what the numbers below show is not an implementation bug but a structural hole in the check itself.

The whole rig is this much — the part you would copy to try it in your own code:

/* side thread: atomically swap the directory with a symlink */
static void *takasci(void *_) {
    while (!dur)
        if (syscall(__NR_renameat2, dfd, "alan",
                    dfd, ".yem", RENAME_EXCHANGE) == 0) takas++;
    return NULL;
}

/* main loop: open the same path with both defenses, read the content */
for (long i = 0; i < N; i++) {
    int f = d2("alan/hedef.txt");   /* realpath + prefix */
    if (f >= 0) { a2++; if (sizdi(f)) s2++; } else h2++;
    f = d5("alan/hedef.txt");       /* RESOLVE_IN_ROOT */
    if (f >= 0) { a5++; if (sizdi(f)) s5++; } else h5++;
}

sizdi() reads from the opened descriptor and checks whether the content starts with DISARIDAKI. So the counter is not counting "the open succeeded" but the wrong file's contents actually reached my hands. My first probe did not draw that distinction and reported the rate higher than it was.

I ran both D2 and D5 in the same run, 200,000 turns:

Defense	Opened	Errors	Leaked outside
D2 realpath + prefix	90,901	109,099	24,378
D5 `RESOLVE_IN_ROOT`	88,565	111,435	0

The swap counter read 314,729 over the same period.

24,378 is 12.2% of all attempts. But that is not the ratio to look at: of the 90,901 files D2 successfully opened, 26.8% were outside the cage. So after the check said "fine, this path is safe," roughly one open in four returned the wrong file.

This is not an estimate or a theoretical window; I read the contents, and they said DISARIDAKI-SIR. 24,378 times.

On the RESOLVE_IN_ROOT side the number is zero. The reason is simple, and that is exactly why it holds: in openat2 the check and the open are not separate steps. The kernel validates the scope while it resolves the path, and never hands control back to you in between. There is no window to miss.

Look at D5's 111,435 errors as well — more than D2's 109,099. There is a cost here and no point hiding it: opens that land on the swap moment return errors, and some of them get EAGAIN from scope ambiguity. I did not break the errors down, so I cannot say how many of those 111,435 were ENOENT and how many EAGAIN. EAGAIN got its own measurement in the October piece: under a synthetic load generating thousands of renames per second it reached one open in five — that figure is not a ceiling, and it is not a rate measured on a real server either. Either way, moving to openat2 is not free security but a trade that requires you to write a bounded retry budget. On the other side of that trade sit 24,378 leaks.

IN_ROOT does not reject, it remaps

The October article's table has a row where opening /etc/hostname with RESOLVE_IN_ROOT returned ENOENT. I recorded the result but never showed why it was ENOENT — and that is exactly where the two flags part ways.

It was ENOENT because there was no directory named etc inside the cage. That was the easy case.

I put etc/hostname in the cage and tried again. The server's real /etc/hostname reads mustafaerbay-vps3; into the fake copy inside the cage I wrote SAHTE-HOSTNAME-KAFESTEN:

Flag	Path	Result
`RESOLVE_BENEATH`	`/etc/hostname`	DENIED `EXDEV`
`RESOLVE_IN_ROOT`	`/etc/hostname`	OPEN → `SAHTE-HOSTNAME-KAFESTEN`
`RESOLVE_BENEATH`	`../../etc/hostname`	DENIED `EXDEV`
`RESOLVE_IN_ROOT`	`../../etc/hostname`	OPEN → `SAHTE-HOSTNAME-KAFESTEN`

The two flags do not do the same job. BENEATH, in the manual's words, does not permit resolution to leave the directory given by dirfd, and treats the attempt as an error. IN_ROOT treats dirfd as the root — so it does not reject the escape, it remaps it inward. Whatever chroot(2) does, but for a single open.

So that October ENOENT did not mean "you cannot leave"; the kernel was saying it looked for kok/etc/hostname and did not find it. The moment you put that file in the cage, the same call returns a file instead of an error.

In practice this means the following. If you are writing an archive extractor, IN_ROOT is exactly what you want: a /etc/passwd entry inside a tar quietly becomes kok/etc/passwd, which is how container images work. But if your case is "the user gives me a path inside my data directory," IN_ROOT is dangerously quiet: when the user asks for /etc/hostname they do not get an error, they get a different file. If an attacker can drop files into the cage — which is precisely what they can do in an archive extractor — they can redirect any path they like to their own content.

My preference: BENEATH for APIs that take a path from the user. I want the error. Not a silently different file.

The cage does not know what it holds

There is also a dimension scoping takes no interest in. I put a FIFO into the cage with mkfifo and opened it with RESOLVE_IN_ROOT:

RESOLVE_IN_ROOT   boru     -> OPEN (type=FIFO)
RESOLVE_IN_ROOT   ok.txt   -> OPEN (type=regular file)

The cage held — the pipe really is inside it. But had I not written O_NONBLOCK, the program would be hanging there waiting for a writer. Scoping flags check where you went, not what you opened. Confining the path to the cage does not mean you opened a regular file.

The kernel has an answer for this, a new one: OPENAT2_REGULAR. The patch written by Dorjoy Chowdhury and taken by Christian Brauner on 21 May 2026 returns EFTYPE when the path is not a regular file. I could not find the flag in v7.1's header; it is there in v7.2 — and today's mainline is v7.3. On my own 6.8 I got EINVAL, as expected: the old kernel does not know the flag and says so rather than ignoring it silently. That distinction and the full list of error codes got their own section in the October piece; I am not repeating it here.

What it costs, and what to do below 5.6

I expected the safe option to be the expensive one. The numbers said the opposite. I opened a four-level path (a/b/c/d/f.txt) 300,000 times, after a warm-up round, in two separate runs:

Method	Run 1	Run 2
`openat` (unscoped)	7,311 ns	7,131 ns
realpath + prefix	32,579 ns	33,244 ns
`RESOLVE_BENEATH`	7,705 ns	7,833 ns
`RESOLVE_IN_ROOT`	7,720 ns	7,307 ns

realpath is roughly 4.5 times more expensive, and the reason is unsurprising: it resolves the path fully once, then makes the kernel resolve the same path a second time. Two full path resolutions plus an lstat per component.

Quoting one number for the scoping flags would be wrong, because the two flags do not sit in the same place. Unscoped openat wanders between 7,131 and 7,311. IN_ROOT wanders between 7,307 and 7,720 — the ranges overlap by four nanoseconds, so I cannot claim a difference there. BENEATH, though, runs 7,705 to 7,833: both runs sit above the unscoped ceiling, roughly 7%. That is a consistent gap, but I have two runs; it is not enough data to write "BENEATH is seven percent slower." What I can say: the cost of the scoping flags is at most a few percent, and in this measurement BENEATH looks slightly more expensive than IN_ROOT.

Either way, the trade was not "security or speed" as I assumed. The leaky defense turned out to be the expensive one by a wide margin.

On portability I have to be straight: openat2 arrived in 5.6, that is March 2020. On kernels below it — or outside Linux — these flags do not exist, and you have to walk the path component by component with O_NOFOLLOW at every step; that is what userspace libraries do. I did not measure that route, so I will not claim it gives equivalent protection.

There is a trap here that is easy to skip: the kernel version is not enough. If you run inside a container, the seccomp profile may not know openat2 — a known situation on older Docker and libseccomp versions — and the call comes back EPERM or ENOSYS. So base your capability detection on a real probe at startup rather than a version number: call it once, fall back to emulation on ENOSYS/EPERM, and make that fallback increment a counter rather than happen silently. That is the only way you notice the machine with a new kernel and an old profile.

You do not have to write the emulation yourself either: os.Root in Go's standard library and filepath-securejoin cover it on the Go side, libpathrs on the C and Rust side. Whichever you pick, if you provide scoping through emulation in your own code, run the race probe above against that emulation too. My realpath check also looked correct when it was written.

The code outside the cage

Finally, something none of this article's measurements cover: the path you open with openat2 is safe, but your program does not only open files.

There is a live example. CVE-2026-41579 (CVSS 3.3, low), closed in runc in June 2026, is exactly this: the advisory says the mount-target lookup "correctly scopes the lookup inside the container." What opens the hole is not the lookup but the os.Remove and os.Symlink calls in setupPtmx and setupDevSymlinks, made on strings built with filepath.Join. If the image's /dev is a symlink, files named ptmx on the host can be deleted — and on the second arm, six fixed-name symlinks (core, fd, ptmx, stdin, stdout, stderr) can be created in an existing host directory. Docker is not affected because it masks /dev with a read-only layer; podman and containerd are. Fixed in 1.3.6, 1.4.3 and 1.5.0-rc.3.

So a scoped open is a component, not a policy. You need to resolve the path once with openat2 and then continue through the descriptor: unlinkat, symlinkat, fstatat all take a dirfd. Every line that joins strings and returns to a path leaves the guarantee you earned right there. The title of the runc fix says as much: make the /dev setup descriptor-based.

In practice that means this transformation:

/* BEFORE: the open is scoped, what follows is not */
int fd = sys_openat2(dfd, rel, &how);      /* safe */
snprintf(p, sizeof p, "%s/%s", kok, rel);
unlink(p);                                  /* NOT safe */

/* AFTER: continue through the descriptor */
int dir = sys_openat2(dfd, dirname_of(rel), &how_dir);  /* O_PATH|O_DIRECTORY */
unlinkat(dir, basename_of(rel), 0);

In the second version there is no second path resolution; unlinkat starts from a descriptor whose scope is already validated. Reduced to one sentence: resolve the path once, then never return to the path.

A decision frame

Do not write a .. filter. In the measurement it stopped one escape of six and rejected a legitimate path. It cannot do either job.
Do not mistake O_NOFOLLOW for scoping. It protects only the final component; everything under a symlink to a directory walks right through.
realpath + prefix is correct on a static tree, not in a race. If you are certain the tree is yours and nobody moves it, it can stay. If you are not certain, the measured rate is 26.8%.
Taking a path from the user? RESOLVE_BENEATH. You want the escape as an error.
Extracting an archive or image? RESOLVE_IN_ROOT. You want the escape remapped inward — but know that this is a deliberate choice.
Do not pass both scoping flags, you will get EINVAL. Close magic links separately with RESOLVE_NO_MAGICLINKS; the manual says the scoping flags do this for now and that it may change.
Write a bounded budget for EAGAIN — three to five attempts — and if it runs out, reject the request and record it in a separate counter. Never retrying produces spurious failures when a neighbour is busy.
Check the file type separately. Before v7.2 with fstat; from v7.2 on with OPENAT2_REGULAR.
Do not return to strings after the open. Use the *at family.
Hold your root descriptor somewhere and think about its lifetime. Opening a root per request burns descriptors; holding one long-lived means the root itself can be renamed or umounted from under you. I prefer keeping the root as a long-lived O_PATH descriptor and checking periodically that it is still the object I expect.
Do not use cost as the excuse. In the measurement the scoping flags differ by at most a few percent; the expensive one is the realpath version (4.5 times).

Conclusion

In October I showed that openat2 closes the window between the check and the open. What this measurement taught me is where that window hides best: realpath plus a prefix comparison gave the right answer on all 54 opens. A static test suite will never go red on it — a single-threaded test assumes the world holds still while it runs. And the defense that looked best in the table turned out to be the most expensive one.

The second lesson is about IN_ROOT, and it struck me as the quieter one: it does not stop the escape, it remaps it inward. For anyone who can drop files into the cage, that means getting the file they want instead of an error. In code that takes a path from the user, what I want is the EXDEV that BENEATH gives me.

Put together, what I am left with is this: a check returning the right answer is not enough — there has to be nobody between the check and the action. What openat2 gives you is not a stricter rule; it is the absence of that gap.

Official Sources

I Documented 29 Keys, Three Systems Lean on One Token

Mustafa ERBAY — Fri, 23 Oct 2026 15:09:06 +0000

This blog's repository contains a file called .env.example. Its job is to tell anyone installing the site from scratch: fill in these keys and the rest works. Next to it sits a setup guide stating that you need, at minimum, four AI keys, both Turnstile values and an admin token. Together they make a promise: if this list is complete, the system stands up.

When the file came up for an unrelated task, one line in it did not immediately ring a bell. The question that stuck was the reverse one: is every key the code reads actually written in this file?

You cannot settle whether a template is incomplete by remembering. You have to count. I counted in three places and got three different numbers.

The count: 29, 97, 14

The first number comes from the document. .env.example defines 29 distinct keys:

grep -E '^[A-Z][A-Z0-9_]*=' .env.example | sed 's/=.*//' | sort -u | wc -l
# 29

The second comes from the code itself. These are the distinct names read through process.env or import.meta.env under src/ and scripts/:

grep -rhoE '(process\.env|import\.meta\.env)\.[A-Z][A-Z0-9_]*' \
  --include='*.ts' --include='*.mjs' --include='*.astro' src scripts \
  | sed -E 's/.*env\.//' | sort -u | wc -l
# 97

The third comes from the live server. The process serving the blog reads /etc/mustafaerbay.env, and 14 keys are set there. I did not look at any value — only the names at the start of each line.

29, 97, 14. Three sources that each claim to describe the configuration of the same system, and no two of them agree.

97 is a floor, not a set

An admission has to come next, because this piece argues "do not remember, count" and the counter itself is faulty. The grep above only sees the form process.env.NAME. If the code reads a variable any other way, it never enters the count.

Two shapes escape it. The first is optional chaining: in src/lib/runtime.ts the Turnstile secret is read as process.env?.TURNSTILE_SECRET, and that question mark keeps it out of the pattern. The second is a name computed at runtime:

envKey = lang === 'tr' ? 'ME_VOICE_ID_TR' : 'ME_VOICE_ID_EN';
const id = process.env[envKey];

scripts/lib/elevenlabs.ts reads three voice IDs this way: ASKER_VOICE_ID, ME_VOICE_ID_TR and ME_VOICE_ID_EN. None of the three is in the template, none has a fallback value, and each throws at runtime if absent. They belong to precisely the category this article cares about most, and none of them made it into the count.

scripts/veo-prompt.ts picks up a provider key by the same dynamic method, but there it does no harm: the three names it reads (GROQ_API_KEY, CEREBRAS_API_KEY, OPENROUTER_API_KEY) are both in the template and read plainly in other files, so they are already inside the count. A dynamic read is not automatically a blind spot; the blind spot appears when a dynamically read name occurs plainly nowhere else.

Widening the pattern to cover the question mark turns 97 into 98. Add the three dynamically read voice IDs and it is at least 101. "At least," because finding those leaks also came down to reading by hand.

Writing this off as a footnote would suit me. It cannot be: a recommendation to "derive the contract from the code" is coming later, and a check step that implements it literally would silently pass the exact class of bug it was built to catch.

What those 73 names actually are

By the narrow count, 73 of the 97 names in the code are missing from .env.example. Leaving the number there would have been easy — write "73 undocumented settings" and close the tab. It would also be dishonest.

Four of them are not mine to write. GITHUB_OUTPUT and GITHUB_STEP_SUMMARY come from the workflow runner, NODE_ENV from the runtime, and ONLY_SLUG is the slug parameter handed to the social posting workflow. They do not belong in the list.

That leaves 69 that someone could plausibly set. Most are harmless tuning knobs: FAQ_SLEEP_MS, AI_RATE_LIMIT_BREATH_MS, MAX_POSTS_PER_RUN and friends, each with a sane default in code, and the system behaves identically if you never touch them. Leaving those out is not a gap but deliberate restraint: put 97 lines in a setup file and nobody reads any of them.

Then I separated the ones shaped like credentials. A pattern matching names that end in API_KEY, TOKEN, SECRET, PASSWORD or SALT returns nine. Three more I added by hand although the pattern misses them — TOKEN_ENC_KEY (ends in _KEY), GEMINI_API_KEYS (plural) and YOUTUBE_CLIENT_ID (ends in _ID). So the real rule is "suffix match plus judgement," and the total is 12:

BLUESKY_APP_PASSWORD · TOKEN_ENC_KEY · VISITOR_HASH_SALT · YOUTUBE_CLIENT_ID · YOUTUBE_CLIENT_SECRET · YOUTUBE_REFRESH_TOKEN · CLOUDFLARE_API_TOKEN · MISTRAL_API_KEY · GEMINI_API_KEYS · BURNCPU_API_TOKEN · GH_API_TOKEN · LINKEDIN_ACCESS_TOKEN

These twelve do not carry equal weight. Some make noise when absent: the path that calls Mistral directly says if (!key) throw new Error('MISTRAL_API_KEY missing'), and the Bluesky endpoint throws without its password. Noise is good — you find out. The dangerous ones continue in silence, and there are exactly two: TOKEN_ENC_KEY and VISITOR_HASH_SALT. When either is missing the code falls through to another value — which one, shortly.

One credential, two names

Line 89 of .env.example asks for this:

BLUESKY_PASSWORD=          # → bsky.app → Settings → App Passwords (UYGULAMA şifresi!
                           #   normal hesap şifreniz DEĞİL)

I wrote that comment, and went out of my way to stress "app password." The social posting workflow uses the same name: social-post.yml passes BLUESKY_PASSWORD from GitHub Secrets on line 125, and scripts/social-post.ts reads exactly that name. So far, consistent.

But the site's own sharing endpoint, /api/share-bluesky, is fed from a different file. Here is what src/lib/bluesky.ts does:

const appPassword = process.env.BLUESKY_APP_PASSWORD;
// … (the line between reads the service address)
if (!handle || !appPassword) {
  throw new Error('BLUESKY_HANDLE ve BLUESKY_APP_PASSWORD env vars eksik');
}

Same credential, two names. Worse: the error raised when it is missing names a variable that appears nowhere in the setup file. Someone following the document to the letter puts BLUESKY_PASSWORD in .env, calls that endpoint, and is told to go find BLUESKY_APP_PASSWORD. It is not in the file, not in the guide — it exists only inside the code.

So I checked which one is actually set on the live server: BLUESKY_APP_PASSWORD is set, BLUESKY_PASSWORD is not there at all. That makes sense, because the process on the server serves the site and its API endpoints, while the posting script runs on the CI side with its own secrets. Two names living in two environments, and both of them work today.

If this arrangement works, it is because I typed two different names into two different places by hand. Nothing anywhere records that the two are the same password. It belongs to the same family as the run where I counted seven claims in my own recipe that described the code incorrectly — document and code each correct alone, wrong together.

Three jobs on one token

The second silent line, TOKEN_ENC_KEY, held me longer. The name says what it is for: a key used to encrypt something. It is read in exactly one place — src/lib/secret-box.ts, the module that encrypts LinkedIn's OAuth tokens with AES-256-GCM before they are written to the SQLite database on disk. (The name appears a few more times in that file, but those are the module's own doc comments and one error string. A count that does not separate comment from code gets this wrong too.)

That module's key derivation looks at TOKEN_ENC_KEY first; failing that it derives a key from ADMIN_API_TOKEN via scrypt; if neither exists it throws and stops. That last step is good behaviour — encryption does not quietly proceed with a weak key.

The problem is the middle step. ADMIN_API_TOKEN is already a documented, mandatory key, and as the name suggests it is the admin panel's login token. Six files in the repository read it, but the work they do comes down to three jobs:

The first job is the token's declared one: admin-auth.ts guards the panel, while share-to-bluesky.ts, share-to-linkedin.ts and social-post.ts are clients carrying that same token to the /api/share-* endpoints. The by-products of panel login derive from it as well — admin-auth.ts uses HMAC over the same token to produce both the session cookie and the Umami embed key.

The other two jobs grew out of a convenience: we already have a secret value, let us not add another one. I am not guessing at that motive — the justification is written in my own comment at the top of secret-box.ts, which says it works with the existing production environment and requires no new secret.

Here is the state on the server: ADMIN_API_TOKEN is set, TOKEN_ENC_KEY is not, VISITOR_HASH_SALT is not. So the convenience is not a theoretical possibility but today's configuration.

What happens if it is rotated

An admin token is the kind of thing you assume is rotatable. If it leaks, you generate a new one, put it in place and log back into the panel. Is that actually true in this setup?

Rather than reason about it, I called the repository's own module from two separate processes — encrypt with one, decrypt with the other. The values are fake; no real token was involved.

1) encrypt with ADMIN_API_TOKEN=A
   v1:QaSUKyZp/exJJgkE:yIEMn9VKKQ94cuDH1yMmaw==:be/w8SroJUurxBSxzh/5KX1IJ5Hv+ZoJmas+wg==
2) decrypt, same token  → PLAIN: linkedin-sahte-erisim-jetonu
3) decrypt with token B → THROW: Error | Unsupported state or unable to authenticate data
4) no token at all      → THROW: secret-box: TOKEN_ENC_KEY veya ADMIN_API_TOKEN env ayarli olmali
                                  (token sifreleme icin)
5) non-envelope value   → PLAIN: duz-metin-eski-jeton

Line three is the answer. The moment the admin token changes, LinkedIn tokens encrypted under a key derived from the old one become undecryptable. It does not return half the plaintext or garbage — it returns nothing: in AES-GCM, if the authentication tag does not verify, the operation fails. Node's documentation covers the missing-tag and tampered-ciphertext cases explicitly: decipher.final() throws, and the ciphertext should be discarded. The wrong-key case does not appear on that page; I take it from RFC 5116's definition plus line three above. The RFC states that authenticated decryption has a single output, either a plaintext value or a special symbol FAIL indicating the inputs are not authentic — and it defines authenticity relative to the key, so a ciphertext not produced under a given key is not authentic for it.

Line five is the module's other face. A value that is not in envelope form comes back out of decryptSecret unchanged. That is deliberate: plaintext rows written before encryption existed stay readable and get encrypted on their next write. It is an understandable migration convenience, but sitting next to "it does not quietly proceed with a weak key" it deserves naming as a branch that keeps a record readable.

The loss is not permanent either. Re-authorising through /api/linkedin-auth produces a fresh token, and because linkedin.ts calls encryptSecret when storing it, it lands under the current key. So the bill for rotating the token is not data loss: it is one LinkedIn re-authorisation, an update in two places (the single GitHub Secret named ADMIN_API_TOKEN and the env file on the server), every open admin session dropped, and a changed embed key. None of that is a catastrophe — and none of it is what you have in mind while rotating a token.

On the visitor side the effect is the harmless kind. Running the hash for one fixed IP and user agent across four configurations, the digest changed as soon as the token changed. As the code's own comment says: if the salt changes the hashes change, and rate limiting and duplicate detection reset once. Nothing breaks; a counter restarts.

One more detail: if both variables are empty, visitor.ts falls back to a string hardcoded in the repository. On the server that branch is never reached, because ADMIN_API_TOKEN is set — so the privacy property holds, but what holds it up is a dependency declared nowhere.

The OWASP secrets management guidance puts this in two sentences. The diagnosis: services often share the same secrets, which makes identifying the source of a compromise or leak challenging. The prescription: it should be immediately apparent what a secret is used for and where to find it. In my setup, what the token is for is apparent — for its declared job. You learn about its other two jobs by reading three files.

What a fresh install gets

All of this could be dismissed as particular to me. The repository's setup script says otherwise.

The header of .env.example states the file's promise clearly: the values you fill in go to two places, the server's /etc/<project>.env for the running site and GitHub Secrets for the automation. One list, two destinations. And when the target file does not yet exist, setup.sh does exactly that:

sudo cp .env "/etc/$SLUG.env"

So when somebody installs from scratch and follows the guide to the letter, the secrets file on their server is nothing more than the filled-in .env.example (the script only appends a DB_PATH line afterwards). Which means BLUESKY_APP_PASSWORD will not be there, because it is not in the template. Neither will TOKEN_ENC_KEY. On my own server the first one is set — I typed it in later by hand, which is the very inconsistency this piece is about. But TOKEN_ENC_KEY being empty on my server too is not an accident: it is the default output of the installation path.

One list does not stretch across two environments, because the two environments do not want the same things. The server process brings up the site and its API endpoints; the CI run generates and publishes content. One has no need for AI provider keys, the other none for the Turnstile secret. That is where the asymmetry in the count comes from: of the 14 keys on the server, 11 are in the template and three are not — BLUESKY_APP_PASSWORD, BLUESKY_SERVICE and LINKEDIN_REDIRECT_URI. It runs the other way too: four of the template's 29 names are never read under src/ or scripts/ at all, because they are R2 backup keys belonging to the deploy/backup.sh shell script.

So the file is not merely incomplete. It behaves as though there were a single runtime environment, when there are three: the service on the server, the CI run and the backup script.

Giving up on maintaining the document by hand

The lesson I take from this is not "keep your documentation current," because this file was never kept current. git log shows two commits: 28 of its key lines were written in one go on 11 June 2026, and one more was added on 28 September. The repository is not old either — the first commit is 31 March 2026. So the template was written four months ago in a single sitting, and it started falling behind the code that same day. A hand-maintained list that does not live next to the code can only decay.

The rules I now hold myself to:

A credential gets one name. If two environments need two names, one should fall back to the other, and that fallback should be visible on a single line of code. On the GH_API_TOKEN side I got this right already: the ops page checks GITHUB_API_TOKEN and falls back to GH_API_TOKEN, both names sitting side by side in one place. On the Bluesky side I did not, because the two reads live in two different files.

A default is a decision, not a convenience. Every sentence that begins "we already have a secret value" writes an undeclared dependency into the system. If a fallback is warranted, the place where the secret is documented should say so — not the module that consumes it: this token does these three jobs, and rotating it affects all three.

Derive the contract from the code, but do not trust your counter. The two grep commands above take seconds and produce the first two numbers on their own; the third needs the line starts of a file on the running server, which a check step cannot easily reach. And the second section of this piece shows the naive pattern missing at least four names. Any such check has to cover the ?. form and flag dynamic reads separately. As this piece goes live that check is not yet in place — the finding is open, not fixed.

If you want to run the same count on your own repository, the order is: extract the names from the template, extract the names the code reads, compare the two lists with comm, then read only the line starts of the file on the running server — you do not need the values. Then sort the difference: variables the runtime supplies itself are noise, tuning knobs with defaults are preference, anything shaped like a credential is a work list. Mine has 12 lines on it, and two are not the sort of thing you fix in one sitting; adopting TOKEN_ENC_KEY on a live box means setting the key and letting existing rows re-encrypt on their next write.

One of the habits that comes with experience is confusing knowing where a system is written down with knowing that it is written down correctly. The run where I counted sixty-eight backups that nginx read exactly one of involved a similar mistake: the existence of a file was taken as evidence that the file was being read.

A system's configuration contract is not the document describing that contract. It is the set of names the code reads — and seeing that set in full is harder than you would think. In my case it grew from 29 to at least 101, and along the way it tied three jobs to one token.

Official Sources

Node.js Crypto API documentation — behaviour of decipher.setAuthTag() and decipher.final(), and the error thrown on failed authentication
RFC 5116 — An Interface and Algorithms for Authenticated Encryption — §2.2 on the single output of authenticated decryption and the FAIL symbol, and authenticity defined relative to the key
OWASP Secrets Management Cheat Sheet — shared secrets, making a secret's purpose apparent, and the re-encryption impact of rotating encryption keys

I Have 188 Packages, Brew Refuses to Read Two of Them

Mustafa ERBAY — Fri, 23 Oct 2026 11:23:30 +0000

My intention was modest: free up some disk space. I typed
brew list --formula | wc -l and got 188. Add five application bundles (casks)
to that. Then I asked "how
many of these did I actually ask for?" and the whole thing turned on me. My
package manager answers that question with three different commands, and they
give three different numbers — and in the gap between those answers sit two
packages that appear in no list at all.

The space-clearing job never finished. What came out instead was a measurement
of how much a machine remembers about its own past. I'll walk
through all of it, because every number here has a mechanism underneath it,
and the mechanisms turned out to be more talkative than I expected.

Three commands, three answers

brew list --formula counts the 188 packages on disk. brew leaves says 51:
"the ones that aren't a dependency of another installed formula or cask."
brew autoremove --dry-run says nothing at all: the output is completely empty, there is
nothing safe to remove. One question, three numbers: 188, 51, 0.

They aren't actually asking the same question, and that's what I had failed to
read properly. brew list enumerates directories in the Cellar: what is on
disk. brew leaves collects the runtime-dependency names from the receipts of
installed packages and prints the ones that don't appear in that set: the
leaves of the graph. autoremove asks a third question: which package is both
a leaf and carries the "I didn't ask for this" record.

Putting the four numbers side by side, I finally saw what I had been asking:

Command	Answer	What it really asks
`brew list --formula`	188	How many directories are in the Cellar?
`installed_on_request: true` in receipts	64	How many installs did I request?
`brew leaves`	51	How many are leaves in the graph of loadable formulae?
`brew autoremove --dry-run`	0	How many are considered safe to delete?

The Cellar totals 8.76 GiB. The 64 packages I asked for hold 5.77 GiB of that;
the 124 that came along hold 2.99 GiB. I assumed the single largest item on my
disk was mine: llvm really is (1,797 MiB), but right behind it comes
llvm@21 (1,561 MiB), which I never asked for. zig and lld@21 pulled it
in. The receipts say the same thing: llvm's record carries the request flag,
llvm@21's does not. One of the two largest items on my disk is a decision of
mine; the other is a side effect.

"Why are you here?" has exactly one bit of answer

Every package carries an INSTALL_RECEIPT.json in its own directory. All 188
receipts have the installed_on_request key: 64 say true, 124 say false.
So my machine's memory of "you asked for this" is a single boolean per package.

There used to be a second field: installed_as_dependency. It is missing from
85 of my receipts, true in 74, and false in 29. I found the reason in
Homebrew's own code; the comment in tab.rb refers to the field as
"long-removed" and keeps backward compatibility so old receipts still load. So
those 85 receipts were written by a newer brew that no longer records it. One
bit is what's left.

Up to here it's an engineering decision, and a reasonable one. What stopped me
was looking at my own records. My shell history sits at its ceiling: the file
holds 1,225 lines, but 225 of them end in a backslash continuation, so the
record count is exactly 1,000 — my SAVEHIST limit. The only brew line in
there is a brew upgrade; there is no brew install line at all.
Every new command of mine deletes an old
one, and somewhere among
those old ones every single install line went away. The only record of my
intent is that bit, kept by the package manager itself.

My first count was wrong, by the way: grep -c brew ~/.zsh_history returned
nothing and I wrote down "zero". The file isn't plain text — file calls it
extended ASCII, so grep treats it as binary and goes quiet. Asked with
grep -a, one line showed up. The same thing happens to me a couple more times
in the rest of this piece.

The thirteen packages between 51 and 64

Only 51 of the 64 packages I requested are leaves. The first 11 of the
remaining 13 are explainable: even though I asked for them, they are now
something else's dependency. 37 packages lean on openssl@3, 20 on libtiff,
12 on glib. If I tried to delete those, half of them would come straight back
down.

The last two taught me a lesson about measurement. The first time I compared
the lists, I found a package that was "in leaves but not in my requested set":
powershell/tap/powershell. No cause for panic, because the cause wasn't real
— brew leaves prints packages by their full name (tap prefix included),
while the Cellar directory is just powershell. The difference between the sets
was in my text formatting, not in the world. In this kind of measurement, the
first finding being an artifact of your own script is a common event; once I
normalised the names, the gap closed.

But two names refused to close: sshpass and supabase. Both are packages I
requested, both say installed_on_request: true in their receipts, and neither
is a dependency of anything, which by definition makes them leaves. brew leaves doesn't count them. It doesn't error. It doesn't warn. They're just
absent.

Two packages on disk that the manager won't read

One command made the reason obvious:

$ brew info sshpass
Error: Refusing to load formula hudochenkov/sshpass/sshpass from untrusted
tap hudochenkov/sshpass.
Run `brew trust --formula hudochenkov/sshpass/sshpass` or
`brew trust hudochenkov/sshpass` to trust it.

Homebrew now refuses to load formulae from third-party taps unless you have
explicitly trusted that tap. The code has a crisp two-line definition: for a
tap to count as implicitly trusted it must be official and come from its
canonical remote (official? && canonical_remote?). Every other tap needs
approval, and approvals live in ~/.homebrew/trust.json. My file is 63 bytes
and holds a single-entry list: powershell/tap/powershell. The file's
modification time is 15 October, 09:05; the pwsh symlink is dated 09:06. So I
granted an approval in the same minute I installed PowerShell.

How that approval is given matters, because there is no spontaneous "do you
trust this?" prompt anywhere in the code: a trust record only comes from a
hand-typed brew trust command or from brew bundle's explicit flag, and
every other path stops at a single error. The full name in the file
(powershell/tap/powershell) says the same thing: I had the tap's identity in
front of me as I typed that line. The gate asked me for the right thing; I'm
the one who reduced it to a formality. A one-second decision, made so as not to
interrupt an install, turned out months later to be the only thing keeping two
packages inside the inventory.

That enforcement entered Homebrew with the Add tap trust enforcement commit
dated 30 May 2026, but the commit message itself says it was opt-in there: the
rule only applied when HOMEBREW_REQUIRE_TAP_TRUST was set. It became the
default through another commit on 8 June, which shipped first in version 6.0.0
(11 June 2026). So the gate has been shut on this machine for four and a half
months; I'm on 7.0.10 today. I installed sshpass on 24 February 2026 and
supabase two days later: both were inside before the gate existed, the gate
was fitted afterwards, and both ended up on the outside of it.

Now the concrete consequence: brew list shows these two packages, because
that command only looks at Cellar directories. brew leaves doesn't, because
it cannot load the formula. brew outdated reports 120 outdated formulae and
neither of them is on that list. Asked by name, it throws the error above. So
the scope of my update scan is not 188 packages, it's 186.

These two aren't dormant files either. The /opt/homebrew/bin/sshpass and
/opt/homebrew/bin/supabase links are in place, both are in my PATH, and both
run. sshpass -V prints version 1.06 and a 2011 copyright line. supabase is
2.75.0; that release went out on 2 February 2026, and the newest one upstream
was published this morning. Eight months sit in between.

The instructive part is this: my package manager didn't tell me, the package
itself did. Run supabase --version and two lines appear under the version
number saying a newer release exists and 2.75.0 is what's installed. The update
information is right there on the machine; the tool whose job is to collect it
and show it to me in one list has dropped that package from its list.

brew leaves being silent is, to me, the actual defect. When it can't load a
formula it drops the package from the list and doesn't tell me; the command's
error stream comes back completely empty. A one-line "I couldn't read these two
packages" would have replaced my hand-rolled set comparison. What the code does
is technically consistent — an unloadable
formula has no place in the graph — but an inventory command should at least
confess that its count is incomplete. If it were mine, I'd print the refusal as
a warning rather than an error: the behaviour stays the same, and the user's
blind spot closes.

I have no objection to the approval mechanism itself, quite the opposite. A
formula is Ruby code that runs during installation; the build steps, the
download addresses, the post-install scripts all come from that file. An
unaudited repository quietly updating that code amounts to permission to
execute code on my computer. I'd put the gate in the same place. My objection
isn't to the gate but to what happens behind it going unrecorded. Refusing to
load a formula I haven't trusted is the right call; dropping it from the
inventory as well erases the difference between "I won't load this" and "this
isn't here". And these two packages don't leave the disk or the PATH, and they
keep running: they just stop being counted.

The security side turns into a pleasant irony here. The gate's purpose is
right: don't blindly run code from a tap I haven't audited. But once the gate
closes, whoever is left inside also falls outside of management. NIST's guide to
enterprise patch management planning puts inventory second among its planning
recommendations (§3.2, Inventory Your Software and Assets) and says
inventories must be constantly updated or they go stale quickly; the same
section notes that a software bill of materials (SBOM) from a vendor can
augment that inventory. Mine is two lines short, and noticing it required
comparing sets by hand.

Why autoremove said zero

I had a bit of paranoia here: I didn't request 124 packages, so some of them
must be orphans. That turned out to be wrong. The graph, assembled from the
runtime-dependency lists in the receipts, shows that every one of the 124
unrequested packages has at least one requester; zero orphans. That's the real
reason autoremove came back empty.

But reading the code showed me there are three separate ways for it to come
back empty. autoremove only deletes packages that pass all three filters:
nothing in the graph depends on them, their receipt says installed_on_request
is explicitly false, and they were poured from a bottle (pre-built).
Packages built from source never even enter the list; three of them are like
that on this machine (powershell, sshpass, supabase). Cask dependencies
are separately protected as well.

On this machine the first of those three filters was enough on its own: with no
orphaned package, the list is empty before the others matter. The other two are
waiting for the day the graph changes: the three source-built packages on this
machine will never enter autoremove's list even if their requesters
disappear. For
someone who
actually wants to free space, the useful information is elsewhere: which of my
requests is single-handedly holding how many packages? My poppler install
holds 18 packages on its own, ffmpeg and ghostscript six each. Requesting
composer brings 56 packages into the closure, php 55, poppler 51. A
single brew uninstall decision frees 18 packages by that count — and none at
all by autoremove's answer today.

There are leftovers, too: brew cleanup --dry-run says roughly 41.9 MB. Four
packages have more than one version sitting in the Cellar; ca-certificates
has three at once (versions dated 19 March, 13 August and 25 September 2026).
Against 8.76 GiB, 41.9 MB is a comic number for my disk-space goal, but at
least nobody is hiding that number from me. After the measurement where I
deleted apps and left 661 MiB of data
behind, I pay
more attention to where the residue hides than to how big it is.

How far I can trust the receipt

With the measurement done, I was going to look at the dates in the receipts and
draw an "age distribution of my packages" chart. Luckily I tripped over an odd
line first: git-filter-repo appears to have been installed on 20 October 2026,
yet the same receipt's homebrew_version field says 4.6.9-6-g24e7d8f. That
tag is dated 3 September 2025, thirteen months before the install. Of the 70
packages installed after 24 June 2026, 24 carry a receipt naming a brew version
older than 6.0.

The answer is in the source again. When installing from a bottle, brew doesn't
write the receipt from scratch; it loads the one that came inside the bottle and
then refreshes only specific fields — install time, installed_on_request,
architecture, source path and so on. homebrew_version is not among the
refreshed fields. So the brew version in a receipt is the version of the brew
that built that bottle, not the one that installed it. The chart I was
planning to draw would have shown the history of Homebrew's build servers, not
of my machine.

The install time (time) field isn't an intent date either: since every
upgrade rewrites the receipt, that field holds "the moment this version was
installed". My oldest receipt is 21 January 2025 (git), the newest 20 October

Between those two dates, "when did I first ask for this package?" is written down nowhere. The intent bit surviving an upgrade is not an accident but deliberate: the code reads the value from the existing receipt and carries it into the new one. The only thing carried over is that bit — not the date.

Running the same measurement on your own machine

None of the numbers above needs a special tool; they all come from reading the
receipts in the Cellar. This single line gives you the request/dependency
split:

for r in /opt/homebrew/Cellar/*/*/INSTALL_RECEIPT.json; do
  /usr/bin/python3 -c 'import json,sys; j=json.load(open(sys.argv[1])); \
print(j.get("installed_on_request"))' "$r"
done | sort | uniq -c

On my machine this prints 128 False and 65 True; that totals 193, not 188.
The difference comes from four packages that keep more than one version in the
Cellar: the command counts every version directory, while my table takes the
newest receipt per package (64/124). If you're going to compare your number
with somebody else's, make sure both are counting the same thing.

To find the invisible ones, comparing two lists is enough, but do not forget to
strip the tap prefixes, or you'll see the same ghost difference I saw on my
first pass:

comm -23 \
  <(ls /opt/homebrew/Cellar | sort) \
  <(brew leaves | sed 's|.*/||' | sort)

For me this command dumps 137 names (188 minus 51), and every one of them means
"on disk, not in the leaf list". Most are legitimate, in that they're somebody's
dependency; brew info <name> tells you in one line which ones can't be read.
The paths differ by architecture: /opt/homebrew on Apple Silicon,
/usr/local on Intel Macs.

There is, however, a far shorter way to do this whole comparison, and I took
the long road until I found it. First, a single flag:

brew list --formula --full-name | grep /

That prints every package coming from a third-party tap by its full name — for
me, exactly three lines: the list of candidates for becoming invisible. Second,
open the gate for the length of one command and measure the difference:

HOMEBREW_NO_REQUIRE_TAP_TRUST=1 brew leaves | wc -l      # 53 for me, normally 51
HOMEBREW_NO_REQUIRE_TAP_TRUST=1 brew outdated --formula  # 121 for me, normally 120

The difference is exactly the two packages I expected, and the only one added
to the update list is supabase: so of the two that fall out of scope, one
really is behind, and the other (sshpass 1.06) is the newest version in its
own tap. Homebrew marks this variable as one that will be removed in a later
release, so it isn't a permanent fix; but for a once-a-year inventory audit
it's exactly what I was looking for. Your numbers will obviously come out
different; the value of the measurement isn't in the number, it's in how far
the four answers drift apart.

What I'll do tonight

At the end of the measurement my to-do list got shorter but clearer:

Settle the two invisible packages. Do I need sshpass? The cost of keeping a binary with a 2011 copyright in my PATH is higher than either making it visible with brew trust or removing it. Before deciding, I'll scan the scripts I use.
Move supabase to where the manager can see it. There's no defensible reason to keep a CLI that's eight months behind outside the patch scan.
Keep my intent somewhere other than brew. A 64-line Brewfile is a better memory than one bit per package; my shell history clearly doesn't remember.
Read the tap inventory once a year. I have four taps, three of them third-party. Adding a line to the trust file took me a second; reading it takes a second too.

There's no new tool on that list, because the missing piece was never a tool.
My package manager answers three questions from three different places, and I
had been treating all three as the same question.

The number that stays with me isn't 188 or 0; it's 2. When a security gate goes
up, we usually think "that thing can't get in anymore". But the gate can also
make invisible what is already inside: the same mechanism both protects and
drops things out of the catalogue. The inventory our machines report to us is
not an inventory of what they measure, but of what they can read.

Official Sources

I Broke the Thundering Herd, One Worker Took 99% of Connections

Mustafa ERBAY — Fri, 23 Oct 2026 06:22:20 +0000

Sharing one listening socket across several workers has a classic price: a connection arrives, the kernel wakes everyone waiting on that socket, one of them takes the connection, the rest go back empty-handed. We call it the thundering herd, and since Linux 4.5 it is supposedly solved by a single flag you pass to epoll_ctl — EPOLLEXCLUSIVE. The man page says so too, though what it actually says is that the flag is useful for avoiding thundering herd problems in certain scenarios.

I wanted to measure it. Eight workers, one socket, 500 connections, five repetitions. The herd really did break up: empty wakeups dropped from 869 to 1, and the workers' total CPU time was cut in half. Then I looked at the distribution table. In the median of the five runs, one worker had handled 497 of the 500 connections — and in all five runs, six workers had never woken up at all.

This piece is about that table. There is already a post on cache stampede and thundering herd on this blog, but that one lives at the application layer — clients asking for the same key at the same moment. The herd here is one floor down: processes sitting in the same socket's wait queue. Both the fix and the side effect of the fix are different.

The rig

I ran the measurement on this blog's own server: 18 vCPUs, Ubuntu kernel 6.8.0-147-generic, Python 3.12.3, load average around 5 at the time. Eight worker processes are created with fork, each builds its own epoll instance, registers the shared listening socket, and then blocks in epoll_wait. The parent dictates the registration order: no child calls epoll_ctl(ADD) until its turn comes. That detail matters later. The whole rig is three Python scripts, 305 lines.

The client side is plain: 500 connections, 2 ms apart, one at a time. Each worker counts three things — how many times it woke with an event, how many connections it accepted, and how many times it woke only to get EAGAIN from accept. That last one is the herd's invoice: you woke up, you ran, the plate was empty.

Four setups went through the rig. No flag: eight workers share one socket, nothing else. Flagged: they all register with EPOLLEXCLUSIVE. SO_REUSEPORT: no shared socket at all, each worker listens on the same address and port with its own. Half-migrated: half the workers registered with the flag and half without — the "we are moving to the flag but we are not done" state. Five runs each. The numbers below are the mean of those five, with the lowest and highest run in parentheses; the individual distributions I show in the text always come from one single run. The CPU figures are Python's figures — do not carry the absolute values over to a C server; the ratios are what matter.

Setup	Wakeups	Empty wakeups	Worker CPU (ms)	Largest worker share
No flag	1369 (1211-1501)	869 (63%)	702 (650-776)	36.7% (31.4-41.2)
`EPOLLEXCLUSIVE`	498 (495-501)	1 (0-1)	327 (294-380)	98.9% (97.6-99.8)
`SO_REUSEPORT`	499 (497-500)	0	351 (300-415)	14.6% (14.2-15.4)
Half-migrated	1294 (1143-1465)	795 (61%)	579 (534-644)	40.6% (38.0-41.8)

The no-flag row shows 1369 wakeups for 500 connections: 2.7 workers get up per connection, and 63% of those trips are wasted. The flag brings the count down to one per connection — 498 to be exact, meaning a couple of wakeups accepted two connections at once. So far the advertising holds.

The problem is in the last column. Let me say up front: that 98.9% is not a discovery, it is the arithmetic of the setup. Work time is zero and connections arrive 2 ms apart, so the first worker is always back in epoll_wait before the next one lands. What happens to the monopoly under load comes shortly. The genuinely odd thing is inside the no-flag run, and I missed it at first.

The no-flag secret: the last registrant wins

Here is the no-flag distribution (one run, in registration order, left to right):

[6, 9, 22, 35, 44, 78, 107, 199]

If the herd means "everyone wakes, one grabs it", then who grabs it should come down to race conditions, and over a long run eight workers should end up with roughly similar shares. Instead the worker that registered last took thirty-three times as many connections as the one that registered first. Same direction, similar magnitude, in every run.

With EPOLLEXCLUSIVE on, the opposite happens: now the first registrant takes everything. Same machine, same program; the single difference of one flag flips the distribution end to end. A symmetry that sharp is not an accident. The answer is in the code.

Two lines, two ends

When epoll starts watching a file, fs/eventpoll.c adds its wait entry like this:

if (epi->event.events & EPOLLEXCLUSIVE)
    add_wait_queue_exclusive(whead, &pwq->wait);
else
    add_wait_queue(whead, &pwq->wait);

The two functions write to two different ends. add_wait_queue puts the entry at the head of the queue; add_wait_queue_exclusive puts it at the tail, via __add_wait_queue_entry_tail. The waking side, __wake_up_common in kernel/sched/wait.c, always walks the queue from the head and stops at the first exclusive entry that answers "yes, I woke up":

if (ret && (flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)
    break;

The rest follows on its own. Without the flag, entries go to the head, so the queue is the reverse of registration order; the last registrant sits in front, gets woken first, and starts the accept race one step ahead. That is the 199-versus-6 gap. With the flag, entries go to the tail, so the queue matches registration order; the walk from the head hits the same first worker every time and stops there. That is the 98.9%.

In other words, "exclusive" is not a distribution policy. It is a stopping condition. The kernel tells you how many wakeups it will perform; it says nothing about which one.

(The measurement ran on a 6.8 kernel while the quotes above come from mainline. The four pieces that matter — ep_ptable_queue_proc, add_wait_queue_exclusive, the stopping line in __wake_up_common, and the ewake block coming up next — are identical in v6.8 and mainline.)

A control: does the winner follow registration order?

That explanation makes a testable prediction. If the winner is decided by epoll_ctl(ADD) order rather than by the order in which processes were created, then reversing the registration order should move the winner.

So I wired it that way. The fork order stayed fixed (0 through 7) and I reversed the registration order: the last process created became the first to register the socket. The result:

registration order: [7, 6, 5, 4, 3, 2, 1, 0]
accepted:           [500, 0, 0, 0, 0, 0, 0, 0]

The winner is worker 7 — born last, registered first. Running the same test without the flag, the winner was again the last registrant (226 against 3). What decides is not the process id, the creation order, or timing; it is the position in the socket's wait queue.

This is useful to know in practice: if one worker on a server is consistently doing many times the work of its peers, the place to look is not the worker, it is the startup order.

The kernel skips a busy worker — without breaking the order

So what happens when the first worker is drowning? There is a subtlety inside ep_poll_callback: for exclusive entries, the "I woke up" answer is only given if someone is actually waiting on that epoll instance (waitqueue_active). If the worker is outside epoll_wait, that is, busy with work in hand, the callback returns zero; __wake_up_common does not count it and moves on to the next entry.

Adding fake work after each accept puts numbers on it:

Work time	Distribution (in registration order)	Workers used
0 ms	`[499, 1, 0, 0, 0, 0, 0, 0]`	2
1 ms	`[460, 37, 3, 0, 0, 0, 0, 0]`	3
5 ms	`[189, 165, 135, 5, 6, 0, 0, 0]`	5
10 ms	`[115, 110, 106, 96, 68, 4, 1, 0]`	7

That table is what finally made the mechanism intuitive for me. The kernel does not treat the eight workers as a pool; starting from the head of the queue, it calls up the next one only as long as the work is not keeping up. The numbers land exactly where queueing theory says they should: serving a connection in 10 ms while one arrives every 2 ms needs about five workers in steady state — and in the measurement the first five carry 495 of the 500. A 5 ms service time needs about two and a half; the first three carry 489. The eighth worker did not wake once in any of these four runs; even at 10 ms, with seven workers busy at the same time, its turn never came.

The table explains one more thing: that 99% up top is not a fixed number, it is a product of low load. Running the same measurement three more times while the server was busier, the first worker's share fell as low as 88.6% — under load that worker spends more time outside epoll_wait, so the turn passes to the second. But across all eighteen zero-work runs of the flag, the one in front was always the first registrant, with a share somewhere between 88.6% and 100%. Load softens the monopoly; it does not remove it.

This is not bad design — it genuinely minimises wakeups, keeps a hot process hot, and is kind to cache locality. But read it as "load balancing" and it will mislead you. It is not a balancer; it is an overflow chain.

Kill the front worker and the monopoly moves next door

Once the mechanism is clear, the first production question is obvious: what happens if the worker at the head of the queue crashes? The rig answers that too — a SIGKILL to the first registrant at connection 250:

killed: registration rank 0 · surviving workers, in registration order
accepted: [250, 0, 0, 0, 0, 0, 0]

The next worker in line took every one of the remaining 250 connections, with zero empty wakeups. The monopoly does not break; it shifts by one seat. Three runs out of three. The operational consequence matters more than it looks: when you restart the crashed worker, it registers the socket again and lands at the tail of the queue. A single crash permanently rearranges your load distribution — yesterday's first is now last.

One legacy worker brings the herd back

There is an easily skipped sentence in the man page: if the same file is registered in several epoll instances, some with the flag and some without, the event goes to all of the non-exclusive ones and to at least one of the exclusive ones.

The "half-migrated" row in the table is exactly that: four of the eight workers registered with the flag, four without — the state of a setup in the middle of a migration. The distribution came out like this:

registration order: 0(E)  1     2(E)  3     4(E)  5     6(E)  7
accepted:            37   57     0    76     0   128     0   202

Wakeups are back up to 1294 and empty wakeups to 795; barely different from the no-flag row. On top of that, three of the four flagged workers accepted nothing. The cause is the queue layout again: non-exclusive entries are added at the head, so they all sit in front of the wake list (and in reverse among themselves: 7, 5, 3, 1), the exclusive ones are behind them, and the walk stops at the first of those. The measured ordering — 202 > 128 > 76 > 57 > 37, then three zeroes — is exactly what the queue layout predicts. A migration left half-finished gives you the bad half of both worlds.

nginx hit this wall in 2022

After finishing the measurement I went to look at what nginx does, since it is the best-known user of this flag. Its change log tells the story in two lines. 1.11.3 (July 2016): accept_mutex became off by default and nginx started using EPOLLEXCLUSIVE on Linux. 1.21.6 (January 2022): a bug fix — with EPOLLEXCLUSIVE on Linux, client connections were unevenly distributed among worker processes.

To see what went missing in between, it is enough to read the accept_mutex documentation: with that directive on, worker processes accepted connections by turn. The docs still say there is no need to enable it on systems that support EPOLLEXCLUSIVE — but when the flag took over the lock's herd-suppression job, it did not take over the turn-taking. The turn came back six years later, in user space.

The fix in the source is called ngx_reorder_accept_events, and its comment matches my table word for word: Linux usually notifies the process that added the socket to epoll first, so "most of the connections are handled by the first worker process". The remedy is to re-add the socket periodically. The interval is visible in the code: a DEL + ADD every 16 accepts (more often as a worker approaches its connection limit). Re-adding moves the entry to the tail of the queue, and the turn passes on.

I tried the same thing in my own rig — re-register every 16 accepts:

[64, 64, 64, 64, 64, 64, 64, 52]

Identical numbers in all three runs; nothing but the arithmetic of 16-accept rounds. Zero empty wakeups, still one wakeup per connection, worker CPU at 335 ms — indistinguishable from the flagged run's 294-380 ms band. So the entire gain from the flag survives and the distribution is fixed too. That a forty-line user-space workaround on top of a kernel behaviour lands this cleanly honestly surprised me.

Which raises a question: if it is this simple, why doesn't the kernel do it? It was proposed. In 2015, alongside EPOLLEXCLUSIVE, a second flag named EPOLLROUNDROBIN was suggested to spread wakeups across waiters in turn. One flag went in; the other did not. There is no EPOLLROUNDROBIN in today's mainline tree — eleven years later, still none.

One epoll, eight threads: same kernel, different outcome

To find out whether the problem lives in the idea of exclusive wakeups or somewhere else, I set the rig up once more: this time a single epoll instance shared by eight threads, no flag.

[95, 59, 62, 44, 76, 42, 57, 65]

No monopoly. At 1.6 wakeups per connection it is also below the 2.7 of the process version, though not zero.

The reason is in the source again. ep_poll re-adds its wait entry on every loop and removes it on wakeup; the removal is done by ep_autoremove_wake_function, and the comment inside ep_poll states the rationale plainly: when there are several waiters, each new wakeup should reach the next one. So the kernel does rotate inside a single epoll instance. Where it does not rotate is where separate epoll instances sit permanently in the socket's queue.

The monopoly is not caused by the flag; it is caused by the entry being permanent. nginx's fix makes exactly that temporary.

The flag's small print

While measuring I also tried, one by one, the things the flag refuses, because the list of restrictions in the man page is long and most of them show up in your code as a quiet EINVAL:

Call	Result
`ADD` with `EPOLLIN\	EPOLLEXCLUSIVE`
A later `MOD` on the same fd	`EINVAL`
`ADD` with `EPOLLEXCLUSIVE\	EPOLLONESHOT`
`ADD` with `EPOLLEXCLUSIVE\	EPOLLET`
`EXCLUSIVE` when the target is an epoll fd	`EINVAL`

The second row is the one that will hurt in practice: a descriptor you added with EPOLLEXCLUSIVE can never be changed with EPOLL_CTL_MOD again. If you want to update the set of events you care about, the only path is delete and re-add. That also explains why nginx's rotation trick is a DEL + ADD: there is no other door.

How to check this on your own server

The one actionable output of this article is a question: are accepted connections spread evenly across your workers? There are two ways to look.

On nginx the cheapest is the access log: put $pid in your log_format (the docs define it as exactly that, the PID of the worker process) and count PIDs over one log file. The distribution shows up right there.

The way that works regardless of the server is to ask the kernel. I tried it on VPS3:

bpftrace -e 'kretprobe:inet_csk_accept /retval != 0/ { @accepts[comm, pid] = count(); }'

My lab run came out of it as these lines:

@accepts[python3, 3259746]: 4
@accepts[python3, 3259745]: 196

In a four-worker run, 196 of 200 connections had gone to a single process. The same dump also listed the server's own services with their own counts — so this is not a lab-only trick but a measurement you can run directly on a live machine. A one-minute run is enough to tell you whether the imbalance is there.

Which one to pick

The decision frame this measurement leaves behind:

With a single worker the flag means nothing. nginx only turns it on when worker_processes is greater than one.
If you care about CPU and not about the spread between workers — for instance, your workers are already loaded enough to saturate — the flag alone is enough. Under load the queue opens downward by itself.
If you want an even spread at low or medium load, the flag alone will not get you there. Either do what nginx does (periodic DEL+ADD) or use SO_REUSEPORT.
If you pick SO_REUSEPORT, know its two blind spots. The first is distribution: the kernel picks the socket with a hash derived from the connection's four-tuple (inet_ehashfn: destination address, local port, source address, source port), with no regard for how busy a worker is. In my rig the shares settled in the 14-15% band because every connection was an exact copy of every other; in real traffic, if one long-lived connection jams a worker, new connections in that queue wait even while the worker next door sits idle. If you want to take the choice back, the door is open: SO_ATTACH_REUSEPORT_CBPF/EBPF lets you pick the socket in the group yourself (on TCP since Linux 4.6). The second blind spot is sneakier, and it has a measurement of its own: when you swap workers, the connections sitting in the closing listener's queue die silently unless tcp_migrate_req is on — and that switch is off by default.
If you are migrating, migrate in one go. Running flagged and unflagged workers side by side brought the herd back in full.

What the docs promised, and what I read

The real lesson I took from this is not about the flag. For EPOLLEXCLUSIVE the man page says that one or more of the epoll descriptors will receive an event. For years I read that sentence as "one of them gets it, problem solved"; what the sentence actually gives you is only a count. It makes no promise about which one. The promise it never made, I had added myself.

This trap is common in kernel interfaces: the documentation tells you a quantity and we read a policy. The distance between "the minimum number of wakeups" and "a fair share" was, in my rig, 497 out of 500 connections. Without measuring I would never have noticed; the server was running, requests were being answered, and six workers simply never woke.

The question you can put to your own setup is simple: have you ever compared the number of accepted connections across your worker processes? If they are not even, that is not a bug — it is most likely exactly what the kernel promised.

Official Sources

I Moved the Binary Twelve Ways, Six Kept the File Capability

Mustafa ERBAY — Thu, 22 Oct 2026 17:23:41 +0000

"Don't use setuid, grant a capability" is good advice. Giving a program the whole of root just so it can open a raw socket, when cap_net_raw alone would do, trades the forty-one capabilities root carries (the kernel's cap_last_cap is 40, i.e. 0-40) for one. I agree with the advice and I follow it myself.

The problem isn't the advice — it's what the advice leaves out. The setuid bit is part of the file mode: ls -l shows it, cp carries it, tar packs it, everyone knows about it. A file capability lives in an extended attribute called security.capability — and extended attributes are quietly dropped by default in most of the tools that move a file from one place to another. So granting the capability correctly isn't the end of the job. The job ends when the file actually arrives at its destination still carrying it.

That is what today's lab run shows. The same binary went through twelve different transfer routes; six arrived with the capability, six silently left it behind. Along the way the capability model showed four more sharp edges: a case where getcap tells the truth and the kernel still hands out nothing, a one-letter typo that makes a binary unrunnable even for root, the same typo turning a loud failure into silent nothing when you change that letter, and the fact that the capability cannot be written at all into a directory mounted from my Mac.

Everything here comes from commands run today. The rig: Docker Desktop 4.37.2 (Docker Engine 27.4.0, Buildx v0.19.2) on my Mac, linuxkit kernel 6.10.14, debian:trixie-slim inside it (Debian 13.7), libcap2-bin 1:2.75-10+deb13u1+b3, tcpdump 4.99.5-2, coreutils 9.7-3, rsync 3.5.0, git 2.47.3.

Source verification was done against security/commoncap.c in Linux 7.3, while the measurements ran on 6.10.14. The two were compared directly: the functions quoted below (bprm_caps_from_vfs_caps and get_file_caps) are byte-for-byte identical in v6.10 and v7.3, so the code quotes apply to the kernel under test as well.

First, an inventory: nobody is using this model

Curiosity first: in a clean Debian image, how many files are setuid and how many carry a capability?

find / -xdev -type f -perm -4000 2>/dev/null | wc -l
getcap -r / 2>/dev/null | wc -l

The answer is lopsided. Eight setuid files:

/usr/bin/chsh
/usr/bin/chfn
/usr/bin/umount
/usr/bin/mount
/usr/bin/newgrp
/usr/bin/passwd
/usr/bin/gpasswd
/usr/bin/su

Files carrying a capability: zero.

That is surprising, because neither mount nor passwd genuinely needs full root, and the model is hardly new — the attribute has been around long enough to go through three revisions (capability.h defines VFS_CAP_REVISION_1, _2 and _3). But the more interesting half of the story is ping. For years ping was the showcase example: once setuid, later migrated to cap_net_raw=ep. Looking at it now:

-rwxr-xr-x 1 root root 213448 Apr 22  2025 /usr/bin/ping

No setuid bit, and getcap prints nothing. Debian has taken ping out of the capability model too. The package changelog says why; the iputils 3:20240905-1 entry (24 September 2024) reads: "ping: Add Recommends on linux-sysctl-defaults and stop setting additional capabilities". Two releases earlier, in 3:20240117-1, the setuid path was closed: "Stop falling back to setuid binaries for raw socket access".

The same changelog, in the same release, also contains the opposite move: "Set cap_sys_nice in iputils-clockdiff". So Debian isn't abandoning capabilities wholesale — it is taking one away from ping and giving one to clockdiff. Capabilities aren't the wrong tool; they became an unnecessary tool for ping.

What replaced it is decided by a sysctl:

ping_group_range: 0 2147483647

One caveat is owed here: that value was read inside a container, and as such it is Docker's contribution rather than Debian's — the debian:trixie-slim image does not install linux-sysctl-defaults, and the range is written into the container by Docker's own default. On a real Debian 13 install the same result is reached through the linux-sysctl-defaults package the changelog recommends. Different route, same destination: ping no longer opens a raw socket at all — it opens an unprivileged ICMP datagram socket, and that range decides who may.

The showcase example has been taken out of the showcase, because the best capability is the one you never need. That's a useful frame for the rest of this article: there are ways to carry a capability correctly, but not carrying one is always cheaper.

So the experiment here rests not on ping but on a program that genuinely needs a raw socket: tcpdump. I wrote about the tool itself in the production packet-capture runbook; here I'm dealing with its privileges.

Granting the capability: three lines, three different outcomes

Copy tcpdump, leave its mode at 755, run it as an ordinary user:

cp /usr/bin/tcpdump /opt/td
su deneme -c "/opt/td -i lo -c 1 -w /dev/null"

td: lo: You don't have permission to perform this capture on that device
(socket: Operation not permitted)

Exit code 1. As expected. Now the capability. Since -c 1 waits for the first packet, and lo sits idle in a fresh container, traffic has to be generated first:

setcap cap_net_raw+ep /opt/td
(ping -c 3 -i 0.3 127.0.0.1 >/dev/null 2>&1 &) ; sleep 0.2
su deneme -c "/opt/td -i lo -c 1 -w /dev/null"

td: listening on lo, link-type EN10MB (Ethernet), snapshot length 262144 bytes
1 packet captured
8 packets received by filter
0 packets dropped by kernel

The file is still owned by root, its mode is still -rwxr-xr-x, there is no setuid bit — and an ordinary user is capturing packets. The model worked exactly as advertised.

Then +p instead of +ep. One letter:

setcap cap_net_raw+p /opt/td

td: lo: You don't have permission to perform this capture on that device
(socket: Operation not permitted)

Back to square one. Meanwhile getcap happily confirms there is a capability on the file: cap_net_raw=p. That e is the "effective" bit, and it means: should the capability go into the process's effective set, or will the program raise it itself? tcpdump is not capability-aware — it doesn't call capset to promote something from its permitted set into its effective set. So with +p it receives the capability and cannot use it.

To see this from the process side, three copies of capsh marked three different ways, each printing its own sets:

marking on the file	process's `Current:` line
no capability	`=`
`cap_net_raw=p`	`cap_net_raw=p`
`cap_net_raw=ep`	`cap_net_raw=ep`

The middle row is the trap itself: the capability is there, in the process's hands, and not effective. My rule became this — if the program you're marking doesn't call capset, write +ep, and don't go hunting for why +p didn't work.

Where the capability actually lives

To see where the difference between +ep and +p is stored, look straight at the attribute:

getfattr -d -m - /opt/td

# file: opt/td
security.capability=0sAQAAAgAgAAAAAAAAAAAAAAAAAAA=

Twenty bytes. Decoded, the structure is entirely legible:

raw hex:   0100000200200000000000000000000000000000
magic_etc = 0x02000001
  revision = 0x02000000   effective bit = 1
permitted = 0x00002000 -> bit set: 13

Twenty bytes matches the kernel's version-2 layout. The capability.h definition reads:

#define VFS_CAP_U32_2           2
#define XATTR_CAPS_SZ_2         (sizeof(__le32)*(1 + 2*VFS_CAP_U32_2))

With VFS_CAP_U32_2 equal to 2, that is 4 × (1 + 4) = 20. Version 3 would be 24 bytes, because XATTR_CAPS_SZ_3 counts one more __le32: the rootid field. That field isn't decoration — inside a user namespace it records which mapped root the capability belongs to, which is what rootless containers need. If you run rootless Docker or Podman, you'll meet v3 rather than v2.

The effective bit, meanwhile, is not a separate set but the lowest bit of the magic_etc field; the entire difference between +ep and +p is that one bit. Only bit 13 is set in the permitted set, and capability.h says #define CAP_NET_RAW 13. The capability granted is where it was expected.

Worth pausing on: a twenty-byte attribute shows up neither in the file size nor in ls -l output. That invisibility is the reason this whole article exists.

Twelve ways to move it, six survivors

The capability is granted and working. Now the real-world question: if the file moves somewhere, does the capability come along? The same binary went through twelve different transfer routes, with getcap checked at the destination each time.

transfer method	capability arrived
`cp`	no
`cp -p`	no
`cp -a`	yes — but silently (see below)
`cp --preserve=xattr`	yes
`install -m755`	no
`tar cf` + `tar xf`	no
`tar --xattrs --xattrs-include='*'` (both ends)	yes
`rsync -a`	no
`rsync -aX`	yes
`mv` (across filesystems)	yes
`git add` + `git checkout`	no
Dockerfile `COPY --from`	yes

Six of twelve kept it. Five rows stand out.

The cp -p row is the one that stings. We read -p as "preserve", but cp --help defines it precisely: -p = --preserve=mode,ownership,timestamps. Extended attributes aren't on that list. Assuming -p is enough is the shortest path to shipping the mistake to production without noticing.

The cp -a row says "yes" but deserves an asterisk, because the coreutils manual describes -a as trying to preserve extended attributes but "ignore any failure to do that and print no corresponding diagnostic". --preserve=xattr is the opposite: "Preserve extended attributes of the file, or fail with full diagnostics." On this rig both carried the capability. But in an environment where CAP_SETFCAP has been dropped, or onto a destination filesystem that can't hold the attribute, cp -a drops the capability with exit code 0 and without a word. It would be odd for an article whose thesis is "the system doesn't tell you" to recommend the flag that is documented not to tell you: in automation, write --preserve=xattr.

The install -m755 row is sneakier, because install is the final step of nearly every Makefile. Here the measurement simply matches the documentation, which states it in one line: "install never preserves extended attributes (xattr)." This isn't a side effect, it's by design. A flow that runs setcap during the build and then puts the file in /usr/local/bin with make install leaves the capability behind every single time. The correct order is the reverse: install first, then setcap the installed file.

The rsync -a row will bite hardest in practice, because rsync is the deployment transport in most places and -a has become a reflex. -a isn't enough here either; -X (--xattrs) is required. The same reflex breaks with tar, and there the measured detail is the one that made me think hardest. Comparing the two archives:

plain.tar size: 1331200
xattr.tar size: 1331200

Identical. The twenty-byte attribute disappears into the archive's block padding. So you cannot answer "does my archive carry the attribute" by looking at file size; you have to unpack it and check with getcap. It also matters that --xattrs is required on both ends: specify it only when packing and the archive will contain the attribute but extraction will drop it.

The git row isn't a shortcoming but a structural fact: git stores no extended attributes. A file that is cap_net_raw=ep before the commit comes back bare after git checkout. The practical consequence is that a capability can never live in a repository. That makes the "install first, then setcap" rule a requirement rather than a suggestion. In production the place this happens is the packaging step: on the .deb side setcap is called from postinst (exactly what Debian does for clockdiff), and on the RPM side the %caps directive does it.

The mv row stands slightly apart as the only true move: within one filesystem mv is a rename, so the attribute simply stays with the file. Across filesystems it genuinely copies, and the documentation reads: "mv always tries to copy extended attributes (xattr)… Upon failure all warnings are emitted, but do not modify the exit status." So mv warns but still exits 0. That yields a three-point spectrum: --preserve=xattr errors on failure, mv warns but doesn't change the exit status, cp -a says nothing at all.

The last row refuted my expectation. COPY --from inside a Dockerfile preserves the capability — I assumed it dropped it, and measuring showed the opposite:

#5 3.439 YAPICI KATMANDA:
#5 3.440 /td cap_net_raw=ep
#8 0.087 COPY --from SONRASI:
#8 0.088 /td-copy cap_net_raw=ep

(The labels in that log are the Turkish echo strings from my Dockerfile: "in the builder layer" and "after COPY --from". The output is quoted verbatim.)

In a multi-stage build, running setcap in the builder stage and carrying the binary into the final image with COPY --from works. I went looking for this in Docker's documentation and couldn't find it: COPY is documented as preserving "file permissions" and says nothing about extended attributes. So the basis for that row isn't documentation, it's the build log above — and undocumented behaviour is behaviour that can change with a version. Measure it once on your own version; this one came out on Buildx v0.19.2.

I can't say the same for an ordinary COPY from the build context, though, because on my Mac a capability-carrying file cannot be put into the context at all. The reason comes a little further down.

`getcap` is telling the truth, and the kernel grants nothing

Now the measurement with the strongest smell of a trap. Two tmpfs filesystems — one normal, one nosuid — with the same binary and the same capability on both.

mount -t tmpfs -o nosuid tmpfs /mnt/nosuid
mount -t tmpfs tmpfs /mnt/suid

--- /mnt/suid    on file: cap_net_raw=ep   -> Current: cap_net_raw=ep
--- /mnt/nosuid  on file: cap_net_raw=ep   -> Current: =

Note that on the nosuid side getcap still reports cap_net_raw=ep on the file. The attribute really is there; getcap isn't lying. But the process receives nothing. Before reading the attribute the kernel looks at the mount, and if it's nosuid it skips file capabilities entirely. The code is in get_file_caps, two lines:

if (!mnt_may_suid(file->f_path.mnt))
    return 0;

The auditing consequence is this: getcap alone is not sufficient as an audit tool. It answers "which files carry a capability"; it does not answer "which files can actually use one". For the second question you also have to look at mount options — and /tmp, /dev/shm, many /home mounts and most mounts inside containers are already nosuid. Just above those two lines there's also a file_caps_enabled check; file capabilities can be switched off wholesale at kernel build or boot time.

The place this rule bit me directly was the bind-mounted directory from my Mac. Docker Desktop shows the mount like this:

fakeowner rw,nosuid,nodev,relatime,fakeowner

Trying setcap there simply didn't work:

Failed to set capabilities on file 'setcap': Operation not supported

Exit code 1. So anyone bind-mounting a directory from their Mac and trying to grant a capability to a binary inside it hits two walls at once: the filesystem can't hold the security.capability attribute at all, and the mount is nosuid anyway. The same limit applies to NFS and to many read-only image layers; the issue isn't macOS, it's whether the filesystem can hold the attribute. The practical upshot is clear — a capability-carrying binary has to live inside the container image, not on a mounted volume. (The error message naming the file 'setcap' is a quirk of libcap 2.75's own wording; it doesn't change the measurement, but it did sidetrack me for a minute the first time I saw it.)

Writing the capability needs its own permission, too. In a container with CAP_SETFCAP dropped, setcap fails:

unable to set CAP_SETFCAP effective capability: Operation not permitted

And that came back as uid 0 — being root inside the container is not enough to write a file capability. On the kernel side the corresponding check is capable_wrt_inode_uidgid(idmap, inode, CAP_SETFCAP) inside cap_convert_nscap.

The bounding-set ceiling: a wrong capability locks the binary

This is the finding with the most potential to hurt. Granting the same binary a capability the container does not have:

setcap cap_sys_time+ep /opt/t

The result isn't the program failing — it's the program never starting:

bash: line 1: /opt/t: Operation not permitted

The exit code the shell reports is 126. Tried as root too: same error, same 126. So a single wrong setcap call renders a working binary unrunnable for everyone in that container. You can't even get --version out of it.

A note on that number: 126 is the shell's language, not the kernel's. The execve() call returns EPERM; it's bash that renders "found but not executable" as 126. The same failure appears as 203/EXEC under systemd, and as plain errno == EPERM in a program that calls execve() directly. When you write a monitoring rule, the thing to match is not a single number.

The cause is the container's capability bounding set. The CapBnd value in /proc/self/status is 00000000a80425fb, which decodes to 14 capabilities:

cap_chown, cap_dac_override, cap_fowner, cap_fsetid, cap_kill,
cap_setgid, cap_setuid, cap_setpcap, cap_net_bind_service,
cap_net_raw, cap_sys_chroot, cap_mknod, cap_audit_write, cap_setfcap

That set of 14 is Docker's default; it matches the DefaultCapabilities list in the moby source exactly, and cap_sys_time is not in it. (A note on naming: "bounding set" is my term — Docker's documentation only speaks of capabilities being "kept" or "dropped" and never says which set it means; the 14 were read from CapBnd directly.)

The kernel states plainly what it does here. The formula in bprm_caps_from_vfs_caps and the check right after it:

/*
 * pP' = (X & fP) | (pI & fI)
 * The addition of pA' is handled later.
 */
new->cap_permitted.val =
    (new->cap_bset.val & caps->permitted.val) |
    (new->cap_inheritable.val & caps->inheritable.val);

if (caps->permitted.val & ~new->cap_permitted.val)
    /* insufficient to execute correctly */
    ret = -EPERM;

X in the formula is the bounding set. The file wants cap_sys_time, the bounding set doesn't have it, the intersection comes out empty — and when a bit the file asked for is missing from the result, the kernel returns -EPERM. The comment states the intent: a program running with a capability missing would do the wrong thing, so the kernel chooses not to start it at all.

The next line of that code predicted something that hadn't been measured yet:

return *effective ? ret : 0;

EPERM is returned only if the effective bit is set. So granting the same wrong capability with +p should let the program start. Setting up that experiment:

marking on the file	exit code	process's `Current:` line
`cap_sys_time+ep`	126, `Operation not permitted`	(never ran)
`cap_sys_time+p`	0	`=`

The prediction held exactly. With +p the program starts fine — and receives no capability at all. The choice between them is a choice between a loud failure and silent nothing. Which is worse depends on your setup: your monitoring will catch the 126 from +ep immediately, whereas the silence of +p only surfaces when the program tries to open a raw socket and fails, probably in the middle of the night.

The same experiment again with one change — giving the container the capability:

docker run --cap-add SYS_TIME ...

The bounding set went from 14 to 15, cap_sys_time joined it, and the binary marked cap_sys_time+ep ran with exit code 0. Same file, same setcap, same kernel; the only variable was the bounding set. That's the cause.

How the decision flows at execve

Every behaviour recorded above sits on a single decision chain:

The order in that diagram is itself a claim: the nosuid check comes before the bounding-set arithmetic. That's why granting a wrong capability on a nosuid mount produces no EPERM — the kernel skipped the attribute before it ever got there.

Secure-exec: you get the side effect too

The last measurement is a behaviour a file capability inherits whether you want it or not. Two binaries, one with a capability and one bare, both asked to preload a nonexistent library:

su deneme -c "LD_PRELOAD=/yok/kitaplik.so /opt/ciplak --version"
su deneme -c "LD_PRELOAD=/yok/kitaplik.so /opt/td --version"

The bare binary complained:

ERROR: ld.so: object '/yok/kitaplik.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

The capability-carrying binary said nothing at all; it printed its version line straight away. Because an exec that gains a capability triggers the kernel's secure-execution mode, and in that mode the dynamic loader silently ignores variables like LD_PRELOAD and LD_LIBRARY_PATH.

The su deneme part is essential, because the condition is not merely "a capability was gained". The assignment in commoncap.c reads:

(!__is_real(root_uid, new) &&
 (effective ||
  __cap_grew(permitted, ambient, new))))
    bprm->secureexec = 1;

!__is_real(root_uid, new) — a caller whose real uid is not root. Run the same command as root and secureexec stays 0, LD_PRELOAD is honoured, and the capability-carrying binary prints the very same ld.so error. A file capability does not put root into secure-exec. (The capability-related part of this condition is identical in v6.10 and v7.3; the setuid checks around it changed.)

Security-wise this is what we want: a privileged binary should not accept injected code from outside. But it has a side effect that can bite you in practice: if you have a wrapper script that wraps the binary with LD_PRELOAD or a custom LD_LIBRARY_PATH, the day you apply setcap that wrapper quietly stops mattering. It doesn't error. It just no longer does anything.

Two ways to avoid granting it at all

Before the checklist, a reminder of the frame from the opening: the best capability is the one you don't grant. If you're writing a service you may not need setcap at all, because systemd can hand the capability to the process rather than the file — via AmbientCapabilities= and CapabilityBoundingSet=. The binary then stays bare, the attribute-transport problem disappears entirely, and the twelve-row table above stops being your concern. I covered the neighbours of those settings on the systemd service hardening side.

There's a warning in the same place: setting NoNewPrivileges=yes silently disables file capabilities — the container equivalents being --security-opt no-new-privileges and Kubernetes' allowPrivilegeEscalation: false. So your hardening setting can cancel your setcap solution without telling you. That is the last of the silent failures on this list.

The second route is replacing the capability with a different mechanism altogether. ping moving to ping_group_range is the example. Similarly, binding to a privileged port can use net.ipv4.ip_unprivileged_port_start or socket activation instead of cap_net_bind_service. In the article on ptrace_scope I noted the reverse direction: some capabilities (cap_sys_ptrace among them) are in practice aliases for root, and granting them is no safer than setuid.

Implementation checklist

The order these measurements produced, and what I apply in my own setups:

First ask whether you need the capability at all. If systemd's AmbientCapabilities=, a suitable sysctl or socket activation will do, don't touch the file.
Always grant the capability after installation. install never carries the attribute by definition and git never stores it, so a capability can only be granted at the packaging/installation step (.deb postinst, RPM %caps).
In automation, write --preserve=xattr rather than cp -a, and rsync -aX rather than rsync -a. The first errors on failure; the second adds the attribute -a skips.
Choose +ep or +p by looking at the program. If it doesn't call capset, use +ep. +p is meaningful for capability-aware programs; otherwise it hands you silent nothing.
Verify the capability is in the bounding set before you grant it. Check with capsh --decode=$(grep CapBnd /proc/self/status | awk '{print $2}'); writing one that isn't there with +ep makes the binary unrunnable.
Check that the mount the file lives on isn't nosuid. findmnt -no OPTIONS <path>. A positive answer from getcap does not replace this check. Scan for NoNewPrivileges / no-new-privileges with the same eye.
In containers, keep the capability-carrying binary inside the image. Mounted volumes are nosuid, and macOS mounts and NFS can't hold the attribute at all. Running rootless, expect the v3 attribute layout.
Audit with two questions: which files have the attribute (getcap -r /) and whether those files sit on a non-nosuid mount. The first question alone misleads.
Measure your backup and restore path once. Is --xattrs on both ends of your tar, does your rsync get -X, does your image build carry the attribute — file size won't tell you, getcap will.

Conclusion

File capabilities are better than setuid; that still holds after measuring. But the difference between the two models isn't only the breadth of privilege granted — it's visibility. The setuid bit lives in the file mode, and every tool knows and carries the mode. A file capability lives in an extended attribute, and extended attributes are a data type that falls off by default all along the plumbing: dropped by cp, dropped by install by design, dropped by tar and rsync unless you ask for a flag, never held by git, present but inert on a nosuid mount, unwritable on a directory mounted from a Mac.

Today's real lesson: in the capability model, the mistake rarely arrives as "I granted the wrong capability". It arrives as "I granted the right capability, then moved the file". And outside two edge cases — the binary that refuses to run and the silence of ld.so — the system doesn't tell you. Six of twelve transfer routes quietly leaving the capability behind shows that the discipline isn't in granting it, it's in everything after.

So the question to ask about your own setup: how many transfer steps did your capability-carrying binary take to reach where it sits in production today, and which of those steps was verified with getcap?

Official Sources

A 9 MiB Clone, Then Checkout Fetched 204 MiB More

Mustafa ERBAY — Thu, 22 Oct 2026 17:20:50 +0000

This blog's repository got fat: 3,538 commits, 46,219 objects, a 276 MiB pack. The weight is not in the history, it is in the present — every article also has a cover PNG, and PNGs do not compress. Once cloning started to annoy me, my reflex was everybody's reflex: --depth 1. Shallow clone. Drop the history, take the current state, get on with it.

The reflex was wrong. --depth 1 fetched 204.3 MiB; --filter=blob:none fetched 9.2 MiB. A twenty-two-fold difference, and against the shallow clone. The real lesson came afterwards: those 9 MiB were not a saving, they were a loan. The first git checkout showed up at the door and collected 204.1 MiB in one payment. A saving that was 30-fold at clone time came down to 1.3-fold once the repository was workable.

Three tables, three traps, and at the end the one combination that actually stays small. Plus a constraint in my own CI that I do not like.

The ground the measurements stand on

Environment: macOS (arm64), Homebrew git 2.48.1. Everything was taken against a single-branch bare copy derived from the repository's origin/main tip — df0de9d4, 22 October 2026:

$ git init --bare srv.git
$ git push srv.git origin/main:refs/heads/main
$ git -C srv.git config uploadpack.allowFilter true
$ git -C srv.git rev-list --count HEAD      # 3538
$ git -C srv.git count-objects -v | grep in-pack   # in-pack: 46219

The single branch matters: rev-list --count HEAD and --all give the same number, and all 46,219 objects in the pack are reachable. That removes the question "which ref set does this number come from".

There is also the matter of the unit. The first round used du -s, and APFS told a lie: two pack files of 289,923,968 bytes with identical md5 sums occupied 566,264 and 590,824 512-byte blocks in two different clones. Same content, 4.3 percent apart. So every number below is not du but the byte sum of the pack files — unchanged by whatever the filesystem decides to do, and the quantity we actually care about.

The first two traps: the filter drops out quietly

The first attempt at a filtered clone gave a local path:

$ LC_ALL=C git clone --filter=blob:none --no-checkout /tmp/lab/srv.git a1
Cloning into 'a1'...
warning: --filter is ignored in local clones; use file:// instead.
done.
$ echo $?
0

Adding the file:// prefix fixed that. Then the second road: right prefix, but the server setting off.

$ git -C srv.git config --unset uploadpack.allowFilter
$ LC_ALL=C git clone --filter=blob:none --no-checkout "file:///tmp/lab/srv.git" a2
Cloning into 'a2'...
warning: filtering not recognized by server, ignoring
$ echo $?
0

Both warn, both return zero, and both produce a full clone — 21,213 blobs, 276 MiB. Filtering has two separate gates, and with either one shut the command still counts as a success. The server-side gate is uploadpack.allowFilter; its documentation says what the setting does ("if this option is set, upload-pack will support partial clone and partial fetch object filtering") but not what happens when it is unset. The measurement says: the request is not refused, it falls back to a full clone without a sound.

The genuinely unpleasant part is common to both. The filter was not applied, yet Git still wrote this into the config:

$ git -C a2 config --local --list | grep -E 'promisor|partialclone'
remote.origin.promisor=true
remote.origin.partialclonefilter=blob:none

The repository is complete but labelled a partial clone, and later fetches honour the label. In the file://-with-the-setting-off case Git also left a .promisor file behind. That file is not a symptom — every partial clone has one. The oddity here is that the pack behind it is complete.

So how do you spot it? The two indicators that come to mind first both turn out to be useless: git config remote.origin.promisor returns true in a successful partial clone too, and the warning: This repository uses promisor remotes. Some objects may not be loaded. line that appears on commands touching the object store shows up in that clone too. Both tell you the repository is labelled partial; your question is whether the filter was applied. The one thing that separates them is counting blobs:

$ git -C a2 cat-file --batch-all-objects --batch-check | awk '{print $2}' | sort | uniq -c
21213 blob
 3538 commit
21468 tree

If you were expecting zero blobs, that table answers the question: 21,213 in the degraded clone, 0 in the successful one.

The clone ladder

All taken with --no-checkout, so the only thing compared is the set of downloaded objects. The server's own pack is 289,908,372 bytes.

clone	time	pack fetched	objects
full	2.9 s	289,923,968 B — 276.5 MiB	21,213 blobs · 3,538 commits · 21,468 trees
`--depth 1`	2.0 s	214,235,359 B — 204.3 MiB	6,321 blobs · 1 commit · 84 trees
`--filter=blob:limit=10k`	0.5 s	14,034,161 B — 13.4 MiB	8,371 blobs · 3,538 commits · 21,468 trees
`--filter=blob:none`	0.4 s	9,656,017 B — 9.2 MiB	0 blobs · 3,538 commits · 21,468 trees
`--filter=tree:0`	0.1 s	1,086,548 B — 1.0 MiB	3,538 commits only

Why the shallow clone is so large sits in the blob column: 6,321 blobs. --depth 1 cuts the history but brings the complete photograph of HEAD — every file as it stands today, cover PNGs included. Dropping the history takes 276.5 down to 204.3, twenty-six percent. Dropping the blobs takes it to 9.2.

The tree:0 row is instructive too. The local man page (2.48.1) says of tree:<depth>: it omits all blobs and trees whose depth from the root tree is greater than or equal to <depth>, and <depth>=0 will not include any trees or blobs unless given explicitly on the command line. That is exactly the table — 3,538 commits, nothing else, 1 MiB.

blob:limit=10k is there because its documentation says it omits blobs of size at least <n> bytes, meaning it keeps the small ones. Your text files are local, the PNGs are remote: 13.4 MiB. A reasonable middle road for anyone who wants the code but not the assets.

The loan: the first checkout

Starting work in the blobless clone changes the table. Pack bytes were re-totalled after each command:

command	time	fetched
(clone)	0.4 s	9,656,017 B — 9.2 MiB
`git checkout`	3.6 s	+214,046,263 B — 204.1 MiB
`git log --oneline` (3,538 lines)	0.0 s	0 B
`git log -p -1`	0.1 s	+15,379 B
`git blame README.md`	0.2 s	+12,571 B
`git log -p -20`	0.9 s	+1,653,754 B — 1.6 MiB

Everything after the first row is good news: walking the history is free. Listing 3,538 commits fetches not one byte, because the commits and the trees are already here. Ask for a patch and Git goes for that commit's blobs; twenty commits' worth of patches costs 1.6 MiB. For someone who says "I want to look at the past, but not at every version of every file", the model fits.

The bad news is the second row. git checkout alone fetched 204.1 MiB. The clone was 9.2 MiB, the working repository is 213.3 MiB, a full clone is 276.5 MiB. The saving that was 30-fold at clone time is now 1.3-fold.

Why is in the partial clone documentation, and the mechanism was fixed by a patch: checkout (and any other command using unpack-trees) has been taught to bulk pre-fetch all required missing blobs in a single batch. Without it things would be worse; the same document says dynamic object fetching tends to be slow as objects are fetched one at a time, and among the known limitations it lists that dynamic fetching invokes fetch-pack once for each item, which may incur significant overhead — and multiple authentication requests — if many objects are needed. The batch pre-fetch is why it finished in 3.6 seconds.

That one-at-a-time fetching leaves a trace on disk. After the five commands above the repository held 27 pack files, every one of them marked promisor: each lazy fetch leaves its own pack. I was about to write something here, and the measurement stopped me. The partial clone design document says gc does "not touch promisor packfiles at all"; but the same document's "Current Limitations" section says repack treats promisor and non-promisor packfiles as two distinct partitions and does not mix them. The document contradicts itself: the first sentence has gone stale, the second matches both git-repack's 2.48.1 man page and today's behaviour. The measurement backs the second — running git gc took those 27 packs down to a single promisor pack. That consolidation came from the gc I ran by hand; at 27, below gc.autoPackLimit's default of 50, automatic maintenance would not have fired yet. In a long-lived copy, once the threshold is crossed, it tidies up on its own. For the record: that one sentence is the stale part. The batch pre-fetch, the per-object fetch-pack and the online requirement I take from the same document all agree with the man pages and with the measurements.

In the treeless clone the ratio is wilder still: a 1,086,548-byte repository climbs to 215,334,800 bytes after checkout — 198-fold. In the blobless clone that multiplier is 23. The absolute figure is smaller than the blobless clone's 213.3 MiB (205.4), but it shows well how little the number you see at clone time tells you.

The rule that falls out: --filter has deferred the download, not reduced it. If you are going to open the whole working tree, you pay the loan at the first checkout.

The shallow clone's bill is not in size, it is in correctness

Size is not the shallow clone's only problem:

$ git -C shallow rev-list --count HEAD
1
$ git -C blobless rev-list --count HEAD
3538

The number dropping was expected. What is uncomfortable is the same line of the same file across the two clones:

$ git -C shallow blame README.md | head -1
^df0de9d (Social Bot 2026-10-22 07:15:19 +0000   1) # Mustafa Erbay — Blog & Teknoloji
$ git -C blobless blame README.md | head -1
d263086f6 (mustafaerbay 2026-06-09 20:55:09 +0300   1) # Mustafa Erbay — Blog & Teknoloji

git blame does not fail. It runs, it produces output, and it attributes that line to the commit a social-posting bot pushed that same morning. The true answer is 9 June, a different hand. The leading ^ marks a boundary commit, but to an eye not trained on that table this is simply a wrong date and a wrong name. A slow tool is liveable; a wrong one is not.

One more side effect is in the documentation: --depth implies --single-branch unless --no-single-branch is given. For CI that is usually right; but the answer to "why does git fetch not see that branch" lands here months later.

The combination that actually stays small

The only way not to pay the loan is never to open the whole working tree. git clone carries --sparse for this, and its documentation says word for word what it does: it employs a sparse-checkout, with only files in the toplevel directory initially being present.

setup	pack	working tree	files
full clone + checkout	276.5 MiB	232.8 MiB	6,336
full clone + `sparse-checkout set`	276.5 MiB	15.9 MiB	1,112
`--filter=blob:none --sparse` (at clone)	9.3 MiB	0.5 MiB	14
+ `sparse-checkout set` (3 dirs)	16.7 MiB	15.9 MiB	1,112

The second row explains why sparse-checkout does not deliver what people expect. The working tree went from 232.8 MiB to 15.9 — but the pack stayed at 276.5 MiB, not a byte moved. sparse-checkout does not shrink the download, and the command's own tagline promises no more than that: it reduces your working tree to a subset of tracked files. The mechanism is described in the documentation's INTERNALS section and has nothing to do with the object store: the skip-worktree bit goes into the index, Git avoids populating those files' contents, and their absence is ignored. The index entry stays put — as in the lab: in a working tree with 14 files, git ls-files still counts 6,336 paths.

The last row is the whole point: a 32.6 MiB setup holding all the articles of one category plus src/lib and scripts. The full setup is 509.3 MiB; less than one fifteenth. And the history is still there. The recipe is three lines:

$ git clone --filter=blob:none --sparse "file://$SRV" work
$ cd work
$ git sparse-checkout set src/content/blog/tutorials src/lib scripts

After the first line you will have 14 files; until the cone widens, the repository is just its root directory. The "it came up empty" panic is normal.

The sparse index does not come unless you ask

Switching to cone mode was supposed to shrink the index too. It did not:

$ git sparse-checkout set src/lib
$ git config index.sparse ; stat -f%z .git/index
                       # no line printed
938361

The documentation says so explicitly: the --[no-]sparse-index option can be used to get a sparse index, and the default is not to use it. With the flag, 938,361 bytes drop to 5,839. In a wider three-directory cone, 936,257 → 162,430 bytes, a factor of 5.8.

On the speed side there was nothing, and that goes down as it is. git status --porcelain ran five times each; best values: 0.040 s with the sparse index, 0.041 s with the full one. The difference sits inside the noise. The documentation says this can have significant performance advantages for commands such as git status or git add; in a repository with 6,336 paths it does not show. This setting's promise is for monorepos with tens of thousands of files. The same page says the feature is still experimental and that some commands might be slower with a sparse index, and the capitalised "this command is experimental" warning sits at the end of the DESCRIPTION section.

One more trap: if files are left outside the cone, Git warns during git status that the sparse index is expanding to a full index, a slow operation — and that run's numbers are garbage. My first attempt landed exactly there.

The constraint in my CI that I do not like

All of this was for my own publishing pipeline. Of 17 workflow files, 7 use actions/checkout; 2 of those declare fetch-depth (one 1, one 0) and the remaining 5 take the default. The pinned version's action.yml gives 1 as that default, described as "Number of commits to fetch. 0 indicates all history for all branches and tags."

So my pipeline implements my reflex: a shallow clone. Per table 1 that is a 204 MiB photograph per run — with the caveat that the 204 MiB was taken on the local file:// side; GitHub's upload-pack packs differently and actions/checkout does not run git clone, it fetches a single ref. It gives the order of magnitude, not the bill. The workflow that says fetch-depth: 0 pays at least 276 MiB — the server in the table had a single branch, while fetch-depth: 0 asks for all branches and tags.

The action also has filter and sparse-checkout inputs — precisely the winning combination above. But the filter input's description is two sentences, and the second is this: it overrides sparse-checkout if set. The two cannot be used together in this action. The smallest setup in this article — blobless clone and a narrow cone — cannot be expressed with actions/checkout's inputs. Either take the 9 MiB clone and pay the loan by opening the full tree, or ask for the narrow tree and download the full object set.

There is a way out, and it is not cheap: use actions/checkout only to place credentials (persist-credentials), then run git clone --filter=blob:none --sparse yourself. The price is that credential handling becomes your problem — and since every later log -p/blame in a partial clone is a network request, that credential has to outlive the checkout step. I have not done it in my pipeline yet; the measurement is on the table, the constraint is named, the decision is pending.

Before you apply any of this

Is history needed? git log, git blame, git bisect. If yes, --depth 1 is off the table: it gives a wrong answer without erroring. If no, it is the simplest road — provided your repository's weight is in its history.
Where is the weight? Of my repository's 276 MiB, 204 is today's photograph. Measure once: clone with --depth 1 --no-checkout and look at the pack bytes. If the photograph is large, cutting history will not help.
Is the whole working tree needed? If yes, --filter defers the download, it does not reduce it. If no, --filter=blob:none --sparse is the only real win.
Will you work offline? The documentation is explicit: use of partial clone requires that the user be online and the origin remote or other promisor remotes be available. When the promisor is unreachable a partial clone does not get slow: commands that make do with local objects (status, commit, log --oneline) keep working, while every command needing a missing object fails — and you cannot repair that offline.
Does the server allow it? uploadpack.allowFilter is not the only gate: uploadpackfilter.<filter>.allow can ban individual filters and uploadpackfilter.tree.maxDepth caps tree:<n>. blob:none may work while tree:0 is refused. On your own Git server this is the first place to look.

Three things to avoid: asking for a filter without file://, asking for one while allowFilter is off on the server (both quietly give you a full clone), and assuming cone mode brings the sparse index along. None of the three breaks anything; they just do not do what you asked, and the first two exit zero while doing it. A two-line assert in a CI job catches those two — but not via the promisor setting, which is true in a successful clone as well. After the clone, count the blobs, or test the pack bytes against a threshold.

A deferred cost is not a reduced cost

What made me think most this round was the delight at first seeing that 9 MiB clone. The number was real, the measurement was correct, it said thirty-fold. The only wrong thing was the conclusion drawn from it: "the repository got smaller". It had not; its bill had been postponed, and it came back the moment I typed git checkout.

The second lesson was in the measuring itself. The first round used du, and showed a 4 percent gap between two copies of the same pack — until it became clear the filesystem was reporting on its own block accounting, not on the repository. In a measurement article, picking the unit is as much work as taking the measurement.

Other instances of the same reflex: the article where Git wrote this repo half a commit-graph, where a correct and fresh cache was not speeding up the question actually being asked, and the one where git status walked 36,811 entries for three lines.

A five-minute exercise on your own repository: take a blobless clone, note the pack bytes, run git checkout, note them again. The gap between those two numbers is the loan you deferred.

Official Sources

My Five TODOs Are 160 Days Old; Flutter Wrote the Two That Closed

Mustafa ERBAY — Thu, 22 Oct 2026 07:14:39 +0000

Every real TODO marker standing in my repositories was written on a single morning: 15 May 2026.
Four of them at 09:48:57 in one commit, the fifth seven minutes and thirty-six seconds later.
Today is that morning's one hundred and sixtieth day, and all five are still in place.

That wasn't the real story, though. I counted the history as well: across the entire history of
my five repos, 21 lines carrying a marker were added and 5 were removed. Among those five
removals there are two that genuinely closed, in the sense that the work they described got done
— and I wrote neither of them. Both came out of Flutter's project template.

What I scanned, and what I left out

Let me state the scope up front, because numbers like these are meaningless without one. I
scanned only my own projects: this blog's repository, köprü, vpsman, burncpu and gercekveri. I
never scanned the client repositories at all. Across these five, git tracks 8,090 files in total;
after the exclusions, the set I scanned is 2,021 files. On the history side there are 4,417
commits, merge commits excluded. I ran the measurement not in my local working copies but in
clean trees checked out from each repo's origin/main — a branch sitting a day behind will
quietly shift every denominator you publish.

The scanner is simple: it looks for the words TODO, FIXME, HACK and XXX on every line of
every tracked text file. What I left out: node_modules, dist, lock files, compiled assets and
third-party vendor directories. In the blog repo three more directories are excluded —
src/content/, public/ and src/assets/ — because that is published prose and assets, not
code. I counted that separately out of curiosity: of the 3,119 files under src/content/, 14
contain these words, on 16 lines in total. So what I excluded is not some vast pile; it is about
the same size as the 15 markers in my code. The reason for excluding it isn't the count, it's the
kind: the word "TODO" in an article is the subject matter, not technical debt.

Before trusting the scanner I tried to catch it lying. I wrote a small file holding three markers
that had to be found and three traps that had to be ignored: a todos variable, an autodoc
function, a TODO_LIST constant. It found the three and skipped the three. Skipping this step is
very tempting, but a text-scanning script has a habit of manufacturing its own findings.

Five of the fifteen hits are real

The raw result is 15 lines. Reading them by hand split them into three groups.

Six are placeholders: urn:li:person:XXX, token=XXX, ADR-XXX, and a me-veo-XXX inside a
filename template. None of them is debt; each is shorthand for "your value goes here."

Two are my own joke about TODOs. In the punchline list I wrote for the shorts pipeline, entry 26
reads: "WROTE A TODO. 2022. STILL TODO." The scanner reported my joke about technical debt as
technical debt — a measuring instrument that cannot recognise its own subject is a brutal summary
of everything that follows.

One isn't even code: a note in a speaker-proposal draft telling me to get a professional
photograph taken. And one is a line I never wrote — the stock TODO Flutter generates in the
CMakeLists.txt on vpsman's Windows side.

The remaining five are real, and all from one family: the four Remotion templates of the
shorts/video pipeline, plus generate-short.ts. All five open with the same phrase,
// TODO Phase 1:.

One small reconciliation gap turned up here. In the köprü repo the history showed two markers
while today's scan found one. I went looking for a deleted line; there wasn't one. The second
marker sits in a third-party CSS file under src/web/public/vendor/, and my scanner excludes
vendor directories by design. Both numbers are right — they answer different questions. Waving
a gap like that away as rounding is the easiest way to break a measurement.

Someone wrote this trap down in 2005

Placeholder XXXs inflating a scan is not a new discovery. PEP 350, "Codetags",
written in the Python community in 2005, discusses this exact word: it gives markers like TODO,
FIXME, HACK and BUG the collective name codetag, and it does not treat XXX as a
canonical tag — it lists it among the synonyms of FIXME. One of its reasons describes my six
false positives precisely: XXX is a useful placeholder where a value is unknown, which is why
FIXME is the preferred spelling.

The PEP's real proposal is more ambitious: put the answers to who wrote this, on what date, with
what due date and what priority next to every marker. Its suggested form looks like
<MDE 2005-09-04 d:14w p:2>. My five TODOs carry none of those fields — no date, no priority, no
owner. The odd part is that the PEP itself was never adopted; its page opens with "This PEP has
been rejected," on the grounds that there was no desire to make the standard library conform to
it. The industry has known about writing a date next to the marker for two decades, and has
declined to do it for two decades. So have I.

Three scanners, three different numbers

I would like to treat my own grep as the measurement, but the result changed beyond recognition
as I moved the settings.

First I turned off case sensitivity: 15 hits became 30. All 15 newcomers were noise —
ADMIN_API_TOKEN=xxx examples, a const todo = rows.filter(...) variable, some xxx-form.tsx
filename templates in a CLAUDE.md. Then I made the term list resemble Visual Studio's task list;
in that environment HACK, TODO, UNDONE and UnresolvedMergeConflict are predefined, and the
documentation states those token labels are case insensitive. Adding UNDONE took it to 31: one
new line, the phrase "escapes are undone" in a Rust comment. I changed the two settings in two
separate steps; flipping both at once and then blaming case sensitivity alone would have been
slander against my own measurement.

Then I went to the other end of the scale. ESLint's no-warning-comments rule exists to do
exactly this job, and its documented defaults read as follows: the terms searched for are
["todo", "fixme", "xxx"], matching is case-insensitive and whole-word, and the location is
"start" — the term must sit at the beginning of the comment. In the rule's own source the
recommended field is false, so it ships switched off in every preset.

I couldn't run that rule, because ESLint isn't installed in the blog repo where the five TODOs
live. Three of my repos do have it — two mobile sub-apps and the root of gercekveri — but none of
the markers sits within those three. So rather than the rule, I rebuilt its documented default in
a script of my own and limited it to the file family ESLint would look at: 831 JS/TS files, 5
reports. The same five real TODOs.

This does not mean the tool is smarter than me; part of why ESLint looks precise is that its scope
is narrow. It never looks at markdown, CSS, Rust or Gradle files, and most of my noise was sitting
in exactly those. Visual Studio's four tokens are themselves defined for C/C++, C# and Visual
Basic, while I applied them to every file type. Still, one lesson is clean: requiring the marker
to sit at the start of the comment wipes out the whole placeholder class on its own.

All of them inside seven and a half minutes

I asked git blame for the age of the five. The four templates were written at 09:48:57 in a
single commit, the fifth — generate-short.ts — at 09:56:33. Seven minutes and thirty-six seconds
between them. Today is 22 October; 160 days have passed since that morning, and to these five
standing markers nothing has been added and nothing has closed since. (The rest of the history
does move; I'll count that shortly.)

The date says something on its own. Those two commits are the commits in which the shorts pipeline
was born. The moment I wrote the markers was the moment the pipeline first ran, and I put the
intention to prettify it into the code as a note. The minute you finish the first version of
something is the minute you hold the most optimistic possible estimate of your future energy.

Did "Phase 1" happen somewhere else instead? No — and rather than assume it, I went and counted.
The contents of the four marker-bearing templates have not changed since the day they were born;
after the two commits of that morning, the only commit to land in remotion/ is a version bump on
28 September. Meanwhile the pipeline is live: the repo holds 20 content files for 10 short videos
across Turkish and English, and the templates are called from Composition.tsx. What I left
undone never stopped the pipeline from working. Which is the real finding: those markers were not
a record of debt, they were a wish.

Twenty-one markers, five deletions

The current state is a snapshot; the history was what interested me. Across the full history of
the five repos I counted lines containing a marker being added and removed: 21 lines added, 5
removed. The honest denominator here is 21, not the commit count — a line that was never written
cannot be deleted. Opening those five deletions one at a time turned out to be the whole point.

Two are genuine closures, and I wrote neither: the build.gradle.kts file on vpsman's Android
side carried two stock TODOs generated by Flutter — "specify your own unique application ID" and
"add your own signing config for the release build." I really did both; one on 8 May, the other on
29 August.

One is fake. On 1 June I rewrote the header comment of unsubscribe.ts; the same commit deleted
one line and added a near-identical one. The counters show an add and a delete while nothing
closed. If you don't filter out add-and-delete pairs inside a single commit, this measurement
inflates in both directions at once.

One fell off. On the morning of 16 May at 10:43 I wrote a line into burncpu's README: "local
Docker: (TODO)". At 11:14, thirty-one minutes later, a commit reorganising the README took that
line with it. The marker went — so the work was done? No: the local Docker instructions landed on
14 July. There are 59 days between the death of the marker and the completion of the work, and no
causal link whatsoever between them.

One was solved by another route. On 15 May I established that video upload to X wasn't working and
wrote this next to it: "or implement Twitter v1.1 chunked upload directly (TODO)". On 11 June I
fixed the problem — but not that way. I wrote a patch for the library I was using, adding the
missing parameters and the status polling. When the fix arrived, the whole paragraph containing
the TODO was deleted. The problem closed; the plan the marker proposed was never carried out.

The denominator is small, so let me put it on the table. Eight TODOs written by my own hand mean
actual work: the five still standing, the photograph note in the speaker proposal, burncpu's README
line, and the X upload line. Only two of them have ever been deleted — the last two cases. So the
sentence "no TODO I wrote was ever closed by doing what it said" rests on two deletions; the other
six are still standing and haven't had their chance yet.

What the two closed TODOs have in common

To see whether a mechanism falls out of this, I counted how many commits touched the files
carrying the markers. The four templates that hold four of the five living TODOs have been touched
by one commit each since birth — the commit that created them. The fifth, generate-short.ts, got
two, both on the same day (one at 09:56, the other at 13:33). In those same 160 days that repo
took 2,759 commits, 1,208 of them under scripts/. The repo was working like mad; those five
files were never opened.

The file of the TODO that closed lives a different life: build.gradle.kts has taken four
commits, the most recent on 12 September. My first explanation was this: what gets a comment read
is not good intentions, it's the file being reopened for an unrelated reason.

But to be honest, three explanations come out of this data and two closures can't tell me which is
right. The second: at least one of the closed items blocked a path — a signing configuration is
required to ship to a store. (I can't claim the same for the application ID; I may have done that
because its turn came, not because it blocked me.) The third, and perhaps the strongest: both
closed markers were generated by a tool, and both sat immediately above a line I had to change
anyway while preparing that file for release. All three land in the same place: there is no such
thing as a self-reading comment.

The ledger lives outside the code

So where do my open items sit? If there are five lines in the code, where is the rest? I looked at
the notebook belonging to this project: 57 files, holding 22 lines that say "open item," "not
fixed yet," "waiting" or "remaining steps," spread across 12 of them. Those 12 files also carry 84
lines bearing a warning mark; across the whole notebook that mark appears on 116 lines, and most
of them are traps rather than open work. (That is one project's notebook alone; across every
project notebook on the machine there are 497 note files.)

Here's the interesting part: PEP 350's author argues the exact opposite. Answering objections, he
says he would rather have the information in one place — the source code — than in various other
documents likely to get duplicated or forgotten. My measurement partly inverts that sentence: the
thing that got forgotten was the marker in the source code. But his deeper point survives, because
what keeps my notebook alive isn't its format either — it's the habit of returning to it every
session. What makes a ledger a ledger is its reader.

When I audited 1,973 runs of my scheduled jobs
I found that no record was being kept. Here a record is kept but has no reader — the same
blindness I saw when I counted the scripts nobody calls.
In all three the mistake sits in the same place: I took a thing's existence for its working.

What I'm changing

The conclusion of this measurement is not "stop writing TODOs." All five markers told me
something; this article exists because I read them. What I'm changing is the question I ask while
writing one.

If I'm about to write a TODO, I now ask first: what force will get this line read? Will the file be
reopened for another reason, or does the work block a gate? If either is true, the marker earns
its place. If neither is, what I'm writing is a wish, and there is no point hiding a wish inside
the code.

Second: I put the marker at the start of the comment — one line of rule, and the entire
placeholder noise class is gone. Third: no undated TODOs. I don't need to make PEP 350's rejected
field list mandatory, but dropping a date next to the marker stops a hundred and sixty days from
passing unnoticed; today I had to run git blame to count those days.

I'll also keep my open-item ledger in prose, because the measurement showed that is the side which
actually gets read. I'll just give it the name it has earned: that is my issue tracker. The marker
inside the code is not a reminder, it's a decision — and writing a decision in the costume of a
comment is the polite way of not making it.

Official Sources

My Firewall Is Off, 42 of 55 Sockets Unreachable from the Network

Mustafa ERBAY — Wed, 21 Oct 2026 15:04:41 +0000

The question was as harmless as a coffee break: is the firewall on this machine
enabled? The known answer runs through a plist file: read the globalstate key,
0 means off, 1 means on. It is settled enough practice that Apple's own release
notes speak to workflows built on modifying that file. I typed it into the
terminal and got back "no such file".

The file is not on this machine. Apple took the settings out of the plist with
macOS 15; exactly when the file itself disappeared is something neither the
release note states nor I can measure. More interesting still, when I asked the
same question in three different places I got three different answers. And in
the end this came out: the question I was asking was wrong too.

Every measurement was taken on the evening of 21 October 2026, on a single
machine: macOS 26.6.2 (build 25G83), arm64, no MDM enrollment
(profiles status → "MDM enrollment: No"). Every command is read-only; I did
not change a single setting on this machine for this article. The numbers are a
snapshot and they drift as the machine runs — what matters is not the
individual figures but the ratio between them.

The file is gone, the tool still answers

The known path is /Library/Preferences/com.apple.alf.plist.
"alf" stands for Application Firewall — the long-standing internal name of
macOS's application-based firewall. I asked about the file:

$ stat /Library/Preferences/com.apple.alf.plist
stat: /Library/Preferences/com.apple.alf.plist: stat: No such file or directory

(Yes, stat: is printed twice — that is BSD stat's own habit, and you will
see it too.) The detail here matters: the error is not "Operation not permitted", it is "No
such file or directory". On macOS, trying to read a protected file gives you a
permission error; a missing file is a different message. So this is not an
access barrier — the file genuinely is not there.

What does the tool say, then? The binary carrying the same name is still in
place and — the part that surprised me — it answers without sudo:

$ /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
Firewall is disabled. (State = 0)
$ ... --getblockall
Firewall has block all state set to disabled.
$ ... --getstealthmode
Firewall stealth mode is off
$ ... --getallowsigned
Automatically allow built-in signed software ENABLED.
Automatically allow downloaded signed software ENABLED.

Four questions, four answers, no root. That is a small security note in its own
right: a process running on my machine with ordinary user privileges can read
the state of my firewall without being asked for a single permission.

Apple wrote down where the gap comes from, in its own release notes. Under the
"Application Firewall" heading of the macOS Sequoia 15 developer notes, in the
"Deprecations" section, there is a single item, and it names the file exactly:
the settings are no longer held in a property list, and any app or workflow that
worked by modifying that file needs to move to the socketfilterfw command-line
tool.

So the check in circulation is not stale, it is retired. And the only way to
find that out is to ask for the file one more time; nobody sends you a notice.

So where does the state live?

I wanted to chase this, because in a security review "the tool says so" is not
a sufficient answer; to verify something you need to know where you are
looking. I scanned the strings inside the binary: socketfilterfw carries two
plist paths — one is the factory template I will open in a moment, the other is
the no-longer-existing /Library/Preferences/com.apple.alf.plist. A path
occurring inside a binary does not mean the binary reads it; but a retired path
still sitting there shows the distance between the tool's history and its
present.

In the places I can read, I could not find the new location: there is no
firewall-related file under /Library/Preferences, /Library/Managed Preferences is empty, and defaults read com.apple.alf and
defaults read com.apple.security.firewall do not come back empty — they come
back with an error: Domain com.apple.security.firewall does not exist.
pfctl -s info wants root; macOS does ship pf, a real packet filter, but that
is another evening's subject.

What ended the search was an answer from an Apple engineer. In a developer forum
thread asking exactly my question
(thread 760135), the reply
from Apple's Developer Technical Support team is short: the location and format
of these preferences is not considered API, so it will not be shared; and there
is no other known way to observe the setting. The developer who asked pressed
on: the release notes point at socketfilterfw, yet the same support team had
said elsewhere that socketfilterfw is not API either — so which is the proper
way? That message got no reply.

Keeping the state of a security control somewhere its own documentation never
names is a technical choice. But the practical consequence is this: you cannot
audit that control, you can only trust it. Those are not the same thing.

The template is not my machine

This was the part most likely to mislead me. The file is absent under
/Library/Preferences — but a file with the same name sits in the
/usr/libexec/ApplicationFirewall/ directory, 4,279 bytes, the factory
template that ships with the system (its version key reads 1.7). I opened
it, and the contents looked exactly like the answer I had got from the tool:
globalstate 0, stealthenabled 0, allowsignedenabled 1,
allowdownloadsignedenabled 1.

I could have stopped there and said "so the live state is in this file". It
would have been wrong, and the template itself says so. Its applications
array is completely empty. Yet the tool lists seven applications:

$ /usr/libexec/ApplicationFirewall/socketfilterfw --listapps
Total number of apps = 7
1 : /usr/libexec/remoted            (Allow incoming connections)
2 : /usr/bin/python3                (Allow incoming connections)
3 : /usr/bin/ruby                   (Allow incoming connections)
4 : /usr/sbin/cupsd                 (Allow incoming connections)
5 : /usr/libexec/sharingd           (Allow incoming connections)
6 : /usr/libexec/sshd-keygen-wrapper (Allow incoming connections)
7 : /usr/sbin/smbd                  (Allow incoming connections)

I compared those seven paths against every list inside the template, one by
one. The template has three separate places: 10 paths under exceptions (all
carrying state: 3 — what that value means is not documented by Apple; the name
suggests an exemption, but I could not verify it), 7 bundle identifiers under
explicitauths (python, ruby, php, nc, ksh, a2p, java), and 10 named sharing
services. Of the seven paths the tool returned, only three (cupsd,
sshd-keygen-wrapper, smbd) match service names in the template. python3
and ruby appear in the template not as paths but as bundle identifiers inside
explicitauths — different form, different list. remoted and sharingd
appear in none of them.

Do not run this comparison with a plain grep: remoted occurs inside the
template's xartstorageremoted path, and sharingd occurs inside
screensharingd. Two words, two false positives, one wrong conclusion. Switch
to full-path matching and both disappear.

The conclusion: the tool is handing me a list it does not read from the
template. So there is live state, and I cannot read it. The template's numbers
agreeing with the tool's answer is not a confirmation but a coincidence — both
may simply be reporting Apple's default.

What "on" does not mean

This is where I got the real lesson. All this time I had been chasing the on/off
question, when that switch says far less than I assumed.

This macOS firewall is not port-based but application-based: it governs not
which port is closed but which program may accept incoming connections. And it
has two switches, both on by default. Apple's own device management
documentation states it plainly: for the AllowSigned and AllowSignedApp keys
in the com.apple.security.firewall payload, "the system ensures that the key
always has a value; if missing from the payload, the system sets it to true".
The first covers built-in signed software, the second downloaded signed
software.

Apple's user guide, on the page describing the settings, says the same thing in
everyday language: apps and services signed by a valid certificate authority are
added to the list of allowed apps without your authorization. An "Important"
note on the guide's companion page — the one describing the firewall itself —
completes the picture:
some applications that have access never appear in the list at all — system
apps, services, processes, and signed programs launched automatically by other
applications. If you want to block those, you have to add them to the list by
hand.

Translate that to my machine. What would happen if I turned the firewall on?
Built-in signed software would go on accepting incoming connections. So would
the signed software I downloaded. What remains — unsigned programs, ones I
blocked by hand, or ones it would ask me about — is a small set. So the "on"
position is not the wall I had imagined; it is more like a doorkeeper who holds
the door for anyone carrying a signature.

There is stealth mode too: in Apple's words, the machine not responding to
probing requests that could reveal its existence. Mine is off. It is useful,
but not answering probes is not the same as not listening, and those two are
easy to conflate.

The real number is in the bind address

I changed the question. Instead of "is the firewall on?": how many sockets on
this machine are waiting for incoming connections, and can they be reached from
the network?

$ netstat -an -p tcp | awk '$NF=="LISTEN"{print $4}' | sed 's/\.[0-9]*$//' | sort | uniq -c
  13 *
  37 127.0.0.1
   5 ::1

Fifty-five listening sockets, 46 distinct ports. Thirty-seven bind to
127.0.0.1, five to ::1 — 42 sockets in total, across 39 distinct ports.
Thirteen sockets bind to *, that is to all interfaces; those come to 7
distinct ports (six of them open one row each on IPv4 and IPv6, one opens a
single row). There is no overlap between the two sets.

The meaning of the number 42 is independent of the firewall, and its source is
not a setting but a protocol rule. Section 3.2.1.3 of RFC 1122, listing the
special address forms, has a single sentence for loopback: addresses of this
form must not appear outside a host — in the item's own words, "MUST NOT appear
outside a host". On the IPv6 side the rule is more explicit: Section 2.5.3 of
RFC 4291 says a packet whose destination address is loopback must never be sent
outside of a single node and must never be forwarded by a router, and that "a
packet received on an interface with a destination address of loopback must be
dropped". RFC 6890, entering both blocks into the special-purpose address
registry, writes False into four of their attributes: cannot be a source,
cannot be a destination, not forwardable, not global. The one caveat is the
footnote under the IPv4 table — a few protocols have been granted exceptions to
that rule; the ::1/128 row carries no such footnote.

Let me be explicit about one thing: the number 42 is not a measurement, it is a
derivation. The command reports the bind address of each socket; the
unreachability follows from that address and from the rule above.

In practice that means this: for a process listening on 127.0.0.1:5432, the
firewall setting is irrelevant. A packet arriving from the network cannot reach
that socket, because a packet carrying that address is not forwarded on the
network — that is what the rule says, and what routers do.
That is the situation for 42 of my 55 sockets. My firewall is off, and for those
42 it makes no difference.

The remaining 7 ports are my surface toward the local network — not toward the
internet, which I will come to shortly. I am not writing the port numbers here;
the service names in the --listapps output above already hint at them, and
adding the numbers gives the reader nothing. The useful part is not my list but
your own. Let me share the question I asked myself: for each of these seven
services, "does it need to be bound to all interfaces, or was it switched on
once and left there?"

The number 42 can fool you

On its own, the sentence "it is bound to loopback, so it is safe" is lazy. Let
me write down the objections.

A socket bound to loopback cannot be reached from the network, true. But put a
proxy bound to all interfaces in front of it and it becomes reachable. On a
developer machine running containers this is not an exception but the normal
arrangement: the service inside the container listens on its own loopback, a
port forwarder on the host side binds to *, and the two are joined.
netstat tells you the truth — but you only see the whole truth if you know
which process is holding each socket at that moment.

Second objection — the question I left hanging above: being bound to * does
not mean "open to the internet", it means "reachable from the network I am on".
There is a router in between, and I did not measure it anywhere in this article;
for most machines on a home or office network, NAT already meets an inbound
connection at the door. But IPv4 and IPv6 are not the same here: netstat
prints * in the address column for both — it keeps the distinction in the
Proto column (tcp4/tcp6/tcp46), and it is the one-liner above that throws
that column away. Yet if you hold a global IPv6 address there is no NAT in front
of that socket at all; what covers you is whatever stateful filter your router
keeps for IPv6 — most home routers default to blocking inbound there, but "most"
is not your device. One line, two different exposures. That is why the
* count is not an alarm but a list of questions.

Third objection: the number 42 holds for connections arriving from the network,
not for code running on the machine. Any process running here can connect to
127.0.0.1. "Loopback only" is a network boundary, not an authorization
boundary. It is no justification for leaving a locally running service without
authentication.

Fourth: I only counted TCP. UDP has no listening state — a socket is either bound
or it is not, and netstat's LISTEN column stays empty there. So this
measurement is not the whole surface, only the slice I was looking at.

Six commands, read-only

For anyone who wants to ask the same question of their own machine, here is the
sequence I verified working today — none of it changes anything, none of it needs
root:

/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate — on or off. If you have a note that reads the old plist path, throw it away.
... --getallowsigned — if two lines come back, read both. If you see "ENABLED", know that turning the firewall on will not stop signed software.
... --listapps — do not mistake the list for a complete one; Apple's note says some programs with access never appear here.
netstat -an -p tcp | awk '$NF=="LISTEN"{print $4}' | sed 's/\.[0-9]*$//' | sort | uniq -c — this is the real number. The figure on the * line is your surface toward the local network.
lsof -nP -iTCP -sTCP:LISTEN — it tells you who holds the sockets, the missing half of the previous step. Do not add its rows up against netstat's: without root it names only your own processes and other users' sockets drop off the list with no error (48 rows against 55 on this machine). lsof also prints per file descriptor where netstat prints per socket, so one socket can take two rows.
profiles status -type enrollment — if the machine is under MDM, you are not the owner of these settings; Apple's documented configuration surface is the com.apple.security.firewall profile payload.

Put steps four and five side by side and what is left is asking, one by one: why
is this port here? The first places to look are a sharing toggle (the Sharing
section of System Settings) and the service's own bind address; both can come up
empty, and then you are left searching by process name. Once you find it, writing
127.0.0.1 instead of * is a more definite move than turning the firewall on.
Every line whose answer is "I don't know" is the work of another evening.

And one more thing: because there is no supported way to observe the state, this
is not something you do once and forget. On a single machine the remedy is to
ask --getglobalstate on an interval and catch the change; across several
machines, hand ownership to a profile — if you enforce the
com.apple.security.firewall payload, at least you are the one keeping the
record.

What was not where I was looking

What I learned this evening is not that my firewall is off — the template that
ships with the system says globalstate 0 too, and for 42 of my sockets it
makes no difference. What I learned is that I can ask about the state of one
setting in three places and get three
different answers: the check in circulation points at a retired file, the
factory template is not my machine, and the list the tool hands me is one whose
incompleteness is written into its own documentation.

There is a shared fallacy underneath all of it: mistaking knowing the name of a
switch for knowing what the switch does. "The firewall is on" is a sentence;
"this socket is bound to 127.0.0.1" is a fact. The first points at a policy,
and the inside of that policy is full of defaults, signature exemptions and
invisible lists. The second rests on a protocol rule and depends on nobody's
default.

So the next time I wonder how open my machine is to the outside, my first
command will not be --getglobalstate. It will be netstat. The switch tells
me what I was trying to do; the bind address tells me what I did.

I have put similar questions to this machine before — there was not a single
address in the 1,945 rows of my download
ledger,
and I owned 146 directories I could not
open — and each
time what produced the answer was not my curiosity but finding the right
command.