DEV Community: Mustafa ERBAY

fsck Found the Draft That reflog Never Saw; Two Weeks on Paper

Mustafa ERBAY — Sun, 15 Nov 2026 14:26:15 +0000

Everyone who works with Git carries one reassuring sentence in their head: "it'll be in the reflog anyway." That sentence is as correct as it is incomplete. A reflog records where the tips of refs have moved — in other words, everything that happens after a piece of work becomes a commit. Work that disappears before it becomes a commit sits in a place the reflog never looks at, and how long it sits there depends on an entirely different setting.

So this week I sat down and put numbers on it, because there is a garbage collector running on my machine every morning at 04:30, and that collector deletes half-finished working directories. I wanted to know the exact distance between "it deletes them" and "data is gone."

Here is the current inventory in the blog repository:

$ /usr/bin/git count-objects -v
count: 1179
size: 30640
in-pack: 49845
packs: 21
size-pack: 336528
prune-packable: 34
garbage: 0
size-garbage: 0

49,845 packed objects and 3,675 commits on origin/main. Now the interesting part — how many of those objects are visible from no branch tip and no tag:

$ /usr/bin/git fsck --unreachable --no-progress | wc -l
2089
$ /usr/bin/git fsck --unreachable --no-reflogs --no-progress | wc -l
2161

One flag is the only difference between those two commands, and the gap between them is 72 objects. Those 72 are the work still counted as "protected" thanks to the reflog. Behind the remaining 2,089, there is nothing at all. By type:

reflogs as roots            with --no-reflogs
1723 blob                   1729 blob
 103 commit                  118 commit
 263 tree                    314 tree

1,723 blobs. Rejected drafts, earlier states of --amended commits, leftovers from deleted working directories. All of them on my disk, all of them readable, none of them with a name.

What the collector deletes, and what it doesn't

Some context: Claude Code agents open a git worktree under the repository for each task, and those pile up until the disk fills. At the end of September that pileup took my Mac's data volume to 100%. The collector I wrote that day runs under launchd every day at 04:30 and deletes a worktree that passes three gates: no tracked changes, not newer than two days, and no commits on HEAD that never reached a remote.

Its running total, from its own log:

$ grep -c 'SILINDI' ~/.claude/scripts/worktree-gc.log
176
$ grep 'SILINDI' ~/.claude/scripts/worktree-gc.log | grep -c mustafaerbay
29
$ grep OZET ~/.claude/scripts/worktree-gc.log | tail -1
2026-11-15 04:30:16 OZET silindi=4 korundu=29 kazanc=404MB bosalan=20GB dry=0

176 working directories deleted, 29 of them in this blog repository. In the comment at the top of the script I had written, in my own hand: "deleting is not data loss — the branch ref and the objects stay in the main repo." I did not want to run it for another year without testing how much of that sentence is true.

Three loss modes, three different answers

I set up three loss scenarios in a clean lab. Same repository, same minute, same Git version: /usr/bin/git 2.54.0 (Apple Git-157). To keep stray settings out of the results I pointed GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null — a line in my own ~/.gitconfig had no business skewing the numbers.

The scenarios: (1) write a file, never git add it, delete it. (2) git add it, don't commit, then take a git reset --hard. (3) commit it, then step back with git reset --hard HEAD~1.

A1 untracked : expected blob 9d8e786752d6084b87d0712679878e243c8f8f47 -> object present? NO
A2 staged    : blob 756d8d4457346ad95d7fb3d51222ece2352462b3 -> object present? YES
A3 commit    : commit d8c76e02b01d09bd2ab78033af99fd98e9fcea17 -> object present? YES

The first line settles the whole argument. A file you never git added leaves no trace on the Git side — no blob, no name, no size. git hash-object can compute what its name would be, but there is no object behind that name. The real form of the recovery question in Git is not "do I have a backup," it is "did this work ever become an object."

I had bumped into that sentence once before: when I set the assume-unchanged bit and found seven of thirteen commands ignoring it, I noted in a single line that the lost change had nothing in the reflog either. That was an aside then; this article is that aside, measured.

In the second and third scenarios the object is there. But does the reflog see them?

$ git reflog --date=iso
2c996a2 HEAD@{...}: reset: moving to HEAD~1
d8c76e0 HEAD@{...}: commit: silinecek commit
2c996a2 HEAD@{...}: reset: moving to HEAD
2c996a2 HEAD@{...}: commit (initial): ilk commit

The third scenario's commit is right there on the second line, by name. The second scenario's blob appears on no line at all — git reflog | grep <blob> returns zero. That is the documented behaviour: the reflog docs say reflogs record when "the tips of branches and other references were updated." The index is not a reference. git add is not a ref movement. So it has no place in the reflog.

What follows from this is that the two tools are not backups for each other:

$ git fsck --unreachable
unreachable blob 756d8d4457346ad95d7fb3d51222ece2352462b3

$ git fsck --unreachable --no-reflogs | sort
unreachable blob 65271b600c21b07fa5711bec44cc7fc13077b0ad
unreachable blob 756d8d4457346ad95d7fb3d51222ece2352462b3
unreachable commit d8c76e02b01d09bd2ab78033af99fd98e9fcea17
unreachable tree 1bd47fcd374c27eda598a1ea6c1c86faaa831e5d

(fsck does not sort its output; I piped it through sort to make it readable.)

One line, four lines. The default fsck counts the reflog as a root too — its own documentation puts it as "all reflogs (unless --no-reflogs is given) as heads." So a commit the reflog protects is not even lost in fsck's eyes. That is usually why people are surprised that "fsck found nothing": what they are looking for is still in the reflog and will come back from git reflog in a single line.

The reverse holds too, and that is where it bites. Work that never became a commit lives where git reflog will never show it; only fsck finds it.

To see that the content really comes back, cat-file is enough:

$ git cat-file -p 756d8d4457346ad95d7fb3d51222ece2352462b3
staged taslak

If you'd rather not copy SHAs by hand, fsck has a mode that writes to disk:

$ git fsck --lost-found
$ find .git/lost-found -type f
.git/lost-found/commit/d8c76e02b01d09bd2ab78033af99fd98e9fcea17
.git/lost-found/other/756d8d4457346ad95d7fb3d51222ece2352462b3

Commits land under commit/, everything else under other/. For blobs the file's name is the SHA and its content is the blob itself — in the docs' words, "the contents are written into the file, rather than its object name." Which means you can walk into other/ and skim your drafts with head. Staring at a directory full of nameless files is not a pleasant experience, but it works.

A careful reader will stumble here: --unreachable gave one line above, yet --lost-found wrote the reflog-protected commit to disk too. Not a contradiction — a separate behaviour, and one the docs don't mention. I separated the two in a clean repository:

$ git fsck --unreachable --no-progress
(no output)
$ git fsck --lost-found --no-progress
dangling commit 6a9eca0b9f4a103672772c567ac949f53682b5f4
$ find .git/lost-found -type f
.git/lost-found/commit/6a9eca0b9f4a103672772c567ac949f53682b5f4

--unreachable finds nothing, because the commit is reachable from the reflog. --lost-found looks at dangling objects and does not treat reflogs as roots; it both lists that commit and writes it out. In practice this helps: a commit you know is in the reflog but cannot find the right line for, --lost-found hands you as a file.

A deleted worktree: prune takes the reflog with it

Now to my own scenario. A main repository, one linked worktree, and inside that worktree a commit, a staged file, and a file that was never added. Then I removed the directory with rm -rf and ran git worktree prune — exactly what my collector does.

### produced inside the worktree
  where are the objects? objects dir: .../ana/.git/objects
  per-worktree HEAD reflog: .../ana/.git/worktrees/is/logs/HEAD

### worktree directory DELETED (rm -rf) + git worktree prune
  worktrees/ admin dir left: 0 entries
  per-worktree reflog file: GONE

### looking from the main repo, are the objects there?
  commit 15edfb1...      : PRESENT
  staged blob d0b5675... : PRESENT
  untracked 42a97a7...   : GONE
  does branch 'is' still exist: YES

Three things at once. First: the object database is shared. A worktree has no objects directory of its own; it uses the main repository's — the repository-layout doc says the objects entry is redirected to the common directory when $GIT_COMMON_DIR is set. So every object you produce in a worktree stays in the main repo even after the worktree is gone. On that point the sentence in my script's comment was right.

Second: git worktree prune deletes that worktree's HEAD reflog. The git-worktree doc does not say so; it only says it removes the information under $GIT_DIR/worktrees for worktrees whose working trees are missing. The reflog file lives inside that directory (worktrees/is/logs/HEAD), so it goes along with it.

I wondered which of the two steps actually does the deleting, so I separated them — the reflog file sits inside the main repository, not inside the directory being removed:

1) while the worktree is alive   : PRESENT
2) after rm -rf, before prune    : PRESENT
   $ git worktree list
     .../ana  9599e0e [main]
     .../is   b667770 [is] prunable
3) after prune                   : GONE

rm -rf doesn't touch the reflog; at that stage worktree list still lists the worktree and marks it prunable. It is prune that takes the reflog. That's something the docs don't state, learned by testing; behaviour like this can shift between versions, so it is worth a single check on your own.

Third: the branch ref survives. prune only collects administrative data, and refs/heads/is stays put. So the committed work in a worktree my collector deleted is still reachable by branch name — that isn't even recovery, git checkout is does it.

So what happens when the branch goes too? I ran git branch -D is and looked again:

--- fsck --unreachable ---
unreachable blob 64ce9a3...
unreachable blob d0b5675...
unreachable tree b20221c...
unreachable commit 15edfb1...
--- fsck --unreachable --no-reflogs ---
(the same four lines)

The two outputs are identical. This is the exact opposite of the first half of the lab: there is no reflog left anywhere, so --no-reflogs changes nothing. prune took the per-worktree reflog, branch -D took the branch reflog; the main repo's HEAD reflog has no trace of that branch, because those commits were made under a different HEAD. This is the scenario where git reflog is of no use whatsoever and fsck is the only road. The rule in one sentence: without a ref the object is invisible, but it goes on existing.

stash drop: ref and reflog both go, the commit stays

There is a fourth loss mode, and it is the most common one: git stash drop. Dropping the wrong stash takes a second, and stash list never shows it again. Here too, measurement changes the picture.

First, what a stash actually is. With git stash push -u I set aside both a tracked change and a file that was never added:

stash commit: 49b98c1dcca45babfa71e4136bc55fc483ecd48e
stash list:
  stash@{0}: On main: yarim is
refs/stash reflog file: PRESENT
files in the stash tree: t.txt
third parent of the stash (untracked files): 39b709e2d270d395e3eae459a149283c1ad861c6

That last line is the most delightful detail in this article. A stash is a commit, and when invoked with -u it gets a third parent: a separate commit carrying the untracked files. The DISCUSSION section of the git-stash doc describes two parents — the working-directory tree, the first parent HEAD, the second parent the index — and never mentions a third. Like the worktree prune behaviour, this is something the docs don't state and a lab does. In other words git stash push -u is the one command that turns the "never became an object" file from scenario A1 into an object. If you want a SHA to exist for a file you never git added, that is the way.

Now drop it:

### stash drop
stash list: ''
refs/stash ref: GONE
refs/stash reflog file: GONE
stash commit object: PRESENT

Three "gone", one "present". drop deletes both the refs/stash ref and its reflog file — the same situation as the deleted worktree: git reflog has nothing for you, but the object is in place. fsck shows three commits at once: the stash itself, the commit holding the index state, and the commit holding the untracked files.

Getting it back needs no stash list, because apply accepts a raw SHA:

$ git stash apply 49b98c1dcca45babfa71e4136bc55fc483ecd48e
On branch main
Changes not staged for commit:
    modified:   t.txt
Untracked files:
    yeni.txt

Both the tracked change and the untracked file came back. t.txt's last line is in place, yeni.txt is in the directory. A dropped stash commit reopens in full — on one condition, that gc has not pruned it yet.

The practical upshot: if you dropped the wrong stash, don't panic. List the commits with git fsck --unreachable | grep commit, find the right one with git stash show -p <SHA>, then apply it. "I deleted the stash, it's gone" is wrong most of the time.

The window: two weeks, and it doesn't close unless gc runs

An object being present is not yet a reason to relax. The lifetime of what fsck finds depends on gc. The defaults are written down in config/gc.adoc, and they are all different from each other:

Setting	Default	What it governs
`gc.pruneExpire`	2 weeks	age at which an unreachable object is pruned
`gc.reflogExpire`	90 days	lifetime of reachable reflog entries
`gc.reflogExpireUnreachable`	30 days	reflog entries pointing at unreachable commits
`gc.worktreePruneExpire`	3 months	administrative data of dead worktrees
`gc.cruftPacks`	`true`	unreachable objects go into a separate pack

I ran git gc in the lab and looked at what happened:

--- BEFORE gc ---            --- AFTER gc ---
count: 7                     count: 0
in-pack: 0                   in-pack: 7
packs: 0                     packs: 2

pack-0d7c747...idx
pack-0d7c747...mtimes     <-- cruft pack marker
pack-0d7c747...pack
pack-0d7c747...rev
pack-76f69dc...idx
pack-76f69dc...pack
pack-76f69dc...rev

A2 blob after gc   : PRESENT
A3 commit after gc : PRESENT

Two packs came out, one of them with a .mtimes file. That one is a cruft pack: unreachable objects are placed into a separate pack instead of being left as loose files, and each object's modification time is kept in the .mtimes file beside it, so the pruning decision can be made against that time. The mechanism arrived in 2.37; its becoming the default was announced in the 2.41 release notes: "Now the use of cruft packs has been made the default and no longer considered an experimental feature." After gc, fsck --unreachable still lists the same blob; being packed is not being lost.

Let me underline that one sentence, because it matters shortly: the pruning decision reads the object's modification time. Not the commit's date.

Close the window and the table changes:

$ git gc --prune=now
  A2 blob   : PRUNED
  A3 commit : PRESENT
  reflog line count: 4

That asymmetry is the summary of this article. Same command, same moment: the work that never became a commit is gone, the work that did is still here. The only difference between them is those four reflog lines. The NOTES section of the git gc doc, listing what pruning leaves alone, names "the index, remote-tracking branches, reflogs" — a reflog entry is a root of reachability, so the commit survived while the blob had nobody. (The doc's other rule, that objects newer than the prune date are kept, cannot explain it: under now no object is newer than the prune date. What saved the commit was not its age but the reflog.)

In my real repository the trace looks like this:

$ stat -f '%Sm %N' -t '%Y-%m-%d %H:%M' .git/objects/pack/*.mtimes
2026-10-03 09:29 .git/objects/pack/pack-5b5e6d6...mtimes
$ git reflog | wc -l
89
$ git reflog --date=iso | tail -1
6998be3b HEAD@{2026-08-11 08:45:28 +0300}: reset: moving to HEAD

The oldest reflog entry is from 11 August, 96 days ago. gc.reflogExpire defaults to 90 days, and the first instinct is to say the entry should have been gone. Resist it — the 90-day clock is only consulted while gc runs. The repository's cruft pack file is dated 3 October, the last trace gc left here. From 11 August to 3 October is 53 days, so at that run the entry had not come close to 90. gc has not run since, nobody has looked at the clock again, and the entry sits there at 96 days old.

The lesson isn't in the number: when a deadline passes, nothing happens. These defaults are not an alarm on a timer, they are a criterion applied when gc runs. That's good news for me — the window doesn't close without my noticing. For the same reason there's bad news too: my .git directory has been carrying 2,089 unreachable objects for months, and some of them are the contents of rejected drafts.

I repeated the measurement with two Git versions: /usr/bin/git 2.54.0 and /opt/homebrew/bin/git 2.48.1. The one-versus-four line difference, the cruft pack, and the --prune=now asymmetry came out identical in both. Having two Gits on the machine changes nothing here; even so I called them by full path, because whichever git the shell resolves is the one doing the measuring.

Finding what you want among 1,723 blobs

Everything up to here is mechanism. In practice the problem is this: fsck hands you 2,089 lines of SHA and you are looking for one draft. I tried three approaches, on the real repository, with a word that appears inside a draft that genuinely got lost. The timings come from zsh's built-in time, the total column, run back to back on the same repository.

The first is the obvious one — stream them all and count with grep -c:

$ awk '$2=="blob"{print $3}' u.txt | git cat-file --batch | grep -c 'LaunchServices'
223

0.35 seconds, and it answers the wrong question. cat-file --batch pours every blob into a single stream; grep -c counts matching lines, not blobs. 223 does not mean "223 drafts," it means "this word appears on 223 lines." On top of that, blob boundaries are lost, so it can't tell you which object it was in. Speed buys nothing here.

The second is correct but slow — scan each blob separately:

$ awk '$2=="blob"{print $3}' u.txt | while read sha; do
    git cat-file -p "$sha" | grep -q 'LaunchServices' && echo "MATCH $sha"
  done | wc -l
94

94 blobs, 17.7 seconds. The right answer, but it spawns two processes per blob (cat-file and grep) — over three thousand of them for 1,723 objects.

The third uses a git grep capability most people don't know about — it takes blob SHAs directly as arguments:

$ awk '$2=="blob"{print $3}' u.txt | xargs git grep -l 'LaunchServices'
7801e0eb00c5c6b7a0192ecb6581c8ac1d00c580
6606d42ef25c403b992e418e73162c8099894f39
480ad0940c1c3fc7aeb56ea67a169dd2958a129b
...

The same 94 objects, in 0.27 seconds. 65 times faster than the loop, and its output is directly usable: one object name per line. I'd reach for this first — except I didn't; the first thing I tried was the fast one that returns the wrong number. The scanning script itself produces findings, and whoever doesn't check what the finding counts puts it straight into the article.

I took a look at one of the objects it found:

$ git cat-file -p 7801e0eb00c5c6b7a0192ecb6581c8ac1d00c580 | head -4
---
title: "I Deleted 18 Apps; 661 MiB of Data Stayed on Disk"
description: "I went looking for free space and found 79 leftovers..."
category: "life"

An article draft, on no branch tip. Its size comes from git cat-file -s: 22,462 bytes. Its date comes from inside the blob — the draft's own frontmatter carries publishDate: 2026-10-12. A blob has no date of its own; an object is just the hash of its content, and you cannot ask Git when a file was written. This is an intermediate version of a published piece, most likely the remains of a commit that got --amended after a review round. Still readable, and its name known only to fsck.

Lining up 103 unreachable commits by date

The blob side is solved by content search. On the commit side there's something better: a commit object carries its date and subject line, which means those 103 unreachable commits can be turned into a chronological list.

$ awk '$2=="commit"{print $3}' u.txt \
  | xargs -n1 git show -s --format='%ci|%h|%s' | sort -r

1.1 seconds, 103 lines. The six most recent in my real repository:

2026-11-11 17:47:43 |24b21836| feat: yeni makale — tar'ın Yuttuğu Dosyayı pax...
2026-11-11 17:27:33 |105d570d| feat: yeni makale — Dosyayı tar Atladı, pax...
2026-11-11 17:05:22 |2931125d| feat: yeni makale — Arşivimde 3.214 Üye Saydım...
2026-11-11 17:04:11 |0b99f9c6| feat: yeni makale — Arşivimde 3.214 Üye Saydım...
2026-11-04 19:30:46 |c5c78013| feat: yeni makale — Kuralı Belgeye Yazdım...
2026-11-02 14:14:03 |3b948f14| feat: yeni makale — 499 Bayt İstedim...

That list reads like the negative of my own publishing history. There are four commits on 11 November and two of them carry the same title: that day I changed an article's title twice and rewrote the commit twice. The published version became tar'ın Yuttuğu Dosyayı pax Aynı Arşivden Çıkardı; the Dosyayı tar Atladı attempt is waiting right here, unreachable. This is exactly the trail rebase and --amend leave behind.

The monthly distribution says more:

   2  2026-05
  60  2026-06
   2  2026-08
  22  2026-09
  11  2026-10
   6  2026-11

The oldest unreachable commit is dated 29 May 2026; today is 15 November, 170 days later. The temptation here is to say "so the two-week window has been exceeded twelvefold." I wrote exactly that. It is wrong.

I'm leaving the note in because this is precisely the kind of mistake the article warns about. Pruning does not look at a commit's date; it looks at the object's modification time. The two are independent. I tested it: I made a commit dated 29 May, rendered it unreachable, expired every reflog, then ran the prune:

commit date                        : 2026-05-29 10:00:00 +0300
object file mtime                  : 2026-11-15
after git gc --prune=2.weeks.ago   : PRESENT

A 170-day-old commit date survived a two-week prune, because the object itself had been written today. The list above tells me what was authored when, not what will be pruned when. A rebased or repacked object can carry an mtime months newer than its commit date.

The real record of prune ages lives in the .mtimes file beside the cruft pack. I looked into the repository's 3 October cruft pack:

$ git verify-pack -v .git/objects/pack/pack-5b5e6d6....idx | awk '/^[0-9a-f]{40} /{print $2}' | sort | uniq -c
 450 blob
  86 commit
 183 tree

719 objects. So the gc on 3 October did not delete those unreachable objects; it moved them into a separate pack and kept them along with their mtimes — which is the entire reason cruft packs exist. Of today's 2,089 unreachable objects, 719 are waiting in there; the rest came into being after 3 October.

The practical reading: "two weeks" is not a timeout, it is a criterion applied to object mtimes whenever gc runs. No gc, no criterion; and when it does run, it reads file ages, not calendar dates. Leaving the recovery job for tomorrow on the grounds that "it'll still be there" is a gamble that is holding for now.

The gap in my own collector

The measurement produced a by-product. Here is my collector's first gate:

if [[ -n $(git -C "$wt" status --porcelain | grep -v '^??') ]]; then
  log "KORU/is  $rel"; ((kept++)); continue
fi

The grep -v '^??' drops untracked files from the list. So if the only work in a worktree is a file that was never git added, the gate calls that worktree "clean" and lets it through:

$ git status --porcelain
?? yeni-taslak.txt
$ git status --porcelain | grep -v '^??'
(empty)  -> gate 1 PASSES

That is scenario A1 of the lab, exactly: since no object was ever created, neither fsck nor the reflog nor a branch ref helps once it's deleted. The collector's other two gates — the two-day age check and the unpushed-commit check — catch most cases in practice; an agent going two days without either committing or adding a file is a rare combination. But "rare" is not "never." Tightening the gate to look at the raw status --porcelain output instead of grep -v '^??' is a one-line change; the cost is that clean worktrees holding a single scratch file get protected too, which means a little more disk.

I'll leave that here as a lesson: the place where you noted "this is not data loss" in an automation deserves one measurement after you write the note. Mine was right for two of three cases.

This is not a backup: a clone brings none of it

Everything so far holds on a single machine. To test that, I cloned the repository on my own disk — but with local shortcuts disabled, forcing it to behave like a real network clone:

$ git clone --no-local --bare file:///.../mustafaerbay /tmp/klontest
4.7 seconds

$ git -C /tmp/klontest count-objects -v
count: 0
in-pack: 48661
packs: 1

$ git -C /tmp/klontest fsck --unreachable | wc -l
0
$ git -C /tmp/klontest reflog | wc -l
0
$ git -C /tmp/klontest cat-file -e 7801e0eb...
ABSENT

Zero unreachable objects, zero reflog entries. The draft I recovered a moment ago is not in the clone.

There is a trap in comparing object counts: in count-objects -v output, count and in-pack do not add up. The prune-packable: 34 line in that same output says 34 of the loose objects are already in a pack as well. The correct arithmetic is 49,845 + 1,179 − 34 = 50,990, and counting distinct objects directly with git cat-file --batch-all-objects gives the same figure. The clone holds 48,661, so the gap is 2,329.

That gap is larger than the 2,161 unreachable, and equating the two would be wrong. A clone doesn't just skip unreachable objects; it also skips objects behind refs it doesn't copy. Beyond refs/heads and refs/tags, my local repository has 67 refs/remotes, one refs/stash and one refs/codex; the clone has only 89 branches and one tag:

local                 clone
  89 refs/heads        89 refs/heads
   1 refs/tags           1 refs/tags
  67 refs/remotes
   1 refs/stash
   1 refs/codex

The practical consequence is blunt: the objects fsck finds exist only on that machine. git push never sent them, they are not on the server, not in a colleague's clone, not in CI's checkout. If your disk goes, nothing you could have recovered with fsck remains. I've written before about a ref-less object staying invisible on the wire, in the notes I wrote on 3,583 commits that a clone never brought along; here is the hardest form of the same rule — no ref, no travel.

So every recipe in this article is a second-chance mechanism, not a backup strategy. Carrying 2,089 unreachable objects on my disk does not save me from being unbacked.

Once you step outside a single machine

These recipes hold inside your own clone. Step outside it and five things change. None of them got a lab in this article, but anyone applying this in production should look at them first.

Git LFS. fsck recovers the LFS pointer, not the file. The content lives under .git/lfs/objects or on the LFS server, and that has its own pruning logic. If the blob you recovered is a 130-byte text file, look inside it before celebrating.

Bare repositories and the server side. In a bare repository core.logAllRefUpdates is off by default, so the second layer — the reflog — effectively doesn't exist there. On GitHub, GitLab or Gitea you cannot run fsck at all; a force-pushed commit is reachable only through the provider's API, for a limited time. The checklist above quietly assumes you have a local clone.

Submodules. fsck in the superproject does not descend into a submodule's object database. If the lost work is inside a submodule, the whole drill has to be repeated in that directory.

Scale, and --lost-found. fsck reads every object; on a multi-gigabyte repository that is not a cheap command. --lost-found can dump thousands of files into .git. And the xargs git grep recipe will hit the argument limit with thousands of SHAs and split into several invocations — the answer stays correct, but "one process, 0.27 seconds" no longer holds.

Leaked secrets, not just disk. Throughout this article I've framed unreachable objects as a disk cost. The sharper angle: a password that was committed and then "removed" stays readable for the whole window. The only thing that closes that window is git gc --prune=now — and if it was ever pushed, there is a separate job waiting on the server.

Recovery checklist

When a piece of work disappears, this is the order I go in:

Was the file ever git added? If not, leave Git alone and go to a filesystem backup or your editor's local history. Git holds no trace of that file.
git reflog and git reflog show <branch>. If what you want became a commit, the answer is most likely here and on a single line. git branch rescue <SHA> is enough to give it a ref.
Was it a deleted worktree? The branch ref may still be there: git branch --all | grep <name>. If it is, no recovery is needed.
Was the wrong stash dropped? The commit is there even when git stash list is empty: git fsck --unreachable | grep commit, identify the right one with git stash show -p <SHA>, reopen it with git stash apply <SHA>. A stash taken with -u brings untracked files back too.
git fsck --unreachable --no-reflogs. I reach for the variant that doesn't treat reflogs as roots, because I want to see what the reflog is protecting as well. Add --no-progress to keep the output readable.
Looking for a commit? Line them up by date: git fsck --unreachable | awk '$2=="commit"{print $3}' | xargs -n1 git show -s --format='%ci|%h|%s' | sort -r. Under a second for 103 commits, and the day you want jumps out.
Looking for a blob? Search the content: git grep -l '<pattern>' <blob SHAs>. Don't write a loop; git grep takes blob SHAs and is 65 times faster.
Bring it back: for content, git cat-file -p <SHA> > rescued.mdx; for a commit, git branch rescue <SHA>. If you want a directory full of files, git fsck --lost-found.
Check the window: git config --get gc.pruneExpire. Unset means two weeks. Don't run git gc --prune=now in that repository until the recovery is finished, and if an automation would run it, stop it.

The inverse of that last item is true as well. If you don't want to carry the contents of rejected drafts inside .git for months, git gc --prune=now cleans up exactly that — but knowingly, after you are sure there is nothing left to recover.

The remaining question

Git's recovery capability has three layers: refs, reflogs, and unreachable objects. Most writing covers the first two, because those are what daily life needs. The third layer comes into play on a person's worst day, and on that day there is no time to read documentation.

Underneath those three layers, though, there is a more basic line, and it runs exactly where git add passes. Git stubbornly keeps everything that became an object — ref deleted, worktree gone, reflog pruned, it still sits on your disk past the two-week threshold until gc runs again. For anything that never became an object it can do nothing at all, and it gives no warning about that.

The question to put to your own setup is this: which layer do your automations touch? My garbage collector doesn't touch two of them, and it believed it was protecting the third — the file that never became an object.

Writing this, I fell into my own trap twice. First I read the repository's commit count off a stale working copy. Then I looked at a 170-day-old commit date and wrote that the two-week window had been exceeded twelvefold; testing that pruning reads object mtimes rather than commit dates, I had to delete the sentence. Both were right answers to the wrong question — much like scanning 1,723 blobs with grep -c and getting 223. The hard part of recovery isn't remembering the commands; it's knowing what your number counts, and what your date is the date of.

Official Sources

git-fsck documentation — definitions of --unreachable, --dangling, --lost-found, --no-reflogs, and reflogs counting as roots by default
config/gc documentation — defaults for gc.pruneExpire, gc.cruftPacks, gc.reflogExpire, gc.reflogExpireUnreachable, gc.worktreePruneExpire
git-reflog documentation — reflogs recording ref tips, expire defaults
git-worktree documentation — prune removing administrative data under $GIT_DIR/worktrees, shared versus per-worktree files
Repository layout documentation — objects and logs being redirected via $GIT_COMMON_DIR
Git 2.41.0 release notes — cruft packs becoming the default

I Wrote Sharing State Into the Post; My Archive Changes Daily

Mustafa ERBAY — Sun, 15 Nov 2026 06:47:21 +0000

The content folder of this blog holds 2,838 files today: 1,419 Turkish posts and an equal number of English twins. The first content commit is dated 31 March 2026, the last one yesterday. An archive written in seven and a half months — not bad.

What I was curious about wasn't the size. I wanted to know how many times those 2,838 files had been edited after they were written. Writing is a one-off job; what about everything that comes after?

I counted: 9,721 edits. Divided across the 2,838 files standing today, that's 3.4 edits per file — edits to files that were later deleted are inside that total too, so the average errs on the generous side.

The number that actually stung came after the division. Of those 9,721 edits, only 998 — that is 10.3% — changed a sentence in a post. The other 8,723 never reached the prose at all: a field added, a field deleted, a value flipped from false to true. I thought I had spent seven and a half months writing; the history says I spent most of that time fixing the labels on files.

I'd have liked to pin this on a bot. I can't: three quarters of those edits came from my own hands.

Count first, claim second

Every number here comes from one snapshot: origin/main = 325c5012, the morning of 15 November 2026, with the -- src/content/blog pathspec. That pathspec matters: run the same count with -- 'src/content/blog/*.mdx' and you get 3,001 additions and 0 renames. If you're going to repeat this in your own repository, know which path you counted along.

Git's --diff-filter option separates changes by type: in the words of the documentation, "Added (A)", "Modified (M)", "Deleted (D)" and "Renamed (R)". Since I wanted to tell adding a file apart from editing one, this is exactly the cut I needed.

Event	`.mdx` count
Added (`A`)	2,999
Modified (`M`)	9,721
Deleted (`D`)	163
Renamed (`R`)	2

There are 2,999 add events against 2,981 unique file paths. The 18 extra events belong to 16 files: fifteen were added twice and one four times (deleted, then brought back). To arrive at today's count: 2,999 − 163 + 2 = 2,838. It balances exactly, which tells me the count isn't leaking.

Let me state the method's two limits up front, because in this kind of counting the error usually hides in the definition. First, what I'm counting is not lines but events: if a commit touches ten lines in one file, that's one edit. Second, git detects renames at 50% similarity by default, and the documentation doesn't say which letter reports a file that was renamed and modified — so in theory R events could mask M counts. In my case the R total is two, so that ambiguity cannot move any number here.

I did not decide "frontmatter or prose" by reading commit messages, because trusting my own messages would be optimism rather than measurement. My first attempt was wrong too. A fixed threshold — "frontmatter ends within the first twenty-two lines" — looked safe enough, but in this repository the frontmatter ends anywhere between line 14 and line 32, and in 46% of the files it runs past line twenty-two.

The right way is this: for every edit, pull the file as it was before and after the commit with git cat-file, cut the frontmatter off, and compare what's left. That gives three buckets:

Where did the edit land?	Events	Share
Frontmatter only	8,025	82.6%
Below it, but only an `import` line, a component tag or a blank line	698	7.2%
An actual line of prose	998	10.3%

I keep the middle bucket separate on purpose: adding a <Mermaid> component to a post changes the body of the file without improving a single sentence. Folding the two together would be counting in my own favour.

What the edits were about

I also sorted the same 9,721 events by subject. I'm keeping this table separate from the count above, because this one is judgement rather than measurement: I put each commit in a bucket by its subject line and left what I couldn't classify under "other".

Subject of the edit	Events	Share
Social sharing markers	4,211	43.3%
Dead platform fields (mastodon, hashnode)	2,081	21.4%
Cover/image schema	1,618	16.6%
Other	796	8.2%
De-fab: removing fabricated content	641	6.6%
i18n / EN structure	282	2.9%
Frontmatter schema	92	0.9%

I came to this table with an expectation: I assumed I had been improving posts over time. Fixing a sentence, updating a measurement, refreshing a link. Of the 998 edits that changed a line of prose, 624 — 62.5% — come from a single cleanup campaign; the rest is scattered.

There's one more thing. Of the 1,392 commits that contain edits, just eight produced 56.0% of all of them — 5,442 events. The other 1,384 commits add up to 4,279. My archive's history consists of eight sweeps and the 1,384 small touches between them.

I opened all eight. Not one of them improves a single sentence:

Date	Files	What it did
27 Apr	337	Converted embedded SVG covers to PNG
7 May	457	Migrated covers to a standard template
9 May	934	Flipped `twitter`/`bluesky` to `true` (and `linkedin` in 490 files)
9 May	480	Wrote separate cover paths for EN
10 May	946	Added `mastodon`, `hashnode`, `devto` fields
16 May	1,042	Deleted the `mastodon` field
16 May	212	Removed duplicate `<figure>` and `import` lines from EN posts
18 May	1,034	Deleted the `hashnode` field

Three cover decisions, four social-field decisions, and one markup cleanup on the EN side. All mine, all after the fact.

Counting commits and counting edits tell different stories

The first row of the subject table — 43.3% — is the bill for a single architectural choice. The frontmatter of a post carries a block like this; the field list has changed over time, and fewer than half the files in the archive carry all six today:

socialMedia:
  twitter: false    # has it gone to X yet?
  linkedin: false   # ...to LinkedIn?
  bluesky: false    # ...to Bluesky?
  devto: false      # ...to dev.to?
  burncpu: false    # ...to burncpu.com?
  hashnode: false   # ...to Hashnode?

These fields don't mean "should this be shared" but "has this been shared". When a post goes to X, a script comes back and writes twitter: true. The idea is simple and, at first glance, tidy: let the record of where a post went live next to the post, so nothing else has to be consulted.

Here's the consequence. A share is an event; a file is a work. The moment I wrote the event inside the work, every share became a commit. Since 9 May, a commit author named Social Bot has pushed 1,236 commits into the content folder — an average of 6.5 commits a day over 189 days. I counted this five weeks ago too; the bot stood at 1,131 commits that day, and it's at 1,236 now.

But the unit of this post is the edit, not the commit. Switch units and the picture inverts:

Author	Commits with edits	Edit events
Me (across four email addresses)	119	7,292 (75.0%)
Social Bot	1,235	2,330 (24.0%)
Other bots (FAQ, Editorial, Content)	38	99 (1.0%)

The bot has ten times my commits and a third of my edits (2,330 against 7,292). It touches one or two files each time; I make a decision once in a while and open a thousand files. Two units, two stories, both true — and if I don't say which one I picked, I'm misleading the reader. My first draft of this post opened with "the biggest editor of my archive is a bot"; counting the authors caught the claim on my own methodology warning.

There is a point where I have to defend my own choice, because it wasn't entirely thoughtless. Keeping share state in the file bought me something: git history gave me a free audit trail of when each share happened. That's exactly why I can produce the bot numbers in this post. An UPDATE in a database would have left no trace; a line in a file did. Still, I think the trade is clear: if I want an audit trail, I keep one deliberately, rather than making every file in the archive writable on a daily basis.

A field that never entered the schema still sits in 1,225 files

The 21.4% held by dead platform fields is the second bill for the same choice — and the part that taught me the real lesson.

On 10 May I wrote mastodon: true into 946 files. At 12:30 on 16 May I deleted that field from 1,042 files. The reason was mundane: my Mastodon account was suspended, and the information the field carried became meaningless.

My first version of this story opened with "on 16 May I removed the field from the schema". The schema says otherwise: the mastodon lines deleted from src/content.config.ts that day belonged not to the blog but to the short-video collection I had opened a day earlier. The blog's socialMedia schema recognised exactly three fields at the time — twitter, linkedin, bluesky. So mastodon had gone into the frontmatter of 946 posts without ever being defined.

Then I counted today.

$ grep -c "^  mastodon:" src/content/blog/*/*.mdx | grep -v ":0" | wc -l
1225

The field sits in 43% of the archive today — 1,223 false and 2 true. Their creation dates explain why:

Month the file was added	Carries `mastodon` today
2026-05	255
2026-06	504
2026-07	400
2026-08	66

The template that generates posts kept writing the field I had deleted in May, right through the end of August. The deletion didn't win; the template did. Nothing objected either: the field wasn't in the schema, so it tripped no validation — and since 29 June the blog isn't even an Astro collection, with content served from content.db, so no gate inspects the extra frontmatter fields at all today.

I fell for the same illusion once more. On 18 May, while hashnode was being removed from 1,034 files, the same commit also cleaned up 24 mastodon: false lines. My first reading was "the sweep two days earlier missed 24 files" — a neat lesson about thousand-file operations never landing in one pass. Then I checked when those 24 files were born: all twenty-four were created after the sweep. The sweep missed nothing; what was missed, again, was the template.

That's the correct lesson: a field is owned by the pipeline that writes it, not by the archive that holds it. Close the gate first, then sweep; do it the other way round and every sweep breeds the next one. Eleven days ago, counting how nine of 3,580 commits followed the rule I had written into a document, I ended up somewhere similar: there a written rule was racing its own template, here what races is where the state lives — and this time the loser's bill gets multiplied by the file count. hashnode is back in 1,593 files today and devto sits in 2,827 — same cause. Producing 1,391 tags and leaving 931 of them on a single post each was the same class of mistake: I fixed the output instead of the gate on the production side.

I made the cover decision at 340 files

The cover/image line — 16.6% — shows best how the cost grows with the size of the archive.

On 27 April I converted covers from SVG to PNG, because RSS readers and social cards wouldn't render embedded SVG. I walked into a measurement trap here too: I asked "how many files were in the archive that day" and took the end-of-day count. The right denominator is the sweep's own tree:

Sweep	`.mdx` in the commit's tree	Touched	Ratio
27 Apr, SVG → PNG	340	337	99.1%
7 May, cover template	467	457	97.9%
16 May, mastodon	1,058	1,042	98.5%

With end-of-day numbers all three came out at "97%"; against their own trees they climb towards 99%. The direction doesn't change, but the pattern gets sharper: a schema decision means touching nearly the whole archive as it stands at that moment.

That tree holds 2,838 files today. A decision that touched 340 files in April touches 2,838 now — 8.3 times more in six and a half months. The hundred posts I write next will nudge that multiplier further. This is the measured version of "decide early", and I did the opposite: I produced fast and fixed the schema on the way.

The wave that touched the prose

The de-fab line — 641 events — is the largest piece of work in this archive that genuinely changed the text. At the end of June, across six commits, I cleaned fabricated numbers, outputs and anecdotes out of AI-generated posts; 455 distinct files were touched.

One detail: all 455 of them are Turkish. The commits' own tree held 1,802 files, but half of those were English twins and the cleanup never reached them. (I took that tree count from the commits themselves rather than from the end of the day — so as not to fall into the trap from the previous section a second time.) I learned that afterwards, while counting the 474 figures that stayed behind in the English twins of the 455 posts I had cleaned — so this line doesn't say "I cleaned the archive", it says "I cleaned the Turkish half of it".

I don't bring it up as an achievement either. A ban on fabrication should have been a rule from the start; applying it retroactively to an archive of 901 Turkish posts in June is proof the rule arrived late. The post where I counted my own fixes turned up something similar: some of those fixes weren't failures but habits. This one is the same kind of thing — 641 edits, the bill for a policy gap.

This work has a name

Google's SRE book calls this sort of work toil and names six characteristics: manual, repetitive, automatable, tactical, "devoid of enduring value" and — the one that cuts closest for me — "O(n) with service growth", meaning it grows linearly as the service grows.

My 8,723 edits that never reached the prose satisfy all six — "manual" included, because three quarters of that work came from my own hands, not from a script; my commit count merely looks small because I touch a thousand files at a time. They satisfy the last one all too well: when the archive doubles, the cost of every schema decision doubles with it.

The book's target is for toil to stay under half of an SRE's time, and Google's own surveys measured an average of 33%. My 89.7% is not in the same unit as those figures: theirs is a share of time, mine a share of edit events. I'm writing it down for the sense of scale, not as a comparison.

Saying "I automated it" doesn't make the cost disappear either; it moves it. I checked this while writing: the deploy workflow's push trigger explicitly excludes the src/content/** path, so those bot commits don't each kick off a publish run — that was my first guess and it was wrong. The cost sits elsewhere: my content folder never goes quiet. Every branch I prepare a post on sits on top of a main branch sliding underneath it, and every measurement turns into a snapshot that is already a few minutes stale. The numbers in this post had to be nailed to a commit for the same reason.

Today's price list

How widespread each field is across the archive today is the price tag on every decision I'm postponing:

Field	Files containing it
`tags`, `readingTime`, `coverImageAlt`	2,838
`socialMedia` block	2,831
`devto`	2,827
`hashnode`	1,593
`translatedFrom`	1,419
`burncpu`	1,349
`mastodon` (not in the schema)	1,225

The decision framework I've distilled comes down to three questions:

Is this field part of the work, or part of what happened to it? What I wrote (title, text, sources) stays in the file. What happens to the post afterwards (shared, viewed, liked) doesn't. One question is enough to separate them: can this value change without me touching the file? If yes, it doesn't belong in the file.
How many files is this decision worth? Before I propose a schema change, I count today's files. 2,838 is too large a number to leave something to "we'll handle it later".
Who writes the field? Deleting it from the schema is not deleting it from the archive. Every cleanup done without closing the production path produces the raw material for the next cleanup — my 1,225 files are the proof.

Astro's content collections documentation describes the schema like this: with Zod it will "validate every file's data within a collection", and for a file that violates it, "Astro will provide a helpful error". So a schema is not a promise but a gate: the moment you add a new required field, the entire archive has to pass through it. The documentation doesn't describe a migration path for that; that part is left to you.

My mistake was never using that gate at all. None of the sharing fields were defined in the blog's schema: it recognised three fields while I wrote six into the frontmatter, and the extras passed silently. A field that doesn't go through the gate doesn't report back when the gate changes — that's the short explanation for the dead field in 1,225 files.

Finding somewhere to move it is harder than moving it

Everything up to here pointed at one recommendation: take the state out of the file and put it in the database. Since 29 June, posts have been served from content.db at runtime rather than at build time, so that looked like the obvious destination.

Opening the code showed me half my own advice doesn't work. content.db isn't an independent ledger but a copy derived from the MDX: the prebuild step runs migrate-content.mjs, which fills the social_media column straight out of the frontmatter. The runtime connection is opened readonly: true, so the site cannot write to that ledger.

The script has run incrementally since 29 June: a file whose content hash hasn't changed isn't re-rendered, and its row is carried over from the previous database. So a "shared" value I wrote by hand wouldn't vanish at the next build — it would vanish the first time that post's MDX changed. In practice it comes to the same thing: the source of truth is still the file, and the database is its echo.

That makes the job bigger than I thought. Deleting the socialMedia block from 2,831 files is the easy half. The hard half is giving share state its own writable ledger that the build doesn't reset, pointing the sharing script at it, and then inverting ownership. That isn't an evening's work — and it's the honest reason I keep postponing it, rather than "I didn't have time".

The lesson I'm keeping

Writing 1,419 posts in seven and a half months is a story about speed. But that speed turned the archive itself into a cost base: every schema decision I make today gets multiplied by 2,838 files, and tomorrow by a bigger number.

Output speed is easy to measure, which is why we all measure it. Measuring maintenance debt means looking backwards, and the past is uncomfortable: nine out of every ten edits never touch a sentence. Through one architectural choice, I turned the upkeep of what I produce as a writer into a daily chore for myself.

Four of my claims collapsed under measurement while I was preparing this post: that the sweep had missed 24 files, that a bot was the biggest editor of my archive (true only when counting commits), that I had removed the field from the schema, and that a fixed line threshold could tell prose edits from label edits. All four sounded good. That's the price of measuring your own history — the numbers may not like the story you're telling. The benefit sits in the same place: a dead field sitting in 1,225 files only becomes visible once you bother to count it.

Official Sources

Opening a File Doesn't Download It, Reading Does

Mustafa ERBAY — Sat, 14 Nov 2026 16:14:39 +0000

Five days ago I wrote about my cloud quota running out. 1,958 items were stuck in the
queue, and that
article's thesis was this: the files taking up disk space are precisely the ones that
can't get to the cloud. Files whose local copy survives because the upload failed.

This week I asked the same question from the other direction. What happened to the ones
that did go up? How much room do the files that escaped the queue take on my disk?

The answer: none. My home directory holds 236,583 files that ls reports a size
for, totalling 61.84 GiB, and the space all of them occupy on disk is zero bytes.

The more uncomfortable part: those 61.84 GiB don't fit in this machine's free space.
I have 28.85 GiB available. If I asked for every file I own at once, more than half the
request — 53.3% of the bytes — could not be served.

Size is a promise, a block is a fact

macOS calls these files "dataless". The name, size, dates and permissions live on disk;
the contents don't. In Finder you see a cloud icon, but from the shell there is no icon,
just numbers. Two different numbers:

$ cd ~/Library/Mobile\ Documents/com~apple~CloudDocs
$ ls -lO _bootstrap.scss
-rw-------@ 1 mustafaerbay  staff  compressed,dataless 1340 Jun 20  2024 _bootstrap.scss
$ stat -f 'size=%z  blocks=%b' _bootstrap.scss
size=1340  blocks=0

A file 1,340 bytes long that occupies zero blocks. The third column of ls -lO names
the flag too: dataless. This is the file system's official way of telling me "here is
this file's size, but don't ask where it is."

Apple explains the situation to users in two sentences: the files don't take up space on
your Mac, and "you can download the original files when you need them." I've just
confirmed the first half — genuinely zero blocks. The second half, "when you need them",
is what this article is actually about: which moment counts as needing, which operation
triggers the download, and who decides. Apple's support page doesn't go there. I did.

When I asked five different tools how much room was left on my disk and found a 13.5
GiB gap, this flag
wasn't in my accounting. It should have been — because one end of that gap is here.

I didn't go looking for a tool to count them, because no tool is needed. lstat(2)
returns a file's flags, and the SF_DATALESS bit is among them. So I wrote a scan that
walks my whole home directory and opens nothing:

import os
SF_DATALESS = 0x40000000
for dp, dn, fn in os.walk(os.path.expanduser('~'), followlinks=False):
    for name in fn:
        st = os.lstat(os.path.join(dp, name))
        if st.st_flags & SF_DATALESS:
            ...   # count it, add it up, but DON'T READ it

The scan took 4 minutes 45 seconds, looked at 2,042,325 files and hit not a single
lstat error. The result:

Directory	Dataless files	Nominal size
`~/Library/CloudStorage`	71,368	57.23 GiB
`~/Documents`	164,204	4.39 GiB
`~/Desktop`	803	222.23 MiB
`~/Library/Mobile Documents`	208	0.50 MiB
Total	236,583	61.84 GiB

That's 11.58% of the files scanned. The space all four rows take on disk, added
together, is zero bytes. I scanned directories in a separate pass with the same method
(lstat only, this time over os.walk's directory list): there are no dataless
directories at all, 0. On this machine only file contents get emptied out; the
directory structure is entirely local.

The interesting part of the distribution: the most common extension is .smali (81,770
files) — decompiled Android sources left over from an old tablet rescue. Most of the
file count sits under ~/Documents, but 92.5% of the bytes are in
~/Library/CloudStorage: large virtual disk images and video files. Two providers, two
different characters. An inflation of small files on one side, heavy files on the other.

That there are two providers matters, because this is not an iCloud feature. Microsoft
says so plainly in its own documentation: "The system logic to decide what files count
against used disk space and what files do not is provided by the File Provider platform,
not OneDrive or Microsoft." The same page reaches for Apple's term, the word "dataless",
when describing file state. The flag belongs to the operating system and so does the
mechanism; the providers are only tenants.

I didn't set this flag, and I can't clear it

Most file flags are mine to play with. I can set nodump with chflags, lock a file
with uchg, or hide it from Finder if I'm in that kind of mood. SF_DATALESS is not
like that. The definition in the kernel source says it in one line:

#define SF_SYNTHETIC    0xc0000000      /* mask of system read-only synthetic flags */
...
#define SF_DATALESS     0x40000000     /* file is dataless object */

The comment above it is blunter: "Synthetic flags. These are read-only. We keep them
out of SF_SUPPORTED so that attempts to set them will fail." And sure enough, the flag
falls outside the mask of superuser-changeable flags (SF_SETTABLE, 0x3fff0000).
0x40000000 isn't inside that mask, which means not even root can set this bit by hand.
I didn't test that part; I derived it from the mask. What I did test is the ordinary-user
side.

And the ordinary-user side surprised me. I tried setting SF_DATALESS on a local file:

SF_DATALESS  chflags OK  -> flags=0x00000000  (bit still set: False)
SF_ARCHIVED  chflags ERROR -> errno=1 Operation not permitted

chflags returned success and silently swallowed the bit. Whereas a flag that
merely requires privilege — SF_ARCHIVED — failed honestly with EPERM in the same
spot. The unsupported flag fails more quietly than the supported-but-forbidden one. A
script that checks the return value and calls it done would be wrong here; I was wrong
for a moment too, until I read the flags back.

And that measurement corrects the comment I just quoted. The source says "attempts to
set them will fail"; on macOS 26.6 the attempt does not fail, it quietly does
nothing. Those aren't the same thing: one returns -1, the other returns 0. Because
the bit sits outside the mask, the kernel drops it and doesn't complain. What the
comment means — the flag can't be written — is true; the word "fail" is misleading in
the sense a shell script would understand it.

Which operation downloads?

This was the real question. What happens if I run grep -r across those 236,583 files?
If I list their names? If I back them up?

I picked nine fresh dataless files — a different file for each operation, because the
first thing to touch a file localises it and the second measurement is no longer the
same experiment. Before and after each operation I read the flag and the block count:

Operation	Downloaded?	Duration
`lstat(2)`	no	0.0 ms
`ls -l`	no	6.4 ms
`stat -f %z`	no	3.7 ms
`du -k`	no	3.1 ms
`open(2)`, without reading	no	0.2 ms
`head -c 1`	yes	1,243 ms
`grep -c`	yes	1,177 ms
`md5`	yes	1,206 ms
`file`	yes	3,869 ms ⚠

The top five work on metadata and never touch the file. du included — which makes
sense, since du counts blocks and there are no blocks here to count. (The flagged
3,869 ms was the session's very first measurement and carries the service's warm-up
cost; trust the median below, not that row.)

The bottom four read, and reading downloads. But look where open(2) sits: in the top
group. Opening the file, even read-only, pulls down nothing.

Since the article's title rests on that row, I didn't leave it at one measurement. I
opened nine fresh dataless files read-only and closed them without reading a byte:

  406 B  open  0.026 ms  dataless True->True  blocks 0->0
  ...
  450 B  open  0.018 ms  dataless True->True  blocks 0->0

open(2) RESULT: 9/9 files STAYED dataless, median 0.020 ms (min 0.016, max 0.026)

Nine out of nine stayed dataless, and the open took a median of 0.020 milliseconds —
nothing that asks the network for anything can be that fast. The thing that downloads is
the first read. Microsoft's documentation says the file "won't download to your
device until you open it"; in everyday terms that's true, because opening a file in an
application involves reading it. At the syscall level, though, the line isn't at open,
it's at read.

I can't read why the line falls exactly there out of the kernel source, and saying so
is part of being honest about this: vfs_materialize_file(), which calls the resolver,
is open in xnu, but the code deciding which filesystem operation triggers it lives
inside APFS, and APFS isn't open source. I can measure the boundary; I can't read it.

So how long does the download take? The table's figures are single measurements; for a
number free of warm-up cost I ran a clean series across nine files, reading a single
byte each time:

median first read: 1,166 ms (fastest 1,092, slowest 1,441)
median second read of the same file: 0.208 ms

About 5,600×. A file's first read costs five thousand times what its second read
does. (That's the ratio of two medians, not the median of the per-file ratios.)

And the "single byte" doesn't stay a single byte. From a dataless file of 3,693,410
bytes I asked only for the first one:

nominal=3693410 B  before blocks=0  after blocks=7216 (=3694592 B)  dataless True->False  2063 ms for 1 byte

I asked for one byte and 3.52 MiB arrived. No partial download happened here: the file
is either present or it isn't.

It would be wrong to pin that on the mechanism as a whole, though, because the kernel
side is open to range requests. The signature of the function that calls the resolver
looks like this:

vfs_materialize_file(
    struct vnode *vp,
    uint64_t op,
    int64_t offset,
    int64_t size)

offset and size are right there — "fetch this range of this file" is an expressible
thing. The whole file arriving is not a kernel limit, it's a choice made by the provider
at the end of this path. What the measurement shows is that a one-byte request ended
with the whole file local; which link in the chain declined to use the range, I can't
see from here.

A practical consequence follows, and I don't like it: a grep -r hunting for content in
my home directory would try to pull down 61.84 GiB. With 28.85 GiB free, it would fill
the disk short of halfway. Microsoft notes on that same page that Spotlight doesn't do
this: it won't make dataless files locally available while indexing. Sensible. The
question is how Spotlight manages not to.

Who decides: the kernel's answer

In the kernel source the mechanism is called the "namespace resolver". A comment block
lays out how it works: when I/O is performed on a dataless object, the file system makes
an upcall to filecoordinationd, the calling thread goes into an interruptible sleep,
and when the work is done the service wakes it by writing to the vfs.nspace.complete
node. Only a process registered as the resolver may write to that node, and the source
notes: "The system filecoordinationd runs as uid == 0." On my machine, there it is:

$ ps -Ao pid,user,comm | grep -E 'filecoordinationd|fileproviderd|bird'
  715 mustafaerbay     /System/Library/PrivateFrameworks/FileProvider.framework/Support/fileproviderd
  732 root             /usr/sbin/filecoordinationd

The resolver runs as root, the File Provider side as my user. (iCloud's own bird
process isn't in this list; it comes and goes on demand — it showed up in the first
round of measurements, not this one.)

What struck me came a few hundred lines further down the same file:

/*
 * The default behavior is to not materialize dataless files;
 * return to the caller that deadlock was detected.
 */
return EDEADLK;

The default behaviour is not to download. Even though every read in my shell
downloads, the system's documented default is "don't fetch it, return an error." The
getiopolicy_np(3) man page says the same thing: when the default policy is selected at
process scope, the system default applies, and that is
IOPOL_MATERIALIZE_DATALESS_FILES_OFF.

So why does my shell download? Because the policy is per-process and inherited. The man
page puts it in one sentence: "New processes inherit the policy of their parent
process." The numbers:

this process (PROCESS): (2, 'ON(2)')
this thread  (THREAD):  (0, 'DEFAULT(0)')
child process PROCESS:  2

My shell is ON. My shell's child is ON too. But if the system default is OFF, where did
this ON come from? I took the same reading in a process started directly by launchd:

pid=97432 ppid=1 comm=/sbin/launchd
sysctl prevent_materialization=1
getiopolicy PROCESS = 1

OFF. Exactly as documented. So on the same machine, running as the same user, one of two
processes can read the file and the other cannot — and what makes the difference isn't
the file, it's whose child you are in the process tree. My shell runs underneath a
desktop application; I inherited the ON from somewhere in that chain. From here I can't
see which link set it, and I'm not going to invent one.

When I turned the policy off in my own process and tried the same read, I got the
kernel's answer first-hand:

setiopolicy_np(OFF) -> 0 | now: (1, 'OFF(1)')
READ ERROR: errno=11 (EDEADLK) -> Resource deadlock avoided   0.0 ms
file: before dataless=True blocks=0 | after dataless=True blocks=0

Zero milliseconds, "Resource deadlock avoided", and an untouched file.

That also answers the Spotlight question: it needs to do nothing special to avoid
downloading — leaving the policy at its default is enough. I couldn't confirm this by
measurement. getiopolicy_np reports only the calling process's policy; there is no
interface for asking about another process (the same is true of the sysctl node). So
this is an inference: Microsoft's sentence and the kernel's default together imply the
indexer stays at the default, but I can't put the question to mdworker and get an
answer.

To my mind that error message is the worst part of the whole mechanism. It's technically
defensible — the kernel is refusing to wait on a service that would have to answer it,
which is a deadlock guard. But for someone staring at "resource deadlock avoided" in a
script's log, the last explanation that will come to mind is that the file is sitting in
somebody else's data centre.

Everything so far was one provider

Every download measurement above was taken on iCloud Drive files. But 92.5% of my
dataless bytes aren't there — they're with the second provider, under
~/Library/CloudStorage, 57.23 GiB of them. Saying "the mechanism works like this"
without putting the question to that side too would be treating the half I didn't
measure like the half I did.

So I asked. Five files, all dataless, between 1 KB and 100 KB, one byte from each:

    1184 B -> errno=60 Operation timed out    719.6 ms  dataless=True blocks=0
    1231 B -> errno=60 Operation timed out    653.4 ms  dataless=True blocks=0
    1763 B -> errno=60 Operation timed out    679.2 ms  dataless=True blocks=0
    1950 B -> errno=60 Operation timed out    708.1 ms  dataless=True blocks=0
  101240 B -> errno=60 Operation timed out   2308.0 ms  dataless=True blocks=0

Five out of five ETIMEDOUT. Not one arrived. The provider's File Provider extension
was running on the machine at the time — it's right there in the process list — so
this isn't a case to wave away with "the app was closed." I did not diagnose why it
times out: the session may have expired, something may be wrong on the account side, it
may be something else. I'm not writing it down, because I don't know.

Even undiagnosed, this stands: 57.23 GiB that Finder shows with sizes cannot be
fetched on this machine right now. Not slow — not arriving. And the brctl you'll see
in the next section is no help on this side; that's iCloud's tool.

So the "median 1,166 ms", the "asked for one byte, got the whole file" and the "5,600×"
in this article are all statements about iCloud Drive. For the other provider, the only
measurement I have is five timeouts out of five attempts.

I got it wrong twice while measuring

The article would look tidier without this section. I'm keeping it, because both traps
are ones somebody else will fall into.

The first. To prove the policy difference I thought I had set up an elegant
experiment: run the same reader script by hand and through launchd. The one run by
hand downloaded; the launchd one failed with EPERM. Just as I was about to call that
proof, something nagged at me — the error I expected was EDEADLK, and EPERM came
back. So I gave the same script one more target: a file in the same folder that is fully
local and not dataless.

dataless iCloud file  -> errno=1 EPERM (Operation not permitted)   46.1 ms
LOCAL iCloud file     -> errno=1 EPERM (Operation not permitted)    0.0 ms
scratchpad local file -> READ OK                                    0.0 ms

The launchd-started process can't read the local file either. So the obstacle isn't the
dataless policy, it's the privacy layer: a process with no identity has no access to user
folders. The experiment said nothing whatsoever about materialisation. The policy
difference is still real — getiopolicy_np reports it directly — but that read attempt
is not the evidence for it. Without drawing the distinction I would have reached a correct
conclusion on false evidence. A number that supports your case deserves to be questioned
twice; the convenient number is the least questioned one.

The second. vfs.nspace.prevent_materialization is a sysctl node. Say sysctl and
people assume a system-wide setting; the name reads that way too. It isn't. I read it from
three places and got three answers: 0 in my shell, 1 in the process where I had turned
the policy off, 1 in the launchd process. The node reports the state of whoever asks.
A monitoring script that reports this as a machine's "setting" will produce different
truths depending on which process it runs from.

Counting is easy, putting it back needs a tool

As soon as the measurements were done I evicted the files I'd downloaded: brctl evict,
11 out of 11 successful. Cleanup complete — or so I thought.

While finishing the article I counted again, since this piece's whole thesis is "don't
assume, count blocks." Good thing I did. My cleanup script only held the list of the 11
files from the timing series; the run that produced the operation table was a different
script, and it never wrote its 4 downloaded files to that list. On top of that, the
target file from the EDEADLK test had been evicted once and then re-downloaded on a later
round. Re-scanning the ~/Library/Mobile Documents tree said so plainly: dataless was
203, not 208.

I evicted the missing five and counted once more:

FINAL STATE Mobile Documents: total 223 dataless 208  (first scan: 208)

Sixteen files downloaded in total: 11 in the timing series, 4 in the operation table, and
one of them twice across two rounds. All evicted, and the count is back to the 208 it
started at. Nothing came down at the second provider — nothing could — so there was
nothing to clean up there.

The difference between saying "I cleaned up" and having cleaned up was exactly the
difference between reading a brctl evict exit status and counting the tree again.

Doing that walked me into a practical gap in the mechanism. brctl is iCloud Drive's
tool only: its man page lists download <path> and evict <path>, and both do their
job. But 92.5% of my dataless bytes aren't in iCloud, they're with another provider
running over File Provider. What's available for that one? fileproviderctl's command
list offers dump, diagnose, evaluate, check | repair and obfuscate — no
per-file download or eviction. A deliberate eviction from the command line is documented
only on the iCloud side; on the other side you fall back to Finder's context menu or the
provider's own interface.

Querying state, though, is provider-independent. On 9 November I wrote that there was
no command to ask about a single file's status, then found what I was after in brctl's
own help text. This time the answer is cleaner, and a little humbling: no command is
needed. The flag is on the file itself. And lstat tells the same truth for both
providers, because neither of them sets that flag — the kernel does.

Turning the policy off myself

If the kernel hands you a door, use it. I wrote a ten-line wrapper that calls
setiopolicy_np and then execs whatever command I give it:

#!/usr/bin/env python3
"""nofetch.py <command> [arg...] — runs it with materialization off"""
import ctypes, os, sys
IOPOL_TYPE_VFS_MATERIALIZE_DATALESS_FILES = 3
IOPOL_SCOPE_PROCESS = 0
IOPOL_MATERIALIZE_DATALESS_FILES_OFF = 1
libc = ctypes.CDLL(None, use_errno=True)
if libc.setiopolicy_np(IOPOL_TYPE_VFS_MATERIALIZE_DATALESS_FILES,
                       IOPOL_SCOPE_PROCESS,
                       IOPOL_MATERIALIZE_DATALESS_FILES_OFF) != 0:
    sys.exit('setiopolicy_np failed: %d' % ctypes.get_errno())
os.execvp(sys.argv[1], sys.argv[1:])

Because the policy is inherited, it still holds after the exec. I pointed grep at
three dataless files:

$ ./nofetch.py grep -l color <three-dataless-files>
grep: .../_bootstrap-dark.scss: Resource deadlock avoided
grep: .../_bootstrap-extended-dark.scss: Resource deadlock avoided
grep: .../_bootstrap-extended.scss: Resource deadlock avoided
$ echo $?
2

All three stayed dataless, block count zero. This was the thing I actually wanted to
know: grep doesn't skip the file quietly. It writes a line to stderr for each one and
exits 2. A scan run through this wrapper tells you which files it couldn't touch.
No silent omission — which matters more than blocking the download does.

I'm not offering this as a general backup recipe; I tried it with grep on my own
machine, and I haven't measured how tar or anything else reacts to that error.

What I'm going to do

My list for myself is short:

Stop counting the second provider's 57.23 GiB as a backup. Five attempts, five timeouts. If I can't read a file I can't back it up either, and Finder showing me a size promises me nothing.
Put content searches through the wrapper. A bare grep -r in my home directory is a download command as far as I'm concerned now. Run through nofetch, the worst case is that it counts the files it couldn't touch.
Ask "what's on disk" in blocks. ls tells you the size; st_blocks tells you whether it occupies anything. On this machine the gap between the two is 61.84 GiB.
Pin what matters. If a file matters enough that I want it to open when I have no connection, it's too important to leave to the cloud's idea of freeing up space.

This is the exact inverse of the accounting error in the piece about
APFS clones making me pay twice for the same bytes.
There, one copy of a byte was counted in two places; here, bytes that don't exist are
counted in one. Both lie to the same question in different directions: how much room
does this file take?

One closing note, in cloud storage's favour rather than against it: the iCloud side
genuinely works well. I can wander around a five-year archive as if I were carrying it on
a 460 GiB disk, and most days I don't feel the difference. What bothers me isn't the
mechanism but its invisibility: the file system shows me not what I have, but what I
think I have.

A file being "here" is no longer a binary state. Its name is here, its size is here, its
date is here; its contents depend on a policy. I didn't write that policy, I mostly
haven't seen it, and until today I had never once checked it. There is 61.84 GiB of
promise on my disk — and today I learned that 57.23 GiB of it doesn't answer when asked.

Official Sources

I Asked Permission Before Sending the Body: 100-continue

Mustafa ERBAY — Sat, 14 Nov 2026 11:24:02 +0000

HTTP has a polite question, and almost nobody asks it. A client that puts Expect: 100-continue in its request headers is saying: the body is ready in my hands, but I want your approval before I put it on the wire. If the server says "go ahead", a megabyte travels. If it says "no", nothing travels.

This week I asked that question on my own machine dozens of times. Two numbers came out of it: when the question goes unanswered the price is exactly one second, and when it is answered the saving is a megabyte. Whose fault the gap is remains unclear — every component in the path behaved correctly according to its own documentation.

There was a side effect too: the threshold in my head had been wrong for six years.

The rig

14 November 2026, my own laptop: macOS 26.6.2, Apple M4 Pro; curl 8.7.1, Node.js 22.16.0, Go 1.26.1 and nginx 1.29.8-alpine inside Docker 27.4.0. Client and server both sit on 127.0.0.1, so the delays below only show when one process turns back to the other.

Seeing who asks the question took a raw TCP listener: a short Node script that prints the headers verbatim, can choose to send 100 Continue or stay quiet, reads the body to the end and returns 204. It also holds the number I was really after — when the first byte of the body landed.

if (i >= 0) {                       // \r\n\r\n seen: headers are over
  headersDone = true;
  process.stdout.write(buf.slice(0, i).toString());
  if (mode === 'continue') sock.write('HTTP/1.1 100 Continue\r\n\r\n');
}

A note on measurement hygiene: I threw my first round of numbers away. Five variants had run one after another on the same port number, and two rows contradicted themselves — one looked as if the request had never been sent, and the row with the suppressed Expect looked as if the header had gone out anyway. Something was clearly getting mixed up between one process shutting down and the next run starting; I never pinned down the exact root cause, and did not try to. Every variant got its own port, the contradiction disappeared, and the rows became repeatable. The numbers below are from that second round.

One byte, one second

The first question was simple: at what body size does curl add this header on its own? I sent the same request in three sizes against the silent server.

Body	`Expect` header sent	First byte of body	curl total
1,024 B	no	0.3 ms	0.01 s
1,048,576 B (exactly 1 MiB)	no	0.5 ms	0.01 s
1,048,577 B (1 MiB + 1 B)	yes	1,003.8 ms	1.01 s

One byte, one second. The threshold is exactly 1 MiB and the comparison is strictly greater than: a whole mebibyte asks nothing, one byte more asks. The source says the same thing — curl's lib/http.h carries #define EXPECT_100_THRESHOLD (1024 * 1024), and addexpect() in lib/http.c adds the header under client_len > EXPECT_100_THRESHOLD || client_len < 0. That second condition matters: if the body length is unknown (chunked), permission is requested whatever the size turns out to be.

This is where my educated guess fell over. I had that threshold filed away as 1 KiB, and plenty of pages on the internet still say so. It was true — until 2020. curl 7.69.0 raised the threshold from 1024 to 1024*1024: a factor of a thousand. So the sentence "curl sends Expect on POSTs over 1 KB" has been wrong for six years, and I was still carrying it. You only notice this sort of thing by taking the measurement.

Where does the one second come from? curl picks its own waiting period and exposes it as --expect100-timeout; curl's own manual documents the default as one second, and in the code it sits in cr_exp100_read() as the comparison ms < data->set.expect_100_timeout. When the time is up curl does not give up — it sends the body anyway, and confesses in a single -v line:

> Expect: 100-continue
* Done waiting for 100-continue

Four variants, the same 1 MiB + 1 B body — each a separate run, which is why the first row sits a few milliseconds away from the third row of the table above:

Scenario	First byte of body	curl total
Server stays quiet	1,004.6 ms	1.009 s
Server sends `100 Continue`	0.8 ms	0.004 s
`--expect100-timeout 0.1`	102.2 ms	0.107 s
Header suppressed with `-H 'Expect:'`	0.5 ms	0.005 s

In raw values that is 1.009073 s against 0.004401 s: a factor of more than two hundred riding on whether one header line gets answered. The first row went through four runs; the first byte of the body landed between 1,001 and 1,005 ms every time, so that one second comes out of the same place on every run. Politeness is free if the other party is polite too.

Go has the same number, but it arrives with the door shut. Go's client never adds this header by itself: when the same body went out through http.DefaultClient, the server saw no Expect at all and the request finished in 4 ms. With the header set by hand, Go waited exactly 1,000.5 ms — because DefaultTransport carries ExpectContinueTimeout: 1 * time.Second. The documentation spells out what zero means: send the body immediately, without waiting for the server to approve. Two separate teams, two languages, the same one second. No calculation produced that number; everyone copied it from everyone else.

Reading the rule changes who is at fault

Up to here I have painted the silent server as the villain. Reading the specification turned the picture around.

RFC 9110 §10.1.1 does not treat this wait as a one-sided wish; it gives orders to both ends. On the server side the language is firm: if an HTTP/1.1-or-later request carries 100-continue, the origin server MUST send either the final status code that the method, target and header fields already determine, or an immediate 100. There is a prohibition on top of that: the origin server MUST NOT wait for the content before sending the 100. Proxies get the same fork: either answer with a final status or forward the request inbound (MUST).

The client side, surprisingly, is left loose. The text names no specific duration; it says the client is not required to wait for any specific length of time, and draws its only line with a SHOULD NOT: it should not wait indefinitely. That is all it says about duration; the client has other obligations (it must not generate this expectation on a request with no content, for one), but the only bound on waiting is that SHOULD NOT. curl's one second and Go's one second are the same number two teams independently dropped into that gap.

The practical consequence: my raw listener, as the party that creates that one second, is the one breaking the rule. Against a well-behaved HTTP/1.1 server the delay never appears — it did not appear with nginx, and it did not appear with a default Node server. So when you see a one-second wait in the field, what you should be hunting is not "the price of the protocol" but a counterpart that ignores the rule: an application hook left half-finished, or an intermediary that does not care about this header. The next two sections measure exactly those two cases.

Who answers the question

The question has a recipient. Node.js's HTTP server answers it even if you do nothing: the documented wording is clear — if the 'checkContinue' event is not listened for, the server "will automatically respond with a 100 Continue as appropriate". That is what the timing showed: against a default Node server the same request finished in 0.011 s, with a 100 Continue on the wire.

The interesting part starts the moment you listen for that event. Adding a listener switches the automatic answer off and hands the responsibility to you — the docs say that too: when the event is handled, the 'request' event is never emitted. An application that adds the listener and forgets to call response.writeContinue() inside it keeps its customer waiting a full second on every large POST:

Node server	`100` arrived	Total
Default (no listener)	yes	0.011 s
`'checkContinue'` listened, no answer	no	1.012 s

This is a bug that can easily go unnoticed for months. The requests succeed, the log says 204, no alarm fires; only every upload is one second longer. Black boxes spend their worst nights without telling anyone.

nginx, for its part, answers the question itself and never passes it on. In the local lab I put nginx in front as a reverse proxy with my raw listener behind it. nginx sent the client exactly one 100 Continue, the request finished in 0.019 s — and the headers the upstream server saw contained no Expect line at all:

POST / HTTP/1.1
Host: host.docker.internal:8777
Content-Length: 1048577
User-Agent: curl/8.7.1

The source confirms this in two separate places. The ngx_http_test_expect() function in src/http/ngx_http_request_body.c writes the literal "HTTP/1.1 100 Continue" CRLF CRLF straight into the socket. And the table that builds the upstream request in the proxy module carries this row: { ngx_string("Expect"), ngx_string("") } — an empty value, which in nginx's idiom means "do not pass this header". So the application behind nginx takes no part in this negotiation and never even learns its outcome.

You can make it pass through if you want. The line proxy_set_header Expect $http_expect; overrides the empty value in that table: with it in place the upstream server saw the Expect: 100-continue line and the request finished in 0.024 s. But nginx still gave the answer — the application staying quiet cost nobody a second. Passing the header through does not put the application in charge of the negotiation, it only keeps it informed.

There is a conformance wrinkle as well. The RFC says a server that sees an expectation it does not recognise MAY respond with 417 Expectation Failed. I asked with a made-up value, Expect: 200-ok: Node returned 417 (0.005 s), while nginx said nothing at all — it processed the request normally, returned 204, and again did not pass the header upstream. The reason is visible in the code: there is no path inside ngx_http_test_expect() that produces a 417 at all — if the value is not 100-continue, the function simply returns NGX_OK, and the expectation is silently ignored. Behind nginx a 417 can never be born; nginx does not produce one, and your application is not allowed to either.

A megabyte of profit

Looking at the tables so far, a reader could fairly ask: if the price is one second, why does this mechanism exist at all? To put a number on the answer I defined a small location in nginx — client_max_body_size 1k — and sent it the same 1 MiB + 1 B body.

Request	Result	Bytes uploaded	Time
With `Expect: 100-continue`	`413`	0	0.002 s
Suppressed with `-H 'Expect:'`	`413`	1,006,159 – 1,048,577	0.003 s

In all four runs the request that asked permission sent zero bytes; the one that did not ask sent 1,006,159, then 1,023,167, then 1,048,577, then 1,048,577 bytes. The lower bound moves from run to run, because there is a race between nginx writing its 413 and closing the connection and curl stopping its write; what does not move is the order of magnitude. The same rejected request: one sent not a single byte, the other put a megabyte on the wire and had the door slammed in its face halfway down the road. This mechanism is not about speed; it is about never sending bytes that are going to be refused.

The same profit can be measured at the application layer. I wrote one handler for Go's server and ran it two ways: one reads the body, the other returns 401 without touching it.

Go handler	Result	Bytes uploaded	Time
`io.Copy(io.Discard, r.Body)`	`204`	1,048,577	0.002 s
`WriteHeader(401)` without reading the body	`401`	0	0.001 s

The second row's secret is one sentence of documentation: the Go server sends the 100 (Continue) header on the first read from the request body. In Go, granting permission arrives as a side effect of starting to read the body. If you want to refuse, what you have to do is nothing at all — write the final status without touching r.Body. Early rejection is just a WriteHeader written three lines earlier.

Scaling it up is easy: at this blog's 16M limit, on a 1 Mbps uplink, pushing a doomed file all the way up takes roughly two minutes. I did not measure that; I derived it from the limit and the link speed — but even derived, it says enough about the bandwidth running into a drain on the server side. It resembles the lesson from the timeout semantics piece published here yesterday: where you think the client is keeping time, something else may be getting measured. There, a "timeout" turned out to be an hourglass set between two I/O events. Here, "one second" is not a calculation but a courtesy period.

Over HTTP/2 the question hangs in the air

Everything so far ran over HTTP/1.1. Two independent code bases say that is no accident.

nginx's ngx_http_test_expect() bails out quietly, before it even looks at the value, in these cases: the request version is below 1.1, r->stream != NULL (that is, it is an HTTP/2 stream), or r->connection->quic != NULL (HTTP/3). curl takes the same side: addexpect() only adds the header on its own while httpversion == 11. So both ends treat this feature as belonging to HTTP/1.1 — even though RFC 9113, which defines HTTP/2, shows in §8.8.5 exactly the case of answering a request that carries Expect: 100-continue with :status = 100. The specification permits it; the implementations do not use it.

I also took this measurement against my own site. I sent a POST with a 1 MiB + 1 B body to a path that does not exist (my site returns 403 from the edge for requests like this; the Server: cloudflare and CF-RAY response headers name who answered, so the three rows below describe that edge rather than my own nginx):

Request	`Expect` sent	`100` arrived	Uploaded	Time (median of 4 runs)
Default (HTTP/2)	no	—	1,048,577 B	1.03 s (0.72–1.77)
`--http1.1`	yes	yes	1,048,577 B	1.01 s (0.71–1.14)
HTTP/2 + header by hand	yes	no	1,048,577 B	1.79 s (1.69–1.95)

These timings crossed the public internet, so they are noisy; the ranges in brackets show it. The solid result is not a number but the "no" in the third row: with the header set by hand over HTTP/2, no answer came, curl printed Done waiting for 100-continue, spent its full second, then sent the body regardless, and the request still ended in 403. Gain zero, cost one second. The second row got its 100 Continue but made no profit either: after the edge said "go ahead" it still returned 403, so the entire megabyte was wasted. For this mechanism to pay off, the party granting permission has to know about the refusal in advance.

The old-version side is closed off by the same logic. I put the Expect header by hand on an HTTP/1.0 request to the local nginx: no answer, a full 1.026 s. The same request over HTTP/1.1: 0.019 s. RFC 9110 §10.1.1 actually commands this — a server MUST ignore that expectation on an HTTP/1.0 request. nginx complies to the letter, and the waiting side pays for that compliance with one second.

Where my own stack stands

I looked at my own blog's live configuration and found something other than what I went looking for. The expected part first: client_max_body_size for this domain is 16M, so an upload over 16 MiB will be refused — over HTTP/1.1, if permission was asked, without a single byte of body. There is not one line in my application code mentioning checkContinue or writeContinue; it would not help if there were, since the nginx in front never passes that header upstream. This negotiation belongs to those two layers, not to my application.

What I did not expect was the version line: nginx/1.30.0, the release from 14 April 2026. The same stable branch reached 1.30.6 on 14 October, and the changelog in between carries five separate *) Security: entries — two buffer overflows in a worker process, a use-after-free with asynchronous file I/O, an uninitialised key in TLS session ticket rotation, and one where part of an unknown HTTP/3 frame can be taken for the request body and slip past the client_max_body_size limit. That last one does not touch me: the blog vhost has no quic listener, and all traffic is HTTP/1.1 and HTTP/2 over TLS. The others are open work.

I was going to close this section with "nothing to change". I am not closing it that way. There is a seven-month patch debt on that server, and a one-line version check run for this article is what surfaced it — I am not doing the upgrade here, because patching a live system should not be the side errand of an article run; it goes in its own maintenance window. The second risk is smaller and entirely in my own hands: if I ever add an upload endpoint, I might leave a 'checkContinue' listener half-finished. The 1.012 s row above is the invoice for that mistake.

A decision frame

The questions worth asking about this mechanism in your own setup, in order of weight:

Who refuses the upload? If the party that knows about the refusal early (a size limit, authentication, a quota) is also the party that can grant permission, the saving is real. For an application that only learns of the refusal after reading the body, 100-continue is just one round trip of latency.
Which version is on the wire? Over HTTP/2 or HTTP/3 this header will most likely go unanswered: gain zero, cost one second. Deliberately keeping an upload endpoint on HTTP/1.1 is a defensible choice.
Who sits in between? With a proxy like nginx in the path, the proxy answers the question and does not pass it on. If you think your application returns 417, look at the proxy's table first.
Does your client even ask? curl asks above 1 MiB or when the length is unknown; Go does not. Rather than assuming what your own client does, time it once against a silent server — the difference is a whole second, so it is visible to the naked eye.
Is the uploader a browser? You cannot send this header from one. In my own browser, building new Request('/x', {method:'POST', headers:{'Expect':'100-continue'}, body:'...'}) and listing its headers showed no Expect at all; putting the same value into a plain Headers object keeps it, so the stripping happens when the request is constructed. Uploads through fetch() and XHR never engage this mechanism; the saving exists only on paths where you write the client yourself — curl, SDKs, server-to-server calls.
Is the thing you measured yours? With a cloud edge, a load balancer or a WAF in front, the 100 Continue probably comes from them. To learn what your own server says, you have to look at it directly.

The closing number is this: an unanswered courtesy question costs one second, an answered one saves a megabyte. Finishing the piece, I got curious about the mechanism's age, looked up its history, and found the opposite of what I expected. The first text of HTTP/1.1, RFC 2068 from January 1997, has neither the Expect header nor the 100-continue token — but the wait itself is there, and it is unconditional. §8.2 tells a server receiving such a request to "either respond with 100 (Continue) status and continue to read from the input stream, or respond with an error status" (MUST), and tells the client to send the header fields first and then wait for "either a 100 (Continue) response ... or with an error status" (MUST). RFC 2616 §14.20, in 1999, removed that obligation and put a header the client has to ask for in its place.

So the mechanism is twenty-nine years old, and its one big change was a retreat rather than an expansion: a rule everyone had to obey became a request anyone may make. In the twenty-seven years since, HTTP/2 left even that request unanswered. The most expensive thing in a protocol is not a wrong rule; it is a right question nobody answers.

Official Sources

Output Waits in the Pipe: Why stdbuf Often Isn't Enough

Mustafa ERBAY — Sat, 14 Nov 2026 06:31:36 +0000

You run a script in your terminal and the lines scroll past. You then add | grep error
to filter it, and the screen stays empty. The program did not hang and did not get stuck;
it is running, and it may well have finished. It just isn't talking.

The thesis of this article: your program does not decide when its output becomes visible.
The destination decides — a terminal, a pipe, or a plain file. And stdbuf, the tool
everyone reaches for to undo that decision, worked in only two of the seven runtimes
measured below. The rest keep their own buffer and never ask libc about it.

This blog has no shortage of "I turned the knob and the system ignored me" pieces. The
difference here: there is no knob to turn. Nothing is configured, no default is changed —
the same binary, without a line altered, behaves like two different programs depending on
what it is attached to.

Everything below was measured on my own machine (MacBook Pro M4 Pro, macOS 26 / Darwin
25.6.0, arm64) and, for comparison, inside Docker on Debian 13 / glibc 2.41 / aarch64.
I ran every line three times; the table values are the median of three runs — and on this
topic I got lucky, because all three runs came out identical.

How I set up the measurement

To measure the claim "output disappears," you have to make the loss visible. The method:
a program writes a fixed number of lines with printf, never calls fflush, then waits
forever. While it waits, I kill it with SIGKILL. SIGKILL cannot be caught, so whatever
sits in the buffer dies there. The number of bytes that reach the other end of the pipe
is the part that was genuinely flushed. Every byte that does not arrive is a log line you
would have lost in real life.

/* prod2.c — every line is exactly 10 bytes, no flush */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv) {
  long n = atol(argv[1]);
  for (long i = 0; i < n; i++) printf("%09ld\n", i);
  pause();                 /* whatever is in the buffer stays there */
  return 0;
}

Pinning the line length at 10 bytes made things easier: 1,638 lines is exactly 16,380 bytes,
four bytes under the 16 KiB boundary. Those four bytes will show up later in a place I did
not expect.

The reading side is a small hand-written Python script: it reads the pipe without blocking,
records when the first byte arrives, counts the total, and kills the child when time is up.
One subtlety: the measuring tool itself must not be buffered, or you end up
measuring your own noise. I read from the raw file descriptor with os.read, not through
sys.stdin.

The destination makes the call

I attached the same compiled binary to two different destinations; the plain-file case gets
its own table shortly. The program did not change; not one line of code differs.

Destination	16,380 bytes written → arrived
Real terminal (pty)	18,018 bytes (all of it)
Pipe (`\	cat`)

That 18,018 looks odd at first because it exceeds what was written. The reason is innocent:
a pty turns every \n into \r\n, which is 1,638 extra bytes for 1,638 lines.
16,380 + 1,638 = 18,018. So attached to a terminal, everything got through. Attached
to a pipe, nothing did.

The difference comes from two separate decisions libc makes: the buffer's mode and its size.

The mode decision is written down in POSIX. The
stdin/stdout/stderr definition
says stdin and stdout shall be fully buffered "if and only if the file descriptor
associated with the stream is determined not to be associated with an interactive device."
So a terminal flushes line by line and a pipe waits until it is full. The standard leaves
the definition of "interactive device" to the implementation.

The size decision is not in the documentation at all. FreeBSD's
setvbuf(3) page only says
an "optimally-sized buffer is obtained" and gives no number. To learn the number you have
to read Apple's own libc source,
makebuf.c.
The line that sets the size is this:

fp->_blksize = *bufsize = st.st_blksize > MAXBUFSIZE ? MAXBUFSIZE : st.st_blksize;

The buffer size is the destination's st_blksize, read via fstat. If fstat fails or
st_blksize is zero, it falls back to BUFSIZ. Line buffering is enabled only for
character devices where isatty returns true — in the source the couldbetty check is
(st.st_mode & S_IFMT) == S_IFCHR, so a pipe (a FIFO) can never pass through that gate.

This is easy to confirm. Here is the st_blksize of each destination:

A caveat on st_blksize: when I asked statx about O_DIRECT alignment, the result was that
the story this same field tells does not hold across five filesystems.
Here I am not using it as a promise about alignment, but as the number libc reads when
choosing a buffer size — and in that role it matches exactly.

Destination	`st_blksize`	Measured flush quantum
Plain file (APFS)	4,096	4,096
Pipe / FIFO	16,384	16,384
`/dev/null` (character device)	65,536	—

And measuring the quantum right at the boundary lines up exactly:

### written to a PIPE, then SIGKILL
  written= 16380 bytes -> arrived=     0
  written= 16390 bytes -> arrived= 16384
  written= 32760 bytes -> arrived= 16384
  written= 32770 bytes -> arrived= 32768
  written= 50000 bytes -> arrived= 49152

### written to a PLAIN FILE, then SIGKILL
  written=  4090 bytes -> in file=     0
  written=  4100 bytes -> in file=  4096
  written=  8190 bytes -> in file=  4096
  written=  8200 bytes -> in file=  8192
  written= 16380 bytes -> in file= 12288

There are those four bytes. Writing 16,380 sent zero bytes into the pipe; writing 16,390
sent 16,384. The difference is ten bytes, one single line. That is how fragile the
relationship is between how much a program says and how much the reader hears.

One note: MAXBUFSIZE in the source is defined at two different values — 4 KiB if
FEATURE_SMALL_STDIOBUF is compiled in, 16 MiB otherwise. Since the buffer on a pipe came out at 16,384
bytes, the small variant is not enabled in the build Apple ships. I did not read
that from the source; I inferred it from the measurement, and I should say that it is an
inference.

stdbuf fixed two of seven runtimes

So far this was C. But nobody writes their log-producing job in C. I ran the same test —
16,380 bytes, to a pipe, no flush, then SIGKILL — across seven different runtimes, and
repeated each one with stdbuf -o L added. You will see nine rows in the table, because
Python was measured in two versions and Go through two different write paths.

macOS has /usr/bin/stdbuf, which surprised me; it is not GNU coreutils' stdbuf but
FreeBSD's version. The man page on my machine says so outright: the utility "first
appeared in FreeBSD 8.4", the idea came from Padraig Brady in GNU coreutils, and Jeremie
Le Hen did the FreeBSD implementation. GNU's own documentation says the tool is installed
only "on platforms that use the Executable and Linkable Format (ELF) and support the
constructor attribute" — macOS, being Mach-O, does not fit that description, so Apple
took the BSD equivalent in its shell_cmds package.

Runtime	Raw	With `stdbuf -o L`
C (stdio)	0	16,380
awk (BSD, macOS)	0	16,380
python3 3.9.6	0	0
python3 3.13.5	0	0
perl 5	8,192	8,192
ruby	8,200	8,200
go (`bufio.NewWriter`)	12,288	12,288
go (`os.Stdout`)	16,380	—
node 22.16	16,380	—

There are a few things in that table.

First: stdbuf only worked for C and awk — it had no effect in four of the six buffered
runtimes I added stdbuf -o L to.

To understand why, I looked at what the tool actually does, and did not find what I
expected. FreeBSD's man page
says it relies on a preload library called libstdbuf(3); on macOS there is no such
library. Comparing the environment of a process launched under stdbuf -o L turned up
exactly one difference: _STDBUF_O=L. No DYLD_INSERT_LIBRARIES, no
/usr/lib/libstdbuf.dylib.

The thing that reads that variable is inside the very file quoted above: __senvbuf() in
makebuf.c calls getenv("_STDBUF_O"), and __smakebuf() invokes it immediately after
__swhatbuf(). So on macOS stdbuf injects nothing; it just leaves an environment
variable behind, and libc itself makes the decision.

That also explains the limit in one sentence: the variable only changes the initial
decision libc makes for the stream. Python, Perl, Ruby and Go's bufio do not get their
buffer from libc in the first place — they build their own layer, so the variable never
reaches them. The GNU coreutils documentation states the same limit from its side: dd is
unaffected because it does not use FILE streams, and tee because it adjusts its own
streams' buffering.

Second: there are five different answers for the same 16,380 bytes — 0, 8,192, 8,200,
12,288, 16,380. One machine, one pipe, one byte count. Perl flushes at 8 KiB and Go's
bufio at 4 KiB (the Go source has defaultBufSize = 4096 and NewWriter passes it
directly — 12,288 = 3 × 4,096). None of them uses the 16,384 that libc picked for the pipe.

Third: Go's os.Stdout and Node are unbuffered. Both delivered everything. So "the
buffering problem" is not universal; it is a property of your language choice.

Let me also be open about the counting, because "two of seven" looks the way it does partly
because the denominator is one of my own picking. Counted as languages there are seven (C, awk, Python, Perl,
Ruby, Go, Node); the table has nine rows because Python appears in two versions and Go
through two write paths. I added stdbuf to the six that buffer, and it worked on two. But
honesty requires the other half: stdbuf was only ever designed for programs using libc
stdio, and within that set it did not fail — in the third section it goes two for two on
grep and cut. So "two of seven" measures not the quality of the tool but the fact that
most of a 2026 log pipeline is no longer C. Read the title's "often isn't enough" that way:
the tool is not broken, its scope is narrow.

Python has a fix, and that fix is not stdbuf. The
-u flag or the PYTHONUNBUFFERED environment variable settles it: the same test under
python3 -u delivered all 16,380 bytes. The Python documentation says -u will "force the
stdout and stderr streams to be unbuffered" and notes that since 3.7 the text layer is
unbuffered as well. That word "layer" is not idle: in Python sys.stdout is a
TextIOWrapper with a BufferedWriter underneath it. Two buffers stacked. That is why
Python's flush boundary is not a clean power of two, and why it also shifts with the
destination:

### macOS, python3, to a PIPE — threshold found by bisection
  written= 16390 bytes -> arrived=     0
  written= 24570 bytes -> arrived=     0
  written= 24580 bytes -> arrived= 16380     <-- first delivery
  written= 65530 bytes -> arrived= 49140
### macOS, python3, to a PLAIN FILE
  written=  8190 bytes -> in file=     0
  written=  8200 bytes -> in file=  8190

Rather than guess the threshold I bisected for it: at 24,570 bytes into a pipe the output is
still zero, and the first delivery lands at 24,580. That number should look familiar —
24,576 is exactly 8 KiB + 16 KiB. Python's 8,192-byte text layer and the 16,384 libc chose
for this pipe appear to be stacked on top of each other. I say "appear" because I did not
trace the interaction of the two layers through the CPython source; what I have is a
measurement and an addition that works out. That is not enough to assert the mechanism, but
it is enough to say where the threshold sits.

The real culprit is the middle of the pipeline

The above is the "what if the source misbehaves" scenario. The far more annoying case in
real life is this one: the source is perfect and the filter in the middle is to blame.

To measure that I wrote a producer that calls fflush on every line and emits 20 lines per
second. There is no flaw on the source side. Then I put a single filter in the middle, waited
two seconds, and counted how many lines reached the end of the pipeline. A healthy value
is 38.

Pipeline	Lines reaching the end in 2 s
no filter	38
`\	grep satir`
`\	sed 's/x/y/'`
`\	cut -c1-12`
`\	tr a-z A-Z`
`\	awk '{print}'`
`\	jq -c .`
`\	cat`
`\	tee /dev/null`
`\	grep --line-buffered satir`
`\	sed -l 's/x/y/'`
`\	awk '{print; fflush()}'`
`\	jq --unbuffered -c .`
`\	stdbuf -o L grep satir`
`\	stdbuf -o L cut -c1-12`

Six filters passed not a single line in two seconds. grep, sed, cut, tr, awk, jq
— that is every tool you would reach for to filter a log. The producer was flushing each
line immediately; the filter poured them into its own 16 KiB container and waited. I tested
jq separately with a JSON-producing source, since that is the standard way to filter
structured logs; the result was the same.

That cat and tee pass things through is no coincidence. The GNU documentation already
told us tee adjusts its own buffering, and here we see it in the measurement too. If you
reach for tee to both watch a log on screen and write it to a file, you have picked the
right tool without knowing it.

The fix side is cheap, fortunately, and the bottom half of the table shows it. Most tools
have their own flag: grep --line-buffered, sed -l (BSD), fflush() inside awk,
jq --unbuffered. All four went through the same test and all four passed the 38 lines. I checked the GNU
side separately in a Linux container: there the flag is sed -u, and it works the same way
— BSD's -l and GNU's -u do the same job with different letters, and you have to notice
that when writing a portable script.

For tools without their own flag, like cut, stdbuf -o L steps in — because those use
plain stdio, which is exactly stdbuf's area of effect. The last two rows of the table
confirm it: stdbuf works here, while a section ago it did nothing for Python. A tool
working in one place and failing in another is not random; the boundary is whether the
program left its buffering to libc.

Let me also record a mistake of my own here, because the correction was instructive. In my
first attempt the wait was done with system("sleep 30"), and awk delivered all 16,380
bytes as though it used no buffer at all. My first thought was "so awk is clean"; then I
realised my waiting method was doing the flushing — the awk on my machine empties stdout
before running system(). I had assumed this was a standard requirement, but POSIX's
awk page contains no
such rule: the only flushing requirement on that page belongs to the fflush function, and
the system() entry says only that the command is executed. So this is the behaviour of the
awk on this machine, not a mandate of the standard. Once I changed the waiting method, awk went
to zero bytes like everyone else.

The general lesson: your measuring tool can change the thing it measures. Ask your own test
rig the same question.

"It worked on Linux" is not evidence

I copied the same source files into Debian 13 inside Docker and compiled them with gcc.
glibc 2.41, aarch64. Not one line of code changed.

Measurement	macOS / Darwin 25.6	Linux / glibc 2.41
Pipe `st_blksize`	16,384	4,096
C stdio, 16,380 bytes → arrived	0	12,288
C + `stdbuf -oL`	16,380	16,380
python3 3.13.5 → arrived	0	8,200
python3 + `stdbuf`	0	8,200
python3 `-u`	16,380	16,380
awk / gawk	0	12,288
Plain-file quantum	4,096	4,096

Same program, same byte count, same pipe: zero on macOS, 12,288 on Linux. The cause is a
single number — Linux reports a pipe's st_blksize as 4,096 and macOS as 16,384. libc
applies the same rule on both; the rule is identical and the outcome still differs fourfold.

I think the most practical conclusion from that table is this: if your CI runs on Linux and
you develop on a Mac, "I can see the logs locally" and "the logs show up in CI" are two
independent facts. The reverse holds too. Personally I would put running this test on both
platforms on the acceptance checklist.

Node's separate trap: no buffer, but still a loss

Node came out clean on the buffering side of this article. But it has a hole of its own, and
it is documented. The process.stdout note says writes
are synchronous or asynchronous depending on the destination: files are synchronous on both
platforms, terminals are synchronous on POSIX, and pipes and sockets are asynchronous on
POSIX. The warning follows immediately — synchronous writes avoid problems such as output
"not written at all if process.exit() is called before an asynchronous write completes."

I tried this, and my first attempt failed. Writing 10 MB line by line in a loop and then
calling process.exit(0) lost nothing at all; all 10,485,760 bytes arrived. I could not
reproduce what the documentation warned about.

To produce the loss I had to change the shape of the write. Giving the same 10 MB in a
single write call and exiting immediately:

  1,048,576 small writes in a loop + exit  -> 10,485,760 bytes  COMPLETE
  one 10,485,760-byte write + exit         ->     65,536 bytes  LOST

The 65,536 bytes that arrived are exactly 64 KiB: as much as the kernel accepted into the
pipe buffer in one go. The rest was waiting in Node's queue, and process.exit() did not
wait for it. That is a 99.4% loss.

The loop version survives because each small write can drain as the reader consumes it. So
to produce this bug the write has to be too large to fit into the pipe at once. I am writing
this down because you are likely to try the same thing, and if you fail to reproduce it on
the first attempt I do not want you concluding there is no problem. The documentation is
right; its condition is just narrow.

The state of my own infrastructure

While writing this I asked my own blog repository the same question, because you should look
at your own house before handing out advice. The results:

I have 17 GitHub Actions workflows. In eleven of them, across 31 lines, a grep, sed, awk, cut or tr is fed from a pipe. But that first count of 31 was misleading; what matters is where the filter sits. The number of lines with another pipe after the filter — genuinely in the middle of the pipeline — is 10: cache-purge.yml 1, content-generate.yml 6, deploy-only.yml 1, translate-archive.yml 2. In the other 21 the filter is the last link in the chain.
Searching .github/, scripts/ and deploy/ for stdbuf, --line-buffered, sed -u/-l and PYTHONUNBUFFERED: zero hits. No precaution anywhere the pipelines are actually built. (The repository as a whole has exactly one PYTHONUNBUFFERED, inside an example systemd unit quoted in an older article — prose, not code.)

The situation is not alarming, and honesty requires writing down why. The 21 last-link lines
fall outside this article's subject: there is no reader waiting behind the filter, the output
goes into a variable or a file, the stream closes when the process ends, and closing flushes.
Buffering only turns into data loss when the job is cut off partway through — and being
cut off, as you will see below, is harsher than I assumed.

The one place that caught my attention was translate-archive.yml. There a job with a
60-minute timeout captures the output of a long translation loop with 2>&1 | tee translate.log.
If it times out, GitHub cancels the job. According to the table above, that pipeline is
safe: tee is one of the two tools that pass data through, and Node is unbuffered anyway.
But had I written grep there instead of tee, the log file would have been empty the
moment a 60-minute job was cut off — which is exactly the moment I would need to know why
it was cut off.

This is also a reminder that log loss is not a new subject for me. I previously measured
losing 33,879 lines in the gap that copytruncate dismisses as "a very small moment".
The mechanism in that article was completely different — there the loss came from logrotate
truncating the file, here it comes from libc's buffer. What they share is this: in both
cases, the component causing the loss reports no error at all. I have also measured the pipe
itself from another angle:
a neighbouring container filling the quota, so my new pipes came back at 8 KiB instead of 64 KiB,
happened just as silently. The 65,536 bytes below are that same 64 KiB.

"What if I stop it gently?" — no

Here I had to test an assumption of my own. When a job is cancelled, SIGTERM arrives before
SIGKILL, and it seems reasonable to think the polite signal would shut things down properly
and flush the buffer. The measurement says otherwise:

16,380 bytes written to a pipe, then a signal:
  SIGTERM -> arrived=0        SIGINT -> arrived=0
  SIGKILL -> arrived=0        SIGHUP -> arrived=0

All four are zero. The reason is simple but easy to miss: a signal's default disposition
is to terminate the process, not to call exit(). Because exit() is never called, atexit
handlers do not run and stdio buffers are not flushed. So what saves the buffer is not the
politeness of the signal but the program exiting properly under its own control. A process
that catches SIGTERM and leaves via exit() keeps its data; one that does not catch it
never does.

That leaves my translate-archive.yml concern standing: when GitHub cancels the job, I
cannot expect a polite signal to rescue me.

In a container, stdout is already a pipe

If you are reading this thinking "but I don't use pipes," look again. Inside a container
stdout is a pipe to the logging driver — what you see with docker logs or
kubectl logs is the far end of it. So the runtime table above applies as-is to every
containerised application: a service written in Python goes silent in a container unless it
is given -u, and that is the classic case where hours get burned on "the logs aren't
coming through." My own blog is served from a container too.

There is also a free way out: stderr. POSIX says stderr shall not be fully buffered when
opened, and the measurement agrees — writing the same program's output to stderr delivered
all 16,380 bytes immediately. Sending diagnostic lines to stderr is the cheapest fix
available, and it needs no flag at all.

A decision framework

If output is delayed in a pipeline, check these in order:

Identify the destination. Checking st_blksize with fstat is not overkill: a pipe is 16 KiB (macOS) or 4 KiB (Linux), a plain file 4 KiB. The delay you should expect comes from there.
Look for the culprit in the middle of the pipeline. Before changing the source, pull the filters out one at a time and retest. In my measurement the source was flawless and grep was the blockage.
If the tool has its own flag, use it. The ones verified here: grep --line-buffered, sed -u (GNU) / sed -l (BSD), fflush() inside awk, jq --unbuffered, -u or PYTHONUNBUFFERED=1 in Python, $| = 1 in Perl. In Go, calling Flush() on a bufio.Writer is your responsibility anyway. These are more reliable than stdbuf because they touch the tool's own buffer.
Trust stdbuf only for tools that use stdio. It works on classic filters like cut, tr and grep. It does not work for Python, Perl, Ruby, Go or dd, and it is unnecessary for tee.
If you cannot change the source and it has no flag, fake a terminal for it. This article's first measurement is already the proof: the same binary gave 18,018 bytes on a pty and zero on a pipe. script -q /dev/null <command> or expect's unbuffer attaches a pseudo-terminal, libc sees isatty as true, and the stream switches to line buffering. For third-party binaries you cannot open, this is usually the only lever you have.
Remember that turning the buffer off has a price. I wrote 10,485,760 bytes to /dev/null and timed it (five runs, median): 0.05 s with the default full buffering, 0.41 s with stdbuf -o L, 0.42 s with stdbuf -o 0. Line buffering is roughly eight times more expensive here. Do not carry that number into production as-is: it was measured against a /dev/null with no reader, and the figures change behind a real consumer. What does transfer is the conclusion — line buffering is not free, so apply it where you watch output live, not everywhere.
Do not use process.exit() in Node. Set process.exitCode and let the process exit naturally. The loss recorded here was 99.4%.
Do not count on the signal being gentle. SIGTERM, SIGINT, SIGHUP, SIGKILL — all four killed the process without flushing. The only thing that saves the data is the program catching the signal and exiting under its own control.

One last diagnostic habit: when you see a process not talking, question the pipeline before
you question the process. I got the same prod2 binary to produce 18,018 bytes and zero
bytes within the same minute, without changing it. The program was innocent; it had done the
same thing both times. The only thing that differed was who its words were going to.

Buffering is a well-meant invention: a few large writes are cheaper than many small ones.
The price of that good intention is not that good news gets held back — it is that bad news
does. The last twenty lines explaining why a job crashed may still be sitting in a buffer at
the exact moment of the crash.

Official Sources

Apple Libc — stdio/FreeBSD/makebuf.c — the buffer size coming from st_blksize, the BUFSIZ fallback, the isatty line-buffering decision and the TTYBUFSIZE cap are all in this file.
Python — Command line and environment (-u, PYTHONUNBUFFERED) — documents that -u makes stdout/stderr unbuffered and that since 3.7 this covers the text layer too.
Node.js — process and the process I/O note — the table of synchronous/asynchronous writes per destination, and the process.exit() warning.
Go — the bufio package — documents that NewWriter returns a buffer of "the default size"; the numeric value (defaultBufSize = 4096) is in the package source.

In August I Cited 328 Publishers, in October 78

Mustafa ERBAY — Fri, 13 Nov 2026 16:08:00 +0000

The least-read part of any post is the source list at the bottom. For a long
time I looked at mine like a badge of conscience: if the links are still
standing, the work is sound. On 12 October those links went through a
liveness count:
seven returned 404 and seventy-five had moved to a different address.
That measurement told me how healthy the list was. What it did not tell me
was this: where does the list look?

This time I counted that. I did not like the number — and in its first form
it was wrong, and the wrong part was not only the number. My explanation of
the cause did not hold either.

What I counted, and what I could not

The repository holds 1,415 Turkish posts (measured on commit 67f09d8c, with
this post left out of the count). 325 of them carry an official-sources
section, and those sections contain 2,189 external links in total. The
remaining 1,090 posts are not source-free, they are source-unrecorded: they
were produced before that section settled into the writing as a habit.

I am stating the denominator up front, because the denominator of a ratio is
its most easily hidden part. 2,189 links belong to 325 posts. Nothing more.

The first table came out in the wrong unit

I ran the first query and the answer looked perfectly clear: 30.2% of the
links — 661 of them — came from a single domain, github.com. Exactly the
kind of bad number that makes you want to write a post. I was about to put
down the sentence "a third of my sources come from one place" when I noticed
the unit error.

github.com is not a publisher, it is a landlord. The URI standard draws
this line from the start: of the authority part of an address it says
"governance of the name space defined by the remainder of the URI is
delegated to that authority," and the host is only one subcomponent of that
authority. GitHub's own documentation says repositories are owned
"individually" or shared within an organization. So the distance between
github.com/torvalds/linux and github.com/systemd/systemd is no smaller
than the distance between nginx.org and sqlite.org.

I split those 661 links by owner: 157 separate owners, 188 separate
repositories. The top ones, in order: torvalds/linux (202),
systemd/systemd (57), git/git (42). There was no monopoly in the middle;
there was a folder I had been counting wrong.

I changed the unit and counted again: for hosting sites the owner, for
everything else the domain, each treated as one "publisher."

At the publisher level the table reads: 546 distinct publishers, the
largest being github.com/torvalds with a 9.2% share, and the top ten
adding up to 34.9%. On top of that, 350 publishers appear exactly once across
the whole corpus. That distribution is not bad; it even has a long tail worth
being proud of.

Then I split the same table by calendar month, and the pride ended there.

A quarter of the publishers

Month	Posts with sources	Links	Distinct publishers	Linux share	Median publishers per post
2026-05	1	7	2	14.3%	2
2026-07	18	151	116	0.0%	7
2026-08	95	620	328	0.6%	4
2026-09	95	751	131	32.1%	4
2026-10	86	490	78	27.6%	3
2026-11 (first 13 days)	30	170	47	18.2%	3

Note the first column's name: these are not all the posts published that
month, only the ones carrying a source section. In July the gap is a chasm —
200 posts went out that month and only 18 of them have a source section.
After August the coverage is nearly complete: 95 of August's 97 posts, 95 of
September's 96, 86 of October's 86. The single post on the 2026-05 row is
there so the table does not lose it; a reader adding the columns should land
on 325 and 2,189.

In August I published 95 posts carrying a source section, and they rested on
328 separate publishers. In October I published 86 posts, and they rested
on 78 separate publishers: a basin 4.2 times smaller.

That last column matters too: the median number of publishers per post fell
from seven to three, yet three is still a reasonable number. Look at any
single post and you see work that was "verified from a few different places."
The narrowing is invisible in one post; it shows up in the corpus. Audit a
library book by book and you will not find a gap on any shelf; step back and
you see that every shelf comes from the same press.

Comparing August with October is apples to oranges

When I first wrote this table the cause was ready at hand: in September the
topic queue ran dry, the pipeline started picking topics on its own, and
every topic it picked led to the kernel. A tidy story. What was missing: that
wide 328 of August was not a product of my attention.

I looked at the message of the first commit that added each post. Of
August's 95 posts, 72 arrived with feat: auto-generated content — the
old bulk pipeline, running over a topic calendar written in advance. In
October the number of posts from that pipeline is zero; 84 of the 86 come
from today's agent pipeline. So August and October are not two months of the
same author, they are months of two different production systems.

The old pipeline's breadth was not curiosity either. 48 of August's 95
posts carry exactly eight sources — median eight, a narrow spread. The
pipeline applied a fixed quota per post, and the topics on that calendar were
product write-ups; each product led to its own documentation, so each post
brought a new publisher by construction. October's median is five, spread
from three to thirteen.

The honest comparison, then, is the one inside a single system:
September 131 publishers, October 78. That is still a narrowing — 1.7
times — but not 4.2. The number in the title is not wrong, it is incomplete:
part of the gap between August and October is explained by the authoring
regime, the rest by topic selection.

Separating the two did not suit me, which is why I am writing them out
separately.

Four separate publishers, one tree

The publisher count is still too optimistic a measure. Look at the first four
names on my list: github.com/torvalds (202 links), git.kernel.org (91),
docs.kernel.org (87), www.kernel.org (32). They count as four separate
publishers. Together they are 412 links, that is 18.8% of all my sources.

All four sit around the same project. The kernel's patch-submission guide
tells you to clone the mainline tree from
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git: the same
tree you see on GitHub under the name torvalds/linux. There is one place
here where I have to be careful — I could not find an official sentence
on kernel.org's pages saying "the GitHub repository is a mirror," so I am not
writing that. The evidence I do have is more modest but sufficient: the
official clone address and the GitHub repository name the same tree.

My diversity counter was celebrating four doors into the same upstream as
four separate sources. September's jump in the Linux share, from 0.6% to
32.1%, sits exactly here.

Why the gate never complained

This blog's source discipline is not an intention, it is code.
source-policy.mjs went live on 17 August 2026 and asks two things of every
post: at least 3 unique official sources, at least 2 unique official
hosts.

The gate did not complain even once in October or November. And it was right
not to: of the 116 posts published in those two months, 45 sit at exactly 2
hosts and 51 at 3 — 82.8% living between the threshold and one step above
it. The minimum became the norm over time; that is how gates usually work,
nobody collects bonus points.

I had already written about scale outstripping the gate on this blog: when
forty-four posts in a row came from a single category
no indicator turned red, and the lesson there was "no gate puts two posts
side by side and asks whether they resemble each other." Six weeks later I am
measuring the second face of the same gap: back then the composition had
narrowed, this time the evidence base.

There is a second trace in the same place: the allowlist of permitted sources
has 71 entries, and 62 of them have been used so far. The nine never used
are: canonical.com, golang.org, haproxy.com, haproxy.org,
mistral.ai, traefik.io, truenas.com, typescriptlang.org, vite.dev.
My intention was broad when I wrote that list; my practice never got there.

Why I did not trust the "new publisher" count

My first pass found a far more striking series: the number of publishers
encountered for the first time each month. 115 in July, 298 in August, 95 in
September, 24 in October, 12 in the first 13 days of November. A decline
ready-made for a headline.

I did not publish it, because that series falls on its own. After reading 546
publishers, running into a new one gets mechanically harder; saturation draws
the same curve as narrowing. A measurement agreeing with your thesis does not
make it evidence.

With the same suspicion I asked about monthly concentration by share rather
than by count: the largest publisher's share in August is 3.1%, with
docs.github.com. In September it rises to 18.2% with github.com/torvalds,
in October 11.4% with git.kernel.org, in November 10.0% with
www.rfc-editor.org. After August no month returned to that scatter.

Where I tried to refute myself

The weakest point of this measurement is that it only counts the source list.
Yet the documents that are valuable but not on my allowlist get cited as
inline links in the body of the post. When I say "my basin narrowed," I
may simply never have counted the part of the basin that lives in body text.

I counted it. Across all body text outside the source sections there are
306 external links going to 142 publishers; 58 publishers appear only
in body text and never made it into a source list. The body does carry an
uncounted basin.

But that basin is not widening. August's body text holds 78 links and 41
distinct publishers; October and November's body text holds 97 links and 44.
While the source list fell from 328 to 78, the body went from 41 to 44. The
body does not compensate for the source section's narrowing; it sits in its
own steady narrowness.

Had I skipped this check the post would still stand, but a reader would
rightly have asked "did you look at the body?"

If you want to repeat the measurement

I am writing the method as it is, because a reader has to be able to
recompute this table. I matched the source heading with the gate's own
regular expression — case-insensitive, accepting the English variants
alongside Resmî Kaynaklar. That matters: one post in the corpus has its
heading written as ## Resmî kaynaklar, in lower case, and carries eight
links. Counting case-sensitively would have dropped that post and shown
August as 326 rather than 328 — I would have been measuring my own policy
with a stricter definition than the policy's own. After that I extracted the
Markdown links from the section under the heading, took each address's host,
and where the host was github.com or gitlab.com appended the first path
component to build the "publisher" unit, then split those by the
publishDate field into months.

That landlord rule covers two names only, because this corpus has no other
tenancy. If yours does — sourceforge.net, codeberg.org, bitbucket.org,
readthedocs.io, npmjs.com, *.github.io — the rule is the same: any
address that rents out name space has to be split at the tenant level. Skip
that and your counter shows you a wider world than you have.

I stepped into three traps along the way. The first is language twins: every
post has an English counterpart and the same sources sit in those files too,
2,203 more links. Counting both would have pushed the total to 4,392 and
changed the diversity table not one bit — a twin brings no new publisher, it
counts an existing one twice. The second is the case sensitivity above. The
third: this post itself is not in the count; when you repeat it, its four
sources will be in the table as well.

Ask your own setup

Is the unit you count genuinely independent? Any counter that treats a landlord as a publisher and a mirror as a second source inflates diversity.
Are the two periods you compare the same system? If the production method changed, the gap in your hands may be a regime difference rather than a behavioural one.
Is the drop saturation or narrowing? The cleanest way to separate them is to put two periods of similar volume — and the same pipeline — side by side.
At what scale does your quality gate work? A gate that works per item cannot see anything that happens at portfolio level.

Conclusion

The real lesson I take from this is not the scale of gates — I learned that
six weeks ago and still did not see this narrowing coming. The real lesson is
the unit of the counter. What I took to be diversity got counted with a unit
of my own making, and as long as that unit was wrong the count did nothing
but reassure me. I counted a domain as a publisher; I counted four
doors into the same tree as four sources; I counted months from two different
production pipelines as two months of the same author. All three errors
pushed the counter up — that is, in my own favour.

The unit of measure is the definition of the thing you are trying to measure.
A wrong unit does not hand you a wrong answer; it hands you a reassuring
one, which is far harder to notice.

My next move is to turn this count into a check that runs once a month: the
distinct publisher count of the last 30 posts, and the largest publisher's
share. For a starting threshold I will take the September-October band —
warn if 30 posts rest on fewer than 40 distinct publishers, or if a single
publisher passes 20%. That may not be the right threshold, but starting
somewhere beats not measuring.

Official Sources

I Typed grep, I Got ugrep: Four Authorities Pick the Program

Mustafa ERBAY — Fri, 13 Nov 2026 11:30:56 +0000

Yesterday I threw away an article before publishing it. The real reason was duplication:
I had already written the same subject a month earlier (Unicode normalization, two methods
giving two different counts in the same ~/Downloads folder), and my first scan had missed
it because the list was cut short with head. But that run produced a second lesson, and
that is the one that actually stung.

The article's surprise was this: a filename was found by grep -F and not by grep -E. I
ran the same measurement with /usr/bin/grep — it found the file in all three modes. The
thing I was presenting as a surprise was not the filename's behaviour. It was my shell's.

I wrote one line in my working notes: "Run every measurement about text or file behaviour
with an absolute path (/usr/bin/grep, /usr/bin/find, /usr/bin/tar) and print its
version in the article." Correct lesson, incomplete one. Because the real question is
bigger: I type the name of a command — who decides which program runs? Today I
counted. The answer sits in four authorities: aliases, the shell's function table, the
order of PATH, and the first line of the script. I set up all four, each at a different time.

The experiment: same command, same flags, three files against one

I wanted to see the mechanism in the smallest possible rig first. An empty directory, a
.gitignore, and three files carrying the same string: one ordinary, one listed in
.gitignore, one under node_modules/. No repository needed — I did not run git init;
the presence of the ignore file is enough.

$ mkdir -p deney/node_modules && cd deney
$ printf 'gizli.txt\nnode_modules/\n' > .gitignore
$ echo 'ARANAN_DIZGE burada' > acik.txt
$ echo 'ARANAN_DIZGE burada' > gizli.txt
$ echo 'ARANAN_DIZGE burada' > node_modules/paket.txt

$ /usr/bin/grep --version
grep (BSD grep, GNU compatible) 2.6.0-FreeBSD

$ /usr/bin/grep -rl ARANAN_DIZGE .
./acik.txt
./node_modules/paket.txt
./gizli.txt

$ grep -rl ARANAN_DIZGE .
acik.txt

Same flags, same directory, same minute. Three files when I call it by absolute path, one
file when I type its name. Two thirds of the matches never appeared — and no warning was
printed. In a counting exercise, that is more than enough silence to write a "clean" report.

type tells you why:

$ type grep
grep is a shell function from /Users/mustafaerbay/.claude/shell-snapshots/snapshot-zsh-...sh

In this shell grep is not a binary, it is a function. Its body hands the command to a
different executable under the identity ugrep and inserts flags such as
--ignore-files --hidden --exclude-dir=.git along the way. --ignore-files reads
.gitignore and skips whatever is listed there. So the function is not misbehaving; it is
doing exactly the job it was designed for. My expectation was wrong: that is the behaviour
I want while moving around a repository, not while counting an inventory.

The most irritating part is this: the function has an escape hatch. When it sees certain
flags it steps aside and calls the real grep.

$ grep -rlZ ARANAN_DIZGE . | tr '\0' '\n'
./acik.txt
./node_modules/paket.txt
./gizli.txt

I added one letter and the result went back to three. Same name, same directory, two
different programs depending on a flag. There is no way to spot that by eye inside a
measurement script.

The order is documented, and it is not in my favour

This is not an accident, it is zsh's documented behaviour. When the shell resolves a command
word it goes: "If there exists a shell function by that name, the function is invoked", then
"If there exists a shell builtin by that name, the builtin is invoked", and only at the end
"Otherwise, the shell searches each element of $path". PATH is the last place it looks.
If a function exists, PATH is never opened at all.

There is one more layer above functions: aliases. An alias is expanded while the line is
being parsed, before resolution starts. A two-line experiment is enough:

$ deney() { echo FONKSIYON; }
$ alias deney='echo ALIAS'
$ deney
ALIAS
$ unalias deney; deney
FONKSIYON

My own terminal has five aliases and none of them shares a name with a command on PATH — so
this layer is empty on my machine. It is not empty for someone who writes
alias grep='grep --color=auto', and most "useful dotfiles" repositories suggest exactly that.

There are two ways out, both in the documentation — but they do not do the same thing.
command grep ... runs "an external command instead of a function or builtin"; that is, it
skips the function, not the ordering. For someone whose PATH has /opt/homebrew/bin/grep
in front, command grep still means GNU grep, not BSD grep. An absolute path cuts through
both. And whence -p grep does "a path search for name even if it is an alias, reserved
word, shell function or builtin". My reflex for years has been to type which; in this
shell that prints the function body — it is answering correctly, but I read it as "a path"
and move on.

The shell also has a memory. The first time zsh runs a command it notes its location
(HASH_CMDS: "Note the location of each command the first time it is executed") and does not
search PATH again on later calls. Add a directory to PATH mid-session and you keep getting
the old binary; rehash (that is, hash -r) empties the table. If you changed PATH while
measuring, this is the most common reason the winner disagrees with your arithmetic.

This shell has 131 functions defined. Three of them share a name with a command on PATH:
grep, find, pkill. Three. Hard to notice among a hundred and thirty-one, enough to
break a measurement.

The PATH inventory: 26 entries, six of them gone

Past the function layer comes PATH. Every number in this section comes from the shell my
agent runs; the PATH in my own terminal is three directories shorter, and I will get to the
difference at the end.

Twenty-six directories. Six of them do not exist on disk. Three are macOS cryptex
bootstrap paths, one is /pkg/env/global/bin — the /pkg directory it points at is not on
this machine at all. The remaining two are my own doing: ~/development/flutter/bin and
~/.pub-cache/bin. I deleted Flutter months ago; the lines still sit in the first two lines
of my .zshrc.

I also wondered where those directories come from. Twelve of them come not from my own files
but from path_helper. Apple's man page is clear about the job: first "default PATH and
MANPATH values are obtained from the files /etc/paths", then the contents of the files in
/etc/paths.d are appended. On my machine /etc/paths has six lines and /etc/paths.d has
four files with six lines in total.

I looked at those four files one by one, because I was about to wave them away as "leftovers
of tools I installed" and that would not have been true. All four are owned by root. Two
(10-cryptex, 100-rvictl) ship with macOS — rvictl is still sitting in
/Library/Apple/usr/bin. MacGPG2 is dated 2020 but it is not a leftover either: there are
twenty executables in that directory and /usr/local/bin/gpg still links into it. Only
10-pmk-global is genuinely orphaned, and it carries the same creation minute as the system
files; I do not know who put it there, and writing that I do not know beats guessing. A dead
directory on PATH does no measurable harm — it is just enough to make you think something is
installed.

One more detail: path_helper runs through /etc/zprofile on every login shell and puts its
own list in front. That is why a PATH you export in .zshenv ends up behind the system
directories. On macOS this is where people trip most often.

85 commands with the same name: the winner is whoever is earlier

Across the twenty directories that exist on this PATH there are 2,290 distinct command names.
85 of them exist in more than one directory; in 81 cases the copies really are separate
files (in the other four, both paths lead to the same file). So there are eighty-one "which
one?" questions on this machine, and the answer is given by directory order, not by me.

Command	In how many directories	Winner
`python3`	4	`/Library/Frameworks/Python.framework/Versions/3.13/bin` (3.13.3)
`gpg`	3	`/opt/homebrew/bin`
`psql`	2	`/opt/homebrew/opt/postgresql@16/bin` (16.11)
`node`	2	`~/.nvm/versions/node/v22.16.0/bin` (22.16.0)
`git`	2	`/opt/homebrew/bin` (2.48.1)

That last row stopped me. /usr/bin/git on this machine is 2.54.0 (Apple Git-157), while
Homebrew's git is 2.48.1. For years I have assumed "the system tool is old, the package
manager's is new" and put Homebrew at the front of my PATH. For this command the opposite
happened: my ordering drops me six minor versions back. What is old here is not the system,
it is the package I forgot to update. Three weeks ago I counted
the two packages my package manager refuses to read from its own list;
there the problem was a package not showing up, here the package shows up fine — what is
invisible is which copy runs.

Which pg_dump? My client is 16, my server is 14

The psql row is more serious. Line seven of my .zshrc puts postgresql@16/bin at the
front of PATH, which is why psql and pg_dump come out as 16.11. But the PostgreSQL server
running on this Mac is postgresql@14: in the process list it is
/opt/homebrew/opt/postgresql@14/bin/postgres, and that cluster's PG_VERSION file holds a
single line, 14. (There is also an idle postgresql@16 data directory on disk; that is
what really sets the trap.)

I connected the client and asked the server who it was:

$ /opt/homebrew/opt/postgresql@16/bin/psql -tAc 'select version()' postgres
PostgreSQL 14.18 (Homebrew) on aarch64-apple-darwin24.4.0, ...

It works. And that is no coincidence: PostgreSQL's own documentation says "pg_dump can also
dump from PostgreSQL servers older than its own version", supported back to 9.2. A newer
client can read an older server.

For the other direction the same page is far blunter: "pg_dump cannot dump from PostgreSQL
servers newer than its own major version; it will refuse to even try, rather than risk making
an invalid dump." Today I am on the good side, but I won that by ordering, not by skill. If
that line in .zshrc were ever deleted — or if I had never written it — pg_dump would
quietly fall back to the 14.18 in /opt/homebrew/bin; and on the day I moved the cluster
to 16, my backup command would not have written half a file, it would have politely refused.
The good news is the politeness of that refusal; the bad news is that the only place the
refusal shows up is the log of a scheduled job.

The third context: launchd's four-directory world

After measuring my terminal and my agent's shell, I had not counted the third context at all:
scheduled jobs. There are 27 plists under ~/Library/LaunchAgents — jobs that pull backups,
fetch exchange rates, open tunnels, feed a panel.

launchd states what it hands over, in its own report:

$ launchctl print gui/501/com.erbay.spark-bulut | /usr/bin/grep -A3 'default environment'
    default environment = {
        PATH => /usr/bin:/bin:/usr/sbin:/sbin
    }

Four directories. They are not arbitrary; they are a constant in Apple's Libc:
#define _PATH_STDPATH "/usr/bin:/bin:/usr/sbin:/sbin". I counted: 1,257 command names
resolve on that PATH — 55% of the 2,290 I counted in my shell. There is no node in it, no
npm, no psql, no gh, no brew.

And there is a python3 — but which one? That job's plist invokes its program as
/usr/bin/env python3. I asked with those same four directories:

$ env -i /usr/bin/env python3 --version
Python 3.9.6

In my terminal python3 is 3.13.3. In the scheduler it is 3.9.6. In Python's own developer
guide the status of 3.9 is "end-of-life", dated 2025-10-31 — an interpreter that has not
received a patch in more than a year runs my script every morning at 07:15. Today's run wrote
two lines to its log and both are ordinary; so this is not something broken, it is a place
where the version I installed myself never comes into play.
The day I counted launchd's retries
taught me something similar: launchd does what it is told, not what I think I told it.

Three plists set PATH themselves. One wrote four directories, one wrote six; both are
reasonable. The third froze an agent session's PATH exactly as it stood at that moment: 32
entries, 26 unique, nine entries (seven distinct directories) that no longer exist. One of
them is ~/.codex/tmp/arg0/codex-arg0adUvOF — a temporary directory, deleted weeks ago. That
plist keeps looking for it on every launch. Among the same files I also found a comment I had
written myself: "the directory holding the claude CLI must be on PATH in the launchd
environment." So I had already learned this lesson, noted it in a plist, and then forgotten it.

And a fourth: the job that walks back into the shell

I was going to write "none of these jobs ever sees my shell"; before writing it I read all 27
plists one by one, and I am glad I did. One of them — my daily worktree cleanup — invokes its
program as /bin/zsh -lc <script>. The -l means a login shell: /etc/zprofile,
path_helper and .zprofile all come into play. But -c means non-interactive, so .zshrc
is not read. I rebuilt its environment and asked:

$ env -i HOME=$HOME USER=$USER PATH=/usr/bin:/bin:/usr/sbin:/sbin \
    /bin/zsh -lc 'echo $PATH; whence -p node'
# 17 entries
/opt/homebrew/bin/node

Seventeen entries: more than launchd's four, fewer than my terminal's twenty-three. And node
there is Homebrew's 23.11.0 — not the nvm 22.16.0 that wins in my terminal. Because nvm is
set up by .zshrc, and .zshrc is never read in this job. That script only calls git, so it
does no harm today; but the question "what do I get when I type node on this machine" has
three answers here: 22.16.0, 23.11.0, and command not found.

The shebang layer: 53 commands look up their interpreter on PATH

PATH does not only resolve the name I type. When I run a script, that script's interpreter goes
through the same mechanism.

I looked at the first line of all 2,290 commands on my PATH: 1,705 are binaries (no shebang),
583 are scripts, two could not be read. 53 of those 583 scripts begin with #!/usr/bin/env;
that is, they look up their interpreter on PATH at run time. The four largest groups: 17 node,
11 perl, 9 bash, 8 python3. The list includes npm, yarn, wrangler, firebase,
netlify and claude.

This small detail has a nice consequence. Homebrew's npm does not use Homebrew's own node
23.11.0, it uses the nvm node 22.16.0 at the front of PATH — because the first line of
npm-cli.js is #!/usr/bin/env node. So the answer to "which node" depends not on the manager
that installed the package but on whichever PATH is in force; the scheduled job in the previous
section is the proof. The only way to measure the difference between being installed and being
used is to ask for the path, not the name.

The remaining 530 scripts name an absolute interpreter. Two of those point at a
/usr/local/bin/bash that does not exist (test-yaml and test-yaml5.34). Two commands,
quietly unusable for months. Nobody noticed, because nobody looked for them.

I broke my own measurement twice

The most instructive part of this article is the traps I fell into.

The first. To compare my terminal with my agent's shell, I ran zsh -l -c 'echo $PATH' from
inside the agent. The output had 30 entries, four of them duplicated, and 43 command names
resolved to different paths between the two contexts: node, npm, psql, the whole pg_*
family. A perfect headline. I even started sorting out which ones were genuinely separate files
— 35 really were. Then I looked at the rig: I had started zsh -l from the agent's shell, that
shell inherited its PATH, and .zprofile and .zshrc added their own entries on top. The
difference I had found was a product of my own rig.

The second, and the embarrassing one. Next came a clean environment:
env -i HOME=... TERM=... /bin/zsh -lic 'echo $PATH'. The output had 24 entries and one of them
was /usr/ucb. That directory does not exist on this machine, appears in none of my dotfiles,
and never enters my PATH in a Terminal window. Where did it come from? env -i removes PATH
entirely, so zsh falls back to its compile-time default:

$ env -i /bin/zsh -fc 'echo $PATH'
/bin:/usr/bin:/usr/ucb:/usr/local/bin

A real login shell does not find PATH empty: launchd hands it _PATH_STDPATH — the same four
directories I printed two sections earlier. So the environment I thought was clean was
committing the article's own error one floor further down. The correct rig:

$ env -i HOME=$HOME SHELL=/bin/zsh TERM=xterm-256color USER=$USER \
    PATH=/usr/bin:/bin:/usr/sbin:/sbin /bin/zsh -lic 'echo $PATH'

The real answer: 23 entries in my terminal (six of them missing from disk), 26 in the
agent's shell, and the number of commands resolving to different paths is zero. The
directories present for the agent but not in my terminal are ~/go/bin, ~/Library/pnpm and
~/.local/bin; of the 10 executables in those three directories, 9 reach me from nowhere else
in my terminal (govulncheck and goimports are two of them — which means that if I call
either in my own terminal, it is not found).

So the only real difference between the two shells is those three functions. Three lines, enough
to demolish an article's surprise; while all forty-three differences were an echo of the rig I
had built. Within twenty-four hours I made three floors of the same mistake: first I failed to
see that my shell had swapped my tool, then I contaminated the environment while comparing, then
I over-cleaned it while cleaning.
The day I counted my shell history the
lesson was close to this one — an assumption about a tool is not a measurement with it.

Questions for your own setup

The numbers from this machine tell you nothing; the questions do.

Before writing a counting measurement, run type <command>. If the output says "shell function" or "alias", your measurement is measuring your shell, not your disk. Print the version in the report too.
Do not confuse command with an absolute path. command grep skips the function, not the PATH ordering; the only thing that guarantees which grep you get is /usr/bin/grep.
If you are comparing, reset the environment — but do not over-reset it. env -i removes PATH as well, and the shell falls back to its own default; to imitate a real login shell, pass PATH with your system's default value.
If you changed PATH mid-session, rehash. Otherwise the shell keeps using the location it noted earlier and your measurement reports the wrong winner.
Read your PATH end to end, once. Which directories are missing, which line comes from which file, whose leftovers are sitting in /etc/paths.d — and are they really leftovers?
Look for binaries that share a name. Database clients, python3, node, git: if ordering picks the version, make sure you set that ordering deliberately.
Do not trust PATH in scheduled jobs. Write the program path absolutely; if you must set PATH, write out the three directories you need instead of copying your current shell. A frozen PATH becomes a list of non-existent directories six months later.
Wherever you see #!/usr/bin/env, read it as "the interpreter will be chosen at run time." Fine if that is the behaviour you want in production; not fine if you meant to pin a version.

The name of a command is an intention; the program that runs is a decision

What I type on the keyboard all day is not a command, it is a command name. Four authorities
decide which program runs: aliases, my shell's function table, the order of PATH, and the first
line of the script. All four came from my own hands, but none of them is the work of a single
sitting — one came from a tool's install script, one from the seventh line of my .zshrc, one
from a directory deleted months ago.

That is why "it works on my machine" now feels incomplete to me. Not on which machine but in
which context: in my terminal (23 directories), in my agent's shell (26 directories, three
functions), in the midnight job opened with zsh -lc (17 directories, a different node), or at
07:15 in launchd's four? Same machine, same disk, four different answers.

What yesterday's discarded article gave me turned out to be a habit rather than a lesson: ask for
the tool's identity before you measure with it. If I do not know who shows up when I say grep,
the number I counted is not a measurement, it is an impression.

Official Sources

The Dripping Server Did Not Trip Most of My Timeouts

Mustafa ERBAY — Fri, 13 Nov 2026 06:35:02 +0000

Back in April I wrote a piece on this blog about timeout budgets and retry storms. The arithmetic there went like this: give every layer a slice of time, keep the lower layer's budget smaller than the one above it, otherwise retries pile onto each other. The arithmetic is right. But that article quietly assumed that the moment you write a budget into a client, the budget is actually enforced.

This week I tested that assumption and found it does not hold in most places.

I wrote an HTTP server that returns an eight-byte body, but sends those bytes three seconds apart. It has no ambition to be the world's slowest web server; its only job is to stall. Then I told the HTTP client of five languages, plus curl, the same thing: five seconds. Ten of them got a number; one I left untouched as a control — eleven setups in all. Seven of them waited twenty-four seconds and returned 200 as if nothing had happened.

The cause is not a bug but a difference in definition. In most clients, what we call a "timeout" is not a deadline placed on the total duration of a request; it is an hourglass set between two I/O events. Every byte that arrives turns it over again.

Machine, server and method

The measurements were taken on 13 November 2026 on my own machine: macOS (Darwin 25.6.0), Apple M4 Pro, everything over 127.0.0.1. Tool versions: curl 8.7.1, Python 3.9.6 with requests 2.32.3, Node.js 22.16.0, Go 1.26.1, OpenJDK 24, Ruby 2.6.10. The Python side ran through the absolute path /usr/bin/python3; my PATH also carries 3.13, and the exception class name differs between them — that is why the table below says socket.timeout. There is no network variable here — no packet loss, no DNS, no TLS handshake. What is left is only how the client keeps time.

The server is a seventy-line Python script. Its heart is this:

headers = (
    "HTTP/1.1 200 OK\r\n"
    f"Content-Length: {BODY_BYTES}\r\n"
    "Content-Type: text/plain\r\n"
    "Connection: close\r\n\r\n"
).encode()
conn.sendall(headers)          # headers instantly
for i in range(BODY_BYTES):    # body drop by drop
    time.sleep(GAP)
    conn.sendall(b"x")

I set up two scenarios:

dripping: headers go out immediately, then eight bytes, each three seconds apart. Total delivery time 24 seconds.
silent: the connection is accepted, not a single byte is sent, and the socket closes at the twentieth second. This is the control group for the question "does the timeout never work at all, or only fail against a drip?"

I wrote as honest a "five seconds" as I could into each client: whichever field the documentation points to for timeouts in that language. The one exception is Node's bare fetch: I deliberately left it alone, as a control for what the default does. The goal is not to pick a winner; it is to see how many different things the same sentence means across five languages and curl.

One limitation up front: every measurement is plaintext HTTP/1.1 over loopback. A TLS handshake, HTTP/2 flow control or a reverse proxy in between would change the table; these numbers show the behaviour underneath those layers.

Two scenes, the same five seconds

Client and setting	silent server	dripping server
`curl --connect-timeout 5`	20.01 s · exit 52	24.03 s · 200 · 8 B
`curl --max-time 5`	5.01 s · exit 28	5.00 s · exit 28 · 1 B
`curl --speed-time 5 --speed-limit 1`	5.00 s · exit 28	5.01 s · exit 28 · 1 B
Python `requests`, `timeout=5`	5.01 s · ReadTimeout	24.04 s · 200 · 8 B
Python `urllib`, `timeout=5`	5.01 s · `socket.timeout`	24.04 s · 200 · 8 B
Node `fetch` (no options)	20.03 s · fetch failed	24.05 s · 200 · 8 B
Node `fetch` + `AbortSignal.timeout(5000)`	5.00 s · TimeoutError	5.01 s · TimeoutError
Go `http.Client{Timeout: 5s}`	5.00 s	5.00 s (while reading the body)
Go `Transport{ResponseHeaderTimeout: 5s}`	5.00 s	24.04 s · 200 · 8 B
Ruby `Net::HTTP`, `read_timeout = 5`	10.0 s · ReadTimeout	24.03 s · 200 · 8 B
Java `HttpClient` + `.timeout(5s)`	5.09 s · HttpTimeoutException	24.19 s · 200 · 8 B

The seven rows in bold are the ones that waited for all eight bytes and returned a successful result. Four rows cut the transfer at the fifth second. Against the silent server the table almost inverts: there, eight setups stand up at the fifth second, because there is no byte to reset the counter. Ruby wakes at the fifth second too, but pushes its total to ten for a reason we will come to shortly.

I repeated the timings three times and report the medians. The spread is surprisingly small: requests gave 24.04 · 24.04 · 24.03 across three runs; Java 24.19 · 24.17 · 24.19; Go 5.00 · 5.00 · 5.00. It should not be surprising, really — the server's delivery time is 8 × 3 seconds, so the number is entirely in my hands. What came out of the lab is not noise; it is a definition.

The most annoying row is the first. A curl command carrying --connect-timeout 5 will wait indefinitely once the connection is up; in the documentation's own words, that option "only limits the connection phase". That flag has been sitting in my scripts for years with the intent of "give up after five seconds". It does not give up.

Why it does not fire: who resets the counter

All seven rows share the same mechanism. The documentation does not hide it; we just do not read it.

Python requests splits the value in two: connect and read. For the read side the guide says the number is the seconds the client waits between bytes sent by the server, and adds in parentheses that "in 99.9% of cases, this is the time before the server sends the first byte". My server is precisely that 0.1%. As long as the three-second gaps stay under five seconds, the counter never fills. The same page also states separately that these values are not wall-clock.

urllib and the socket layer beneath it are even more bare: the settimeout docs define the value for "blocking socket operations", meaning every recv() gets its own five seconds. There is one interesting exception — sendall() has measured total duration since Python 3.5, and the docs say so explicitly. So in the very same library the sending side counts totals while the receiving side counts gaps. Two different clocks inside one language.

Node's fetch is built on undici, and both relevant undici options default to 300 seconds: headersTimeout and bodyTimeout. The body side documents itself: it "monitors the time between consecutive body chunks". Per chunk, in other words. A server dripping once every three seconds can keep that alive indefinitely. Node's own fetch documentation makes no mention of a default total limit; on the AbortSignal.timeout() side there is a single sentence: it returns a signal that will be aborted after the given milliseconds. In my measurements that signal was the one piece that actually worked.

Go is the only language here that names the header and body phases separately and gives each its own knob — requests documents a connect/read split too, but both of its halves count gaps. The comment on Transport.ResponseHeaderTimeout states that the time "does not include the time to read the response body". Because my server sends headers instantly, that setting never engages; the body trickles for 24 seconds and nobody objects. I read that comment not from the remote docs but from go doc net/http.Transport on my own machine — the same source as the Go version that ran the measurement.

Ruby's read_timeout is again per block read. But the 10 seconds it produced in the silent scenario gave away something else: Net::HTTP has a default max_retries of 1, so it retries an idempotent request once when it hits Net::ReadTimeout. The server log holds two separate GET /silent lines, five seconds apart. Running the same script with max_retries = 0 brought the time down to 5.01 seconds. So where I wrote five seconds, the library quietly spent ten — and that is the smallest, least charming version of the multiplication effect I described in my retry budget piece.

Java is the interesting case, because here the measurement fills a gap in the documentation. The HttpRequest.Builder.timeout() guide says an HttpTimeoutException is thrown if the response is not "received" within the given time, and adds a very clear sentence: not setting a timeout is the same as setting an infinite duration, i.e. blocking forever. Not a word about whether the body is covered. The measurement answers it: it is not. A request made with BodyHandlers.ofString() took 24.19 seconds and succeeded, despite its five-second timeout.

Third scene: the dripping byte is in the headers

Dripping the body is half the story. I played the same game with headers: a minimal 57-byte response header sent one byte per second, stretching header completion across 57 seconds.

Client and setting	while headers drip
`curl --max-time 5`	5.00 s · exit 28 · no status code
Go `Transport{ResponseHeaderTimeout: 5s}`	5.01 s · "timeout awaiting response headers"
Java `HttpClient` + `.timeout(5s)`	5.13 s · HttpTimeoutException
Python `requests`, `timeout=5`	57.35 s · 200 · 8 B
Node `fetch` (no options)	57.39 s · 200 · 8 B

Here Java's picture is complete: .timeout(5s) bounds the header phase as a total and does not bound the body at all. When the docs say "if the response is not received", they really do mean the headers; the body is not their concern. Go, meanwhile, named the two phases separately and gave each its own knob — the only language in the table that did. requests counts gaps here too: because the one-second gaps stayed under five, it never woke up for 57 seconds.

On the Node side this scene does not discriminate: undici's headersTimeout defaults to 300 seconds and my header drip lasted 57, so the measurement only shows that it stayed under 300. What the measurement cannot say, the documentation does: headersTimeout is the time the parser waits to receive the complete HTTP headers. Its sibling on the body side counts the gap between chunks; this one is a total for the header phase. Two settings side by side in one library, two different clocks — the same split we have been following all along.

The four that fired, and curl's small trap

Three of the four that cut the transfer make the same statement: they apply the number to the whole request. The fourth (--speed-time / --speed-limit) is a different mechanism altogether — not a deadline but a rate threshold; I take it apart in the next section.

curl --max-time says in its manual that it sets the maximum time you allow each transfer to take. Go's Client.Timeout comment is even clearer: the timeout includes connection time, any redirects, and reading the response body; the timer also keeps running after Get/Do return and will interrupt reading Response.Body. That is exactly what happened in the measurement: the error arrived while the body was being read. AbortSignal.timeout(5000) belongs to the same class — the signal cancels the request itself and does not care about the rhythm of the bytes.

There is a small but harmful detail here. Yes, curl --max-time 5 cuts the dripping transfer at the fifth second; but the %{http_code} output still reads 200, while %{size_download} is only 1 byte. If the exit code were not 28, my script would have counted that transfer as a success. Which means every shell line that decides success by looking at the status code will take a half-downloaded body for a complete one. The right question is not "what HTTP code?" but "what exit code, and did I get as many bytes as I expected?".

And there is an open-ended piece of arithmetic. One byte every three seconds means 1024 × 3 = 3072 seconds for a 1 KiB body, roughly 51 minutes. I did not measure that; I derived it from the rate in the table. But the size of the number tells you the point: as long as the gap counter never fills, the upper bound on total time depends on the server's patience, not on your setting.

The stall detector's threshold has to sit above the drip rate

curl also has a slow-transfer detector: --speed-limit and --speed-time. Per the manual, a transfer is aborted if it stays below speed-limit for a speed-time period. The defaults are defined in terms of each other: give only --speed-time and the limit is 1 byte per second; give only --speed-limit and the window is 30 seconds.

To show that this is a threshold, I sped the server up: 40 bytes, half a second apart, i.e. roughly 2 bytes per second.

--speed-time 5 --speed-limit 1  -> 20.14 s · 200 · 40 B · exit 0   (NO abort)
--speed-time 5 --speed-limit 5  ->  5.02 s · exit 28 ·  9 B        (abort)

The real rate in the first line is 40 ÷ 20.14 = 1.99 bytes per second. Since the limit was 1, the transfer counted as "fast enough" and flowed to the end. (curl's own %{speed_download} field rounds to an integer and prints 1B/s here; I computed the ratio from the raw numbers myself.) Raise the limit to 5 and the same server gets cut at the fifth second.

The practical takeaway: this detector is not a free safety net, it is a choice of threshold. Unless you write a number above the lowest rate you consider reasonable, the dripping side will sit just above your threshold and stall you as long as it likes.

Setting the total limit by hand: the patch that cuts at six seconds

Go and curl hand you a deadline out of the box. requests does not, so the deadline has to be hand-rolled: read the body as a stream and check the clock on every chunk.

with requests.get(url, timeout=5, stream=True) as r:
    for chunk in r.iter_content(chunk_size=1):
        chunks.append(chunk)
        if time.monotonic() - t0 > TOTAL:      # TOTAL = 5.0
            raise TimeoutError("total deadline exceeded")

All three runs gave 6.02 · 6.01 · 6.02 seconds. Not five, six. The reason is a pleasant detail: the loop only wakes when a chunk arrives, and the bytes arrive at the third, sixth and ninth second. So a deadline at the fifth second is only noticed at the sixth.

That overshoot looks harmless but it scales: the excess equals the gap between bytes. Six seconds on a three-second drip; two minutes on a sixty-second drip. A deadline held inside the read loop is not an upper bound but an approximate upper bound. If you want a hard limit, it has to live outside the loop: either shrink the socket's own timeout to the remaining budget before every recv, or set a watchdog that kills the process from the outside. A real deadline is about whose hand the clock is in.

The same question on the server side: my own nginx

This article puts the client on the bench, but the question is symmetric: how long will my server hold a slow counterparty?

In this blog's nginx configuration, deploy/nginx-mustafaerbay-final.conf carries proxy_read_timeout 60s; in two places — which is nginx's default anyway. The documentation warns in the same entry: the timeout is set only between two successive read operations, not for the transmission of the whole response. So as long as my backend sends one byte every 59 seconds, the nginx worker waits there. The same sentence and the same 60-second default apply to proxy_send_timeout, send_timeout and client_body_timeout.

The exception sits one line away and it is valuable: client_header_timeout terminates the request with a 408 if the client does not transmit the entire request header within the given time. So nginx holds one example of each kind — one counts gaps, one counts the total. Saying "my timeouts are configured" without knowing which directive is which means putting the configured and the unconfigured into the same bag.

One more thing: client_header_timeout alone is not protection against a slow client. A thousand connections that each finish their header in 59 seconds stay inside the rule and can still fill the workers; the answer there is not a timeout but a cap on concurrent connections and request rate. My own file has one of the two: the /api/ location carries limit_req zone=mb_api burst=20 nodelay (10 requests per second) — but only there, and there is no limit_conn at all. So the root location that serves the pages is unprotected against the number of connections held open.

In the first version of this article I was going to close this section with "and here is how I fixed it". I did not fix it: 60 seconds is a reasonable ceiling for this blog, and the real gap is a layer that adds a total limit next to the gap-counting directive. I will not change it before measuring it.

Counting in my own code: 75 call sites, 52 with no limit

Measuring the outside world and skipping my own repository would have been the easy way out. I wrote a small script that scans the fetch( calls in this blog's code: it extracts each call's balanced parentheses and checks whether signal appears inside.

The result: 75 call sites, 23 with a signal, 52 without. The distribution is instructive too:

scripts/ (the CI side, the code that talks to providers): 49 calls, 21 with a limit.
src/ (the app and the parts that run in the browser): 26 calls, only 2 with a limit.

The two groups do not carry the same risk. The ones under scripts/ go out to external providers and an agent waits on them during a CI run — one that hangs burns minutes. Most of the ones under src/ run in the browser against my own API; there, no deadline shows up as a spinner that never stops and the visitor reloads the page. The headline 52 sums both; the real bleeding is the 28 calls in the first group.

I should state the script's own limitation: it only looks for signal in the call text. If the codebase had a shared wrapper that added a deadline, this count would be wrong. There is none — every AbortSignal.timeout call in the repo is written directly into the fetch options, and there is no shared fetchWithTimeout helper. So 52 really does mean "this call has no deadline of its own".

Among the ones without a signal is this blog's own social-sharing step: the caption generation in scripts/social-post.ts calls fetch with no deadline for Gemini (line 467) and for the OpenAI-compatible providers (line 518). Elsewhere in the same file there are AbortSignal.timeout(5000) and AbortSignal.timeout(15000) calls — so the habit exists, the coverage does not. On top of that, .github/workflows/social-post.yml has no timeout-minutes line at all; for comparison, deploy-only.yml carries three separate limits of 50, 40 and 5 minutes. A dripping provider could keep that run busy, unnoticed, up to GitHub's own job ceiling.

This stays as open work, and I am writing it here so it does not get lost: the two calls on the caption path will get a deadline, and the workflow an outer limit. I did the counting for this article; I did not ship the fix with it.

Counting gaps is not always a defect

So far I have treated gap-counting settings as a defect; to be fair, sometimes they are exactly what you want. Put a hard total deadline on a client that listens to a server-sent event stream, or collects a language model's answer token by token, and you will cut off a long but healthy stream with your own hand; the right question there is not "how many seconds in total" but "how much silence between two chunks is acceptable". So bodyTimeout counting between chunks is not a flaw, it is the behaviour that job was designed for. The rule: if you expect a response with a known end (JSON, a file, an API call), set a total deadline; if you are listening to an open-ended stream, tighten the gap limit and bound the total from somewhere else.

Since I did not measure it, let me be explicit about the alarm side too: the signal that catches these hangs is not an error counter but a duration distribution. A request that takes 24 seconds and returns 200 shows up nowhere on an error graph; to see it you need a threshold on the request-duration histogram and on the bytes-per-second rate. My own pipeline has no such threshold; this measurement showed me that too.

A checklist: six questions before you accept a client

Does this setting count a gap or a total — and which phase? If the docs say "between", "successive" or "per chunk", it counts gaps. The phase is a separate question: Java's .timeout() treats the header phase as a total and does not count the body at all. If there is no single total deadline, the client has no upper bound.
Where does the total limit come from? Client.Timeout in Go, AbortSignal.timeout in Node, --max-time in curl, a hand-built deadline in Python. If the library does not give you one, add it.
Is the stall threshold above the lowest rate you expect? You will not catch a dripper with --speed-limit 1; its rate may well be 2 bytes per second.
What do retries multiply the budget by? Ruby's default max_retries = 1 turned five seconds into ten. If you use curl --retry, the --max-time counter restarts on every attempt; --retry-max-time is what caps the total.
What are you measuring success by? The status code can read 200 while the body is incomplete. Check the exit code and the expected byte count too.
Which directive counts gaps on the server side? In nginx, proxy_read_timeout counts gaps and client_header_timeout counts the total. Trusting both equally is a mistake.

A timeout says who is allowed to be slow, and for how long

What I took away from the measurement is not a setting value but a shape of question. When we ask "how many seconds is the timeout?", we are conflating two separate things: how long the other side may stay completely quiet, and how long the whole job may take. Most libraries answer the first and leave the second to us — and they say so honestly in their documentation.

The gap between the two is invisible under normal conditions. The network either works or breaks; when it breaks, the counter fills, the error arrives, everyone feels safe. The gap only opens when the other side is slow but alive: a swollen queue, or a deliberate dripper. A seventy-line script sending one byte every three seconds held seven setups across five languages and curl for twenty-four seconds without sounding a single alarm. The same script could hold a CI run for hours — or take the slots in a connection pool one by one and close them to healthy requests.

The question to ask of your own setup is this: in this request, as long as the other side keeps sending me one byte per second, who decides when the job is over? If the answer is "nobody", what is missing is not a number of seconds — it is the clock itself.

Official Sources

pax Recovered the File tar Swallowed from the Same Archive

Mustafa ERBAY — Wed, 11 Nov 2026 15:06:32 +0000

In the post where the same commit got packed twice,
published on 3 November, something turned up as an aside: macOS tar writes
members into the archive that I never put there, and then doesn't list them
back to me. The direction there was outbound — the Mac packs, the other side
finds extra files.

This post is the inbound face of the same mechanism. The question: if an
archive arriving from the other side holds a real file whose name begins
with ._, what does macOS do?

I tried six different modes. In all six the file never reached disk; in four
tar errored and exited 1, in two it said nothing at all and exited 0. And
pax, from the same machine's /bin, pulled the same file out of the same
archive with its contents intact.

"Why would such a file even exist"

That's the first objection, and it's a fair one. But the ._ prefix is special
only to macOS; on Linux and BSD it's an entirely ordinary name. An editor
backup, a tool's output, a deliberately hidden helper file, or a twin that once
came from a Mac and settled in on the other side — any of them can start with
._, and to the other side they are simply data.

So the lab lives on Linux. Two files: one ordinary, one a real text file whose
name starts with ._. (The Turkish line inside it, BU BIR DUZ DOSYA, means
"this is a plain file" — it matters later, because it is plainly not metadata.)

docker run --rm -v "$PWD":/ark alpine:3.21 sh -c '
  mkdir -p /ark/tersyon/ic && cd /ark/tersyon
  echo gercek > ic/veri.txt
  printf "BU BIR DUZ DOSYA\n" > ic/._veri.txt
  tar czf /ark/tersyon.tgz ic'

Both are in the archive, and on Linux both land on disk:

$ docker run --rm -v "$PWD":/ark:ro alpine:3.21 tar tzf /ark/tersyon.tgz
ic/
ic/._veri.txt
ic/veri.txt

The environment: macOS 26.6.2, bsdtar 3.5.3 - libarchive 3.7.4, and on the
other end busybox tar 1.37.0 inside alpine:3.21 (Docker 27.4.0). I write the
versions down because this whole post hangs on the question "which tool".

The listing is already short

Before extracting anything I wanted to list the archive. Two tools, two
answers:

$ tar tzf tersyon.tgz        $ pax -zf tersyon.tgz
ic/                          ic
ic/veri.txt                  ic/._veri.txt
    → 2 members              ic/veri.txt
                                 → 3 members

This archive wasn't made on a Mac; it was packed on Linux, with nothing called
AppleDouble anywhere in it. tar drops ._veri.txt from the listing anyway.
The loss doesn't begin at extraction — it begins while you're still looking at
the table of contents: the first command you ask about the archive's contents
gives you an incomplete answer.

Six modes, two of them silent

I opened the archive on macOS:

$ tar xzf tersyon.tgz -C chk
ic/veri.txt: Failed to restore metadata: File exists
tar: Error exit delayed from previous errors.
$ echo $?
1
$ find chk -type f
chk/ic/veri.txt

veri.txt is there, ._veri.txt is not. tar didn't see it as a file; it
took it for its neighbour's metadata and tried to apply it. I tried every flag
I could think of:

mode	exit code	files written	message
default	1	1	`Failed to restore metadata`
`--mac-metadata`	1	1	same message
`--no-xattrs`	1	1	same message
`-p`	1	1	same message
`--no-mac-metadata`	0	1	none
`--no-same-permissions`	0	1	none

Six modes, one outcome: in none of them does the file reach disk. I could not
find a flag that persuades tar to write that member out as a plain file.

The last two rows are the ones that bother me. With those flags the command
finishes successfully, you get not a single warning, and the file still isn't
there. The modes that error are being honest; the modes that report success
make the loss invisible. In a script running under set -e, the first four
rows stop you and the last two don't.

A small addendum: the tar(1) man page on this same machine says
--no-mac-metadata is "the default behavior if tar is run as non-root in x
mode". I ran the measurement as uid 501, and the default run tried to restore
metadata and failed — it did not behave like the fifth row. That sentence
doesn't hold for the version I have.

The decision is made on the name, not the contents

What's inside ._veri.txt? Plain text. The magic number of the AppleDouble
format — 0x00051607, defined by RFC 1740 — is nowhere in it. tar took it
for metadata anyway. It decides not by looking at the contents but at the
first two characters of the name.

The prefix itself is one line in Apple's copyfile source:

#define ATTR_FILE_PREFIX   "._"

I isolated the name-based rule with a second experiment: this time I put a lone
._abc.txt into the archive, with no sibling (its contents a 13-byte line).

$ tar xzf yalniz.tgz
ic/._abc.txt: Truncated tar archive: Unknown error: -1
tar: Error exit delayed from previous errors.
$ ls -l ic/._abc.txt
-rw-r--r--  1 mustafaerbay  wheel  13 Nov 11 17:19 ic/._abc.txt

The file is on disk, with its name and its size. But look inside:

$ xxd ic/._abc.txt
00000000: 0000 0000 0000 0000 0000 0000 00         .............
$ tr -d '\000' < ic/._abc.txt | wc -c
       0

All thirteen bytes are zero. tar tried to parse the file as AppleDouble,
decided the contents were a truncated archive, gave up halfway, and left behind
a shell with the right size and nothing in it. That's worse than the first
experiment: there the loss was at least visible in a file count, here both the
file count and the size look correct.

Two experiments, two outcomes, one rule: if the name starts with ._, nobody
looks inside.

Three tools in the same box: two lose it, one recovers it

"Open it on Linux" is an easy thing to say, but the framing is wrong. The
problem isn't the operating system, it's the tool's code — and I can show that
inside macOS alone. I had three built-in tools extract the same archive:

tool	written to disk	`._veri.txt`
`tar xzf`	1 file	missing
`cpio -id`	1 file	missing
`pax -rzf`	2 files	with its contents

Here is what pax prints:

$ cd paxcik && pax -rzf ../tersyon.tgz
pax: Unable to set metadata on ic/veri.txt <File exists>
$ find . -type f
./ic/._veri.txt
./ic/veri.txt
$ cat ic/._veri.txt
BU BIR DUZ DOSYA

Same machine, same archive, a message from the same "File exists" family — but
the file is on disk with its contents intact. pax tries to apply the metadata
and, when it can't, writes the member out as a plain file. tar and cpio
give up. All three ship in macOS's /bin or /usr/bin; no Docker, no brew.

The man page contradicts itself

While hunting for the flags in this post I tripped over the local
documentation twice.

First: the same tar(1) page introduces the same pair of flags with two
different mode lists.

$ man tar | col -b | grep -A1 -- '^     --\(no-\)\?mac-metadata$'
     --mac-metadata
             (c, r, u and x mode only) Mac OS X specific.  Archive or extract
--
     --no-mac-metadata
             (x mode only) Mac OS X specific.  Do not archive or extract ACLs

One says "create, append, update and extract", the other says extract only —
even though they're defined as each other's reverse. I tried which one is
right: tar --no-mac-metadata -czf works fine in create mode and produces a
twin-free archive. The line upstream libarchive writes for the same flag is
already "(c, r, u and x mode only)". The second entry's mode list is incomplete
in the version I have.

Second, and more interesting: the COPYFILE_DISABLE variable. The page's
ENVIRONMENT section has five entries (TAR_READER_OPTIONS,
TAR_WRITER_OPTIONS, LANG, TAPE, TZ) and this isn't one of them; it
isn't in the 591-line copyfile(3) page either, which has no ENVIRONMENT
section at all. The name is defined in copyfile.h:

#define COPYFILE_DISABLE_VAR    "COPYFILE_DISABLE"

But the only code in copyfile.c that reads that name is switched off:

#if 0
    /* Temporarily disabled */
    if (getenv(COPYFILE_DISABLE_VAR) != NULL)

So the library isn't the thing listening to the variable. Instead of guessing
where it is, I asked:

$ strings /usr/bin/tar | grep -i 'copyfile_disable\|disable-copyfile'
COPYFILE_DISABLE
disable-copyfile

Both strings live inside the tar binary. It's tar itself that reads the
variable; the definition in the source only supplies the name, and the check
moved out of the library. One more detail: the check looks at presence, not
value — setting COPYFILE_DISABLE=0 also suppressed the twins, taking the
archive from twelve members down to six. A reader who writes zero to turn it
off gets exactly the opposite.

A "clean" archive still carries your attributes

A side measurement, because the common advice stops short here. An archive
built with COPYFILE_DISABLE=1 or --no-mac-metadata has no ._ members —
but that doesn't mean the attributes are gone. I scanned all three archives:

$ for a in varsayilan kapali nomac; do echo -n "$a: "; gzcat $a.tgz | strings | grep -c 'LIBARCHIVE.xattr'; done
varsayilan: 9
kapali: 9
nomac: 9

Nine headers in each, belonging to three distinct attribute names (six
com.apple.provenance, two of my own com.ornek.not, one
com.apple.quarantine). So the archive I called clean carries the quarantine
tag of a downloaded file to the other side; only the AppleDouble members are
gone.

Do those headers do anything? I extracted two archives with two tools and
compared:

archive	extracted by	did `com.ornek.not` come back
default (AppleDouble present)	`tar`	yes
default (AppleDouble present)	`pax`	yes
`--no-mac-metadata` (pax header only)	`tar`	no
`--no-mac-metadata` (pax header only)	`pax`	no

On the macOS side what restores attributes is the AppleDouble member; the
archive format's extended headers (LIBARCHIVE.xattr.* — these belong to the
pax format, not to the /bin/pax tool) travel but are never applied.

Linux only reads those headers if you ask explicitly. A plain tar -xzf says
"unknown" to each of them; with --xattrs --xattrs-include='*' it tries, and
runs into a different wall:

tar: setxattrat: Cannot set 'com.apple.quarantine' extended attribute for
file 'kaynak/indirilen.txt': Operation not supported

My first reading was "this filesystem doesn't support extended attributes".
Trying three names on the same file showed otherwise:

user.test              → OK
com.apple.quarantine   → Operation not supported
security.foo           → Operation not permitted

The filesystem takes attributes fine; what refuses is Linux's namespace rule.
A user attribute's name has to start with user., so com.apple.* can't be
written on any Linux filesystem. The names and values travel to the other side,
but they aren't reborn there as Mac attributes.

Cleaning up on the receiving side: `dot_clean` and its three modes

If you've ended up with a tree full of ._ files, macOS has a tool for it and
its man page describes it in one line: "Merge ._ files with corresponding
native files."* I ran all three modes over the same tree:

command	twin files	did `com.ornek.not` land on the file
`dot_clean <dir>` (default)	deleted	no
`dot_clean --keep=dotbar <dir>`	deleted	yes
`dot_clean -m <dir>`	deleted	no

No surprise for -m: the man page says "Always delete dot underbar files" —
a delete mode. The one to watch is the default. Its verbose output claims it
did the job:

$ dot_clean -v n2
Merging directory altdizin
merging ._ic.txt & ic.txt
Using the most recent stored info
Deleting: ._ic.txt

Yet afterwards xattr -l shows no com.ornek.not on the file. By the
documentation it should be there: DESCRIPTION says the most recent copy wins
for an attribute that is also present on the native file, and the
--keep=mostrecent entry says that otherwise the information stored in the
AppleDouble file is used. This attribute wasn't on the native file, so by the
rule it should have landed. With --keep=dotbar the output changes
("Replacing ic.txt's EAs with ._ic.txt's EAs") and the attribute arrives. The
default mode doesn't do what its own documentation describes; cleaning up and
restoring are not the same mode of the same command.

They pass extension-based filters

The sneaky part about the twins isn't their count, it's that their names look
legitimate. A ._report.mdx matches the *.mdx pattern. I tested this on my
own publishing pipeline; the script that moves the blog's MDX files into the
database finds them like this:

const files = execSync("find src/content/blog -name '*.mdx'", ...)

The pattern catches the twins — not a guess, a measurement:

$ find . -name '*.mdx' | sort
./technology/._gercek.mdx
./technology/gercek.mdx

In a pipeline that moves content through an archive, these files get taken for
posts. Mine takes content from git, so the scenario is theoretical for now —
and the measurement came back clean:

$ find ~/IdeaProjects -name '._*' -type f | wc -l
0

Zero. It scanned in two and a half seconds and found none, because I don't
extract Mac-made archives into my own tree. This mechanism doesn't accumulate
locally; the damage happens exactly at the boundary, where one machine packs
and another unpacks. That's the good half of the bad news: the number of places
to check is small and well defined.

Checklist — for the receiving side

Don't let tar do the counting. The gap between two listings shows the loss in one line: if pax -zf a.tgz | wc -l and tar tzf a.tgz | wc -l disagree, the archive holds ._ members. In a script, a gate comparing those two numbers catches the pipeline that loses files quietly under --no-mac-metadata.
The exit code isn't enough. --no-mac-metadata and --no-same-permissions return 0 and don't write the file; trusting set -e alone hides the loss.
Use pax to recover. If you have an extraction that genuinely needs members named ._something on disk, use pax -rzf; tar has no flag that does it.
If you clean up, choose the mode. Just getting rid of the twins: dot_clean <dir>. Also wanting the attributes inside them back: dot_clean --keep=dotbar <dir>.
Remember the "clean" archive carries the quarantine tag. Even an archive with no ._ members takes attribute names and values along as pax headers; to really strip them you have to clean the source before packing.
Watch out for extension-based filters. ._report.mdx matches *.mdx too. Add -not -name '._*' to the find/glob patterns on the receiving side.

Wrapping up

What this measurement left me with isn't the lost file itself; it's that a tool
decided what a file is by looking at its name. Had it looked at the
contents, the plain-text line would have been obvious in the first bytes —
AppleDouble's magic number doesn't match. A name-based rule is fast and simple;
in exchange it's blind to anything that imitates the name.

And there's this: I wanted to write the sentence "macOS behaves this way", and
I couldn't. Of three tools in the same box, two lose the file and one recovers
it. What decides is not the operating system but that binary. Before I write my
next "platform X does it this way", I'll try asking the second tool on the same
platform.

Official Sources

One of the Three Doors I Opened for Security Reports Leads Nowhere

Mustafa ERBAY — Wed, 11 Nov 2026 11:21:38 +0000

The projects page on this site lists fourteen projects, and seven of them carry a clickable address. So I publish seven open doors: one is this blog, six are applications a stranger can walk into.

Say somebody landed on one of those addresses and found something — a form accepting more than it should. That person wants to tell me. Where do they write?

I asked my own infrastructure that question today. Instead of guessing the answer, I went and checked: the same request sent to each of the seven addresses, the files that came back read line by line, and the contact addresses inside them tested at the DNS level for whether they can actually receive mail.

Three addresses had a door. One of those three has no delivery path behind it. And the standard I had followed says, in its own words, that this case may be worse than having nothing at all.

The bad part is that the audit did not stop there. I found the same pattern — the sign is up, the thing it points at does not work — in four more layers.

Seven addresses, three doors

There is an agreed-upon place on the internet for vulnerability reports: /.well-known/security.txt at the root of a domain. RFC 9116 defines it. I requested that path from all seven addresses, and here is what came back around midday on 11 November 2026:

Address I publish	`/.well-known/security.txt`
burncpu.com	200
spamkalkani.com	200
hrmarge.com	200
hesapciyiz.com	404
islistesi.com	404
gercekveri.com	404
mustafaerbay.com.tr	404

Three out of seven. The part of this table that bothers me is not the number of 404s but the bottom row. mustafaerbay.com.tr is the site that lists the other six. If somebody gets curious about any of these projects and tries to reach the person behind them, this is the first place they land, and there is no reporting door here.

The gap between the three and the four is not ignorance either. I got it right three times. I know how it is done, I have done it on three separate domains, and on four I have not. When those two sentences sit next to each other, the question stops being "how do you do this" and becomes "why did it not spread."

What is behind the door

I read all three files that came back. At first glance they were better than I expected:

# burncpu.com
Contact: mailto:mustafa@mustafaerbay.com.tr
Expires: 2027-07-14T00:00:00Z
Canonical: https://burncpu.com/.well-known/security.txt
Policy: https://github.com/merbay-erp/burncpu/security/policy
Preferred-Languages: tr, en

RFC 9116 makes two fields mandatory. For Contact the text is unambiguous: this field must always be present in a security.txt file. For Expires the requirement is just as firm — it must always be present and must not appear more than once. The standard also recommends that the value sit less than a year in the future so the file does not go stale.

All three files have both mandatory fields in place. The expiry dates are 2027-07-14 for burncpu, 2027-05-05 for spamkalkani and 2027-07-11 for hrmarge — roughly six to eight months out from today, so all valid and all inside the one-year window the standard recommends. All three are served over HTTPS as text/plain, both of which the RFC requires.

I could have stopped here and written "three projects have a reporting channel." It would have been wrong, because what I had checked was only the two mandatory fields. I had not looked at the fields the standard merely recommends. When I did, all three turned out to have no Encryption field — although the RFC recommends encryption when Contact is an email address. None of the three files is signed either; the standard separately recommends an OpenPGP signature.

A valid file, an undeliverable address

The three Contact values belong to three different domains. For an email address to receive anything, the domain needs an MX record — or, failing that, its address record has to speak SMTP on port 25. That fallback is called implicit MX and it comes from RFC 5321.

I checked all three in turn:

`Contact` address	MX record	Delivery path
mustafa@mustafaerbay.com.tr	`mail.sysroot.app`	yes
destek@spamkalkani.com	`mail.sysroot.app`	yes
destek@hrmarge.com	none	not found

hrmarge.com publishes no MX record at all. I asked three separate resolvers (1.1.1.1, 8.8.8.8, 9.9.9.9) and all three came back empty; the domain is not NXDOMAIN, it resolves cleanly on Cloudflare nameservers. So I fell through to implicit MX: the A records point at two Cloudflare addresses (104.21.95.108 and 172.67.144.162), and port 25 is closed on both from here.

That measurement needs a control, because plenty of ISPs block outbound port 25 — if mine did, every address would look "closed" and the test would say nothing. So I connected from the same machine to my own mail server, 137.74.47.57 behind mail.sysroot.app: port 25 open. The closure is not on my side.

The domain does carry an SPF record, v=spf1 a:mail.sysroot.app -all. That misled me for a moment, because my mail server's name appears in it. But SPF governs the sending side: it says who may send mail on behalf of that domain. It has nothing to do with where incoming mail is delivered.

Let me state the limit of the measurement too. I did not send a test message to that address. What this check covers is the delivery path at the DNS level; I derived the existence of the mailbox from that. But a stranger's report would travel the same path, and if the path is closed, what is at the end of it does not matter.

The verdict the RFC gives this case

Before interpreting the finding I went back to the standard, because RFC 9116's security considerations have a section on exactly this: incorrect or stale information. The text says that if the information in the file is not kept up to date, security reports either never arrive or go to the wrong contacts. Then comes this: not having a security.txt file may be preferable to having stale information in it.

It follows that with four instructions to organizations: the information in this file and the resources it references — web pages, email addresses, telephone numbers — should be kept current, be accessible, be controlled by the organization, and be kept secure.

After reading that I re-read my own table and the number changed. I have three addresses that publish a security.txt, and two that can receive a report. The third is a picture of a door: standards-compliant, unexpired, correct in form, empty behind. For a researcher that is worse than a 404. A 404 says "there is no channel here, look for another way." destek@hrmarge.com says write the message, send it, consider the job done. And I, having received nothing, assume there is nothing wrong.

The worst box in that diagram is not the one at the bottom right. The worst part is that the path leading to it returns a 200.

What is at the end of the Policy line

I was about to stop here, pleased with the lesson, when I noticed I had not asked the same question of the file's other lines. Policy points at an address too, and I had only checked whether it opened. All three opened.

Reading what opened broke the table again. The Policy line on spamkalkani.com goes to /privacy.html and from there to the /privacy page. That page is titled "Gizlilik Politikası — Spam Kalkanı" — a privacy policy — and contains no section on vulnerability reporting at all; searches for "vulnerability" and "disclosure" come back empty. So a researcher who follows the Policy line finds a privacy notice, not a reporting policy. HTTP 200, irrelevant content.

The Policy line on burncpu.com goes to the security policy on GitHub, and that file is genuinely good: a scope list, response-time targets, a safe-harbour clause, a clear email address. It also contains this instruction — if possible, encrypt the report with the maintainer's key, available at keys.openpgp.org.

I went looking for that key. keys.openpgp.org returns 404 for the address in question, and for all three of my address variants. To make sure I was not testing it wrong, I asked for a key I knew existed: security@torproject.org returns 200 from the same endpoint. The query method works. The key is not there.

That makes three instances of the pattern. File present, address dead. File present, policy irrelevant. Policy present, key missing.

The name on the address is a contract too

The Contact values in my three files begin with mustafa@, destek@ and destek@. None of them is security@.

When RFC 9116 defines the Contact field it says security email addresses should follow the conventions in section 4 of RFC 2142. That section is a short table: ABUSE for customer relations, NOC for network infrastructure, and SECURITY for network security — security bulletins or queries. A convention that has been sitting there since 1997.

This is a recommendation, not a requirement. But here is why it earns its keep: a security@ box is bound to a role rather than a person. All three of my addresses go either to me or to a general support box. When the first goes on holiday, or the second becomes a queue full of password-reset requests, a security report gets lost in the same pile. A role address does not fix that by itself, but it separates it — and you can write a filter against a queue that has been separated.

The same pattern on the repository side

If a stranger cannot get through the site, they look at where the code lives. My merbay-erp account has eight public repositories. I asked each one whether it has a SECURITY.md — in the root, under .github/ and under docs/, since GitHub reads all three. Three have one: burncpu, CORTEX-Private-Intelligence-Framework and zee. Five do not.

This morning I would have written "three out of eight" and moved on. But since the whole point of this piece is not settling for the presence of a file, I opened all three and got caught by my own thesis.

burncpu/SECURITY.md gives an address: mustafa@mustafaerbay.com.tr. An address with a delivery path. That file does its job.

CORTEX-Private-Intelligence-Framework/SECURITY.md directs the researcher to GitHub's "Report a vulnerability" button, and if that is unavailable, to contact the maintainer through their GitHub profile. Both routes are shut: that button depends on private vulnerability reporting, which is disabled on all eight of my repositories, and my GitHub profile publishes no email address.

zee/SECURITY.md opens with "Depo özeldir" — the repository is private — except the repository is public; the API reports visibility: public. It offers the security advisory form as the reporting route, which somebody without repository access cannot use while private vulnerability reporting is off. And the file's own sentence closes the matter: no new contact address is allocated by this document.

So on the repository side there are three files, and one of them gives a stranger a working route.

As for the front door

On this site itself there is no SECURITY.md in the repository, no public/.well-known/ directory, and no contact page — neither iletisim nor contact under src/pages/.

There is a gap in the structured data too. The file I audited in the piece where I wrote my identity into four registries, src/lib/profile.ts, is the source of everything this site says about me. Line twenty-one reads:

email: undefined as string | undefined,

The field is declared and has no value. I pulled the schema.org person record the homepage emits and checked it live: there really is no email key. I publish no machine-readable contact information at all.

On the human side there are addresses, and rather too many. The privacy page carries hello@mustafaerbay.com.tr as a clickable mailto: link. The subscription result pages carry mustafa@mustafaerbay.com. burncpu.com's security.txt carries mustafa@mustafaerbay.com.tr. Three variants across two domains; both domains have MX records pointing at the same server of mine, so all three have an open delivery path.

There is also an easter egg I print to the console, and in Turkish it reads: "If you are poking around here, write to me, let's have a coffee." I wrote a line that says exactly the right thing to somebody poking around, back in May. The trouble is that none of these addresses sits where a researcher would look: nobody hunts for a reporting channel inside a privacy notice or in the browser console. They look at /.well-known/security.txt.

Six signs, three of them working

At the end of the audit I counted. I publish six documents that tell a stranger where to write: three security.txt files and three SECURITY.md files. The test is simple — can a stranger who reads that document, with no access to the repository, reach me?

Document	Gives a stranger a working route
burncpu.com `security.txt`	yes
spamkalkani.com `security.txt`	yes
hrmarge.com `security.txt`	no — the address has no delivery path
burncpu `SECURITY.md`	yes
CORTEX `SECURITY.md`	no — a disabled button and an empty profile
zee `SECURITY.md`	no — allocates no address

Three out of six. Exactly half.

What bothers me about that number is that all six would pass a formal check. They all exist, they are all in the right place, they all return 200. If I wrote a scanner, it would mark all six green. The wall a stranger hits starts one layer past the existence of the file — and until today I had never looked at that layer.

Why it did not spread

There is no position here from which I can defend myself, but there is one from which I can understand it. I wrote a security.txt for three projects because the subject was on my mind while setting them up, and not for four because it was not. There was nothing in between: no checklist, no template I run when opening a new project, no audit asking whether this domain's reporting channel works. If you apply a practice project by project, it is not a habit; it is a coincidence.

The irritating part is learning afterwards that the propagation mechanism was within reach the whole time. If an account has a public repository named .github, GitHub uses the SECURITY.md inside it as the default for every repository that lacks its own — whatever the target repository's own visibility. The file can sit in the root, in the .github folder or in the docs folder; a repository's own file wins. So I do not need five files for five repositories, I need one. That repository does not exist yet either: merbay-erp/.github returns 404.

That convenience does not cover this blog, though, because the blog's repository lives under a different account — itwise-bilisim/mustafaerbay. The default file works per account, so that is a separate repository and a separate commit. On the site side there is no shortcut at all: every domain has to serve its own security.txt.

It is instructive to look at how this gets institutionalised in the public sector. The binding operational directive CISA issued in 2020 requires federal agencies to develop and publish a vulnerability disclosure policy — and in a specific place: at the /vulnerability-disclosure-policy path of the agency's primary .gov website, in plain text or HTML. It mentions security.txt too, noting that its use is not required under the directive but that it can help some people work out who to share findings with. Even there the mechanism is not "everyone remember": fixed path, fixed format, deadline.

From here

What I take from this audit is not a checklist but a change of habit: after setting up a channel, walk it once from the other side. Requesting the file is not enough — you have to ask whether the address inside it has a delivery path. A 200 on the Policy line is not enough — the page that opens has to actually be a reporting policy. Writing "encrypt it with my key" is not enough — the key has to be published. All three were steps I skipped, and one question catches all three: what does the stranger reading this line do next, and will what they do work?

This site has a history of leaving open items for its readers; in the piece where I audited my own promises, eleven of the thirteen commitments I could check were still open. So I will keep the list short. Three items, all measurable:

For hrmarge.com I will either build the delivery path or change the address in its security.txt to one that has a delivery path; if neither happens, the RFC's own verdict says the right move is to remove the file. Whichever it is, I will not call it done before a real test message reaches that address.
I will add a public .github repository to the merbay-erp account with a single SECURITY.md inside it; that closes the gap for five repositories. CORTEX and zee need separate fixes, since their current files point at a disabled button: either I enable private vulnerability reporting or I put a working address in the files.
I will put a security.txt at the root of this site with security@mustafaerbay.com.tr as the Contact. Before writing the file I will create the mailbox and test it with a real message, and I will set Expires under a year with a renewal reminder — otherwise today's file becomes next year's dead door.

The piece where I found that none of my seventeen workflows run when a PR opens arrived somewhere similar: there the missing piece was a trigger, here it is a mailbox. What they share is me looking at a configuration and calling it done. Configuration shows what you intended; whether it works is something you only learn by walking it from the other side.

Official Sources

RFC 9116 — A File Format to Aid in Security Vulnerability Disclosure — the format of security.txt; the mandatory Contact and Expires fields (2.5.3, 2.5.5), the signature and Encryption recommendations (2.3, 2.5.4), and the ruling that stale information may be worse than no file (5.3).
RFC 2142 — Mailbox Names for Common Services, Roles and Functions — section 4, which defines the SECURITY role mailbox.
RFC 5321 — Simple Mail Transfer Protocol — section 5.1: with no MX record, the address record is treated as an implicit MX.
GitHub Docs — Configuring private vulnerability reporting for a repository — enabling the structured reporting form on public repositories.
GitHub Docs — Creating a default community health file — how a SECURITY.md in the .github repository becomes the default for other repositories.
CISA — BOD 20-01: Develop and Publish a Vulnerability Disclosure Policy — the fixed path and format required of federal agencies, and where security.txt sits in the directive.

The Process Blew a Budget That Wasn't Its Own

Mustafa ERBAY — Wed, 11 Nov 2026 06:36:20 +0000

This machine was set up in January 2025 — the stamp on /var/db/.AppleSetupDone
says the 20th. My expectations were low on opening
/Library/Logs/DiagnosticReports: the name suggests crash dumps, and crashes are
rare on this machine. The folder had 58 files in it. Application crash reports: zero.

What was inside was something else: the records of two resource budgets my machine
keeps on every process running on it. Fifty-five of them in seven days. Every single
one carried the same line — Action taken: none.

In a moment I'll get to the thing I missed on my first read: there is a process
name at the top of each report, and the budget does not belong to that process. But
let's count what is in the folder first.

What is in the folder

The measurement is from the morning of 11 November 2026. The machine is a MacBook
Pro Mac16,7 with 24 GB of memory, running macOS 26.6.2 (build 25G83) on kernel
xnu-12377.161.14. The whole folder is 10 MB. The reports span eight separate
calendar days but only 6 days 23.8 hours end to end: the earliest violation is
4 November 08:34:06, the latest 11 November 08:21:38. "Seven days" throughout this
piece means that span.

$ ls /Library/Logs/DiagnosticReports | sed 's/.*\.//' | sort | uniq -c
   1 DiagnosticLogs
  56 diag
   1 ips
   1 panic
   1 Retired

Two of those entries are empty directories (Retired, DiagnosticLogs). The single
.ips file is not a crash report either: its first line reads "bug_type":"298",
making it a jetsam record — a process killed under memory pressure. It is stamped
8 November 06:30, names com.apple.Virtualization.Virtual as the largest process,
and lists all 699 processes alive at that moment. Of the 56 .diag files, 55 are
likewise neither crashes nor panics: they are resource budget records.

The .panic file is its own story and goes in here as a margin note, because I had
no idea it existed. Its first line reads "bug_type":"210", stamped 5 November at
08:57:01. A ResetCounter file written three seconds later says "Reset count: 1". I
asked an independent source when the machine had come up:

$ sysctl -n kern.boottime
{ sec = 1793858196, usec = 877287 } Thu Nov  5 08:56:36 2026

So the panic report was written 25 seconds after boot. That is the expected order,
because macOS writes the panic record on the next boot rather than from the kernel
that died. Counting backwards from the Time Since Boot: 516200s field in one of
the reports from the morning I took the measurement — 11 November 08:19 — lands on
08:56:27, ten seconds off what kern.boottime says and close enough for two
independent methods.

The file's panicString says what happened: Sleep transition timed out after 180
seconds while creating hibernation file. So the machine did not crash under me at
the keyboard; it hung on the way into sleep. My applications not crashing was true
enough — what failed was the machine's own sleep transition. On the morning of
5 November it restarted, filed the paperwork, and I found out six days later while
looking for something else. This article is not about that panic,
but it is a decent reminder of why I opened the folder: records you don't read keep
counting things on your behalf.

Two kinds of event turned up, and the split surprised me:

cpu usage    22
disk writes  33

My first move here was to blame my own command, because my own command was guilty.
The sed 's/.*\.//' above takes everything after the last dot, so it counted all 56
files as diag — but the filenames already separate the two kinds. Twenty-two
reports end in …cpu_resource.diag and every one of them is cpu usage; the other
33 are plain .diag and every one is disk writes. No crossover at all. The folder
didn't mislead me; my one-liner trimming the extension did — and this is only the
first of two places in this piece where my own scanning command produced the
finding.

Two budgets, two opposite policies

Reading the reports turned up two separate limits. On the CPU side what is identical
across all 22 reports is the limit lines; the measurement line changes in every one.
Here is one of them:

Event:            cpu usage
CPU:              90 seconds cpu time over 111 seconds (81% cpu average),
                  exceeding limit of 50% cpu over 180 seconds
CPU limit:        90s
Limit duration:   180s

The numbers on the left move across the 22 reports: the CPU time spent is always
exactly 90 seconds, but the window it fits into runs from 94 to 170 seconds and the
average from 53% to 96%. There are 18 distinct CPU: lines across 22 reports, and
the one above appears in exactly one of them. What never changes is the right-hand
side: exceeding limit of 50% cpu over 180 seconds, CPU limit: 90s,
Limit duration: 180s.

Both numbers sit in the kernel source. Lines 261-262 of osfmk/kern/task_policy.c
read DEFAULT_CPUMON_PERCENTAGE 50 and DEFAULT_CPUMON_INTERVAL (3 * 60). Fifty
per cent, three minutes. Every process that produced a report on my machine sits at
that default, so none of them had raised its ceiling. A process that raised its
ceiling and never crossed it wouldn't appear in this list at all, and this data
can't tell me about those. Where the right to raise it stops is written in the same
file: a process asking for a percentage or an interval above the global cap is
pulled back to the default unless it is cpumon_entitled — and the kernel doesn't
do that quietly, it printfs a "denied attempt to escape CPU monitor" line. One
wanting to switch the monitor off entirely has to present the
com.apple.private.kernel.override-cpumon entitlement. The word private in that
name tells you plainly enough who gets to use it.

The write side is not fixed. Across the 33 reports I found four different limits:

Limit as printed	Actual value	Reports
2147.48 MB	2 GiB / 24 hours	21
8589.93 MB	8 GiB / 24 hours	9
34.36 GB	32 GiB / 24 hours	2
137.44 GB	128 GiB / 24 hours	1

Each is four times the one above it. That is not a coincidence; it is a rule written
down in the source. In osfmk/kern/coalition.c, the part of coalition_io_rate_exceeded() that runs
after the report has been sent looks like this:

/*
 * Continue to monitor the coalition after it hits the initital limit, but increase
 * the limit exponentially so that we don't spam the listener.
 */
new_limit = (lei.lei_limit / 1024 / 1024) * 4;
coalition_lock(coalition);
if (new_limit > COALITION_MAX_LOGICAL_WRITES_LIMIT) {
    coalition_io_monitor_ctl(coalition, COALITION_IO_LEDGER_DISABLE, -1);
} else {
    coalition_io_monitor_ctl(coalition, COALITION_IO_LEDGER_ENABLE, new_limit);
}

The comment states the intent honestly: the point is not to punish, it is to avoid
flooding the listener with notifications. But the consequence is this — whatever
writes heavily to disk gets a budget four times wider after every violation. The
ceiling lives in osfmk/kern/coalition.h as 1ULL << 24 megabytes, which is 16 TiB.
Start at 2 GiB and quadruple: by the seventh violation the limit in force has
reached 8 TiB, the new limit that violation asks for comes to 32 TiB, the ceiling is
overshot and monitoring is switched off altogether. Make enough noise and an app
group earns silence.

What this budget actually counts

I had to understand the number before writing the sentence "it wrote 2 GiB," because
the report does not say written. It says file backed memory dirtied. That is not
bytes landing on the SSD.

The function that feeds the ledger is in osfmk/kern/task.c and it handles four
separate kinds: TASK_WRITE_IMMEDIATE, TASK_WRITE_DEFERRED, TASK_WRITE_METADATA
and TASK_WRITE_INVALIDATED. The first three are added to the account; the fourth —
pages dirtied and then invalidated before reaching disk — is subtracted from it.
So this is not a total, it is a running account that goes up and down. A process that
dirties memory and then abandons the work gets part of its budget back.

The second detail is more practical. The call that updates the group ledger only runs
for writes to internal storage; writes to external storage go to a separate ledger
called logical_writes_to_external and never touch the group budget. Move a heavy job
to an external disk and you become invisible to this budget. That is not a suggested
workaround; it is the reason reading these reports as "here is what my machine writes
most" will always come up short. Work that writes to an external disk never shows up
in this list at all.

The third shows up in the numbers themselves. In 20 of the 33 reports, Writes caused sits at most 0.02 MB above the limit, and in 26 of them the overshoot stays
under 1 MB: the notice fires the instant the ledger touches the line, so the spill
is close to nothing. Exactly one report makes that pattern unrecognisable:

Command:          Autoupdate
Path:             /Applications/ChatGPT.app/Contents/Frameworks/Sparkle.framework/…
Writes:           2379.91 MB of file backed memory dirtied over 11 seconds
                  (216.27 MB per second average), exceeding limit of
                  24.86 KB per second over 86400 seconds

It spent an entire day's budget in eleven seconds. The ledger caught it 232 MB past
the line; the largest overshoot among the other 32 reports is 6.40 MB. It also took
me a moment to see that the "24.86 KB per second" in that report is the same 2 GiB
budget divided by 86400 seconds, written two lines down in a second unit.

I never once broke the CPU budget

Not one of the 22 CPU reports names an application I installed. The distribution:

7  knowledgeconstructiond
6  spotlightknowledged
3  FPCKService
2  duetexpertd
2  com.apple.Virtualization.VirtualMachine
1  mds_stores
1  apfsd

Nineteen of the paths start under /System/ and three under /usr/libexec. All of
them are macOS's own binaries. The top two — knowledgeconstructiond and
spotlightknowledged — are the background workers of Spotlight and the on-device
knowledge index. Thirteen times in seven days they filled half a core across a
three-minute window and had a report written about it. Because the CPU limit does not
rise after a violation, the same process gets reported again every time. For
knowledgeconstructiond that comes to seven filings in seven days — and the last one
is stamped 08:21 on the morning I took the measurement.

Even the apparent exception, com.apple.Virtualization.VirtualMachine, is Apple's
binary — it just happens to be hosting my virtual machine. So even in the CPU ledger
the binary being named belongs to Apple while the work belongs to me.

The other side inverts. Of the 33 write reports, 12 belong to a named third-party app
group, 6 to a com.apple.* group, and 15 to a group carrying only a number. My
machine tires itself out on CPU; I'm the one tiring out the disk.

The finding: the process on the report doesn't own the budget

On my first read I took the report to mean: a process crossed a limit, the system
noted it. I had it wrong. There are two separate fields at the top, and I had skipped
the second:

Command:            git
Path:               /opt/homebrew/*/git
Event:              disk writes
Writes limit:       2147.48 MB
Resource Coalition: "com.anthropic.claudefordesktop"(17488)

The budget is not git's. It belongs to the coalition git was running inside —
macOS's grouping of an application, its helper processes and its children under a
single resource account. git was only the last straw. The kernel draws that distinction far more clearly in its own log line than the
report does:

kernel: Coalition [64708] caught causing excessive I/O (flavor: 2).
        Task I/O: 8192 MB. [Limit : 8192 MB per 86400 secs].
        Triggered by process [89137]

"Coalition … caught" and "triggered by process …" sit in the same sentence: the
group is what gets caught, the process is what pulls the trigger. Because the report
file doesn't draw that line in its heading, anyone who reads Command and stops
there knocks on the wrong door.

My data showed this most clearly on a single day. On 5 November, group number 795
produced four reports:

Violation time	Process on the report	Limit at the time
09:02:31	`find`	2 GiB
09:36:41	`qemu-system-x86_64`	8 GiB
10:10:06	`qemu-system-x86_64`	32 GiB
18:04:43	`Python`	128 GiB

Four different binaries, one budget, quadrupling each time. Only the last of the four
resolved the group's name: "com.openai.codex"(795). The other three carry just the
number — I'm inferring they belong to the same group because the number matches; the
reports don't say so themselves. So that day four violations were charged to a coding
agent's account, and three of the reports never mention the agent at all.

Among the named groups, the two heaviest writers are the two AI tools I use every
day: the name com.anthropic.claudefordesktop appears on four reports,
com.openai.codex on two. Count the whole 795 chain above as Codex and the second
figure becomes five — but that is an inference from a number, not something the
reports state. On the Claude side, three of the four reports name something the tool
runs (git, node, ditto) and the fourth names the tool itself (Claude). I had
been asking which thing
wears my disk out most in the setup running on this machine; the answer came back as
"the things working on my behalf." I should have expected that, and I didn't.

What these files say about me

I was curious about the privacy side, because the folder's name makes you cautious.
All 55 reports carry the executable's path, the hardware model and a stack dump;
43 carry a Codesigning ID and 32 a Team ID — seven distinct developer identities
in total. The stack dumps are not small: the 56 .diag files come to 50,447 lines
together.

My first attempt reassured me, and then I saw that what reassured me was my own
command. I searched for my home directory and got zero:

$ grep -l "/Users/mustafaerbay" /Library/Logs/DiagnosticReports/*.diag | wc -l
0

Searching for the folder rather than the username changed the picture: seven reports
contain /Users/. The forms they take are /Users/USER/*/cargo,
/Users/USER/*/rustc, /Users/USER/*/node and /Users/USER/Library/Application.
macOS rewrites the username to a literal USER and intermediate directories to * —
the /opt/homebrew/*/git you saw in the git report above is the same redaction. So
the string I searched for could never have matched; my command produced the "nothing
here" result. The second time in this piece that the same mistake caught me.

What survives the redaction is not nothing. That I run cargo, rustc and node is
readable straight off these files. The panic report carries a stable per-machine
crashReporterKey; the resource records don't. Which document I opened isn't written
here — but 50,000 lines of stack dumps say plenty about which software I run and
when.

The two records don't agree

Having counted the report files, I wanted to test the count with a second method. That
os_log line the kernel writes at the moment of a violation was exactly right for it:
independent of the report file, written straight off the ledger.

/usr/bin/log show --style compact \
  --predicate 'eventMessage CONTAINS "caught causing excessive I/O"' \
  --start "2026-10-27 00:00:00"

Although I started the query on 27 October, the oldest kernel line I could find in
the unified log was from 9 November 08:18 — I
measured why that window is shorter than I assumed
a while ago. Within the overlapping range the kernel wrote five violation lines. Over
the same range there are four write reports on disk. Three of them pair up.

So two violations exist in the kernel log with no report file, and one report file
exists with no kernel line. Neither record is complete on its own. I'm writing this
down because I have already paid for reading a number as "not on the list, therefore
it didn't happen" on this blog: those 55 reports are not every violation that
occurred, only the ones that became files.

Three places I tripped while measuring

The report's date is not the event's date. Every report carries both a Date/Time
and an End time. Date/Time is the start of the measurement window; the violation
happens at End time. Because the window runs for hours in write reports, the gap
between them reaches 23 hours. I grouped by Date/Time first and the events landed on
the wrong days; worse, I paired a Claude report with the wrong kernel line and produced
a "finding" that the two records disagree by a factor of four at the same instant. No
such finding exists. Matching on End time — 01:47:44 plus 18008 seconds, exactly
06:47:52 — made the two records agree precisely. The stamp in the filename is the
moment the file was written, not the moment of the event either.

My scanning command corrupted the result three times. One of the six lines the
log query returned was not a violation but the record of my own log show command:
the command line carried the text I was searching for as an argument, so it matched
my own search. Together with the two traps above — the sed trimming the extension
and the grep defeated by the USER redaction — that makes three. In all three the
finding came not from the data but from how I looked at it.

The source for my kernel version isn't published. The running kernel is
xnu-12377.161.14; the newest tag in Apple's open source repository is
xnu-12377.121.6. The lines I quoted above come from the published tag — same branch,
but not literally my kernel. The numbers matching the reports on my machine closes that
gap, but I would rather write it down than assume it closed.

The part I couldn't establish

I could not find where the initial 2 GiB limit comes from in published source. The
kernel takes the group budget in from outside: the
COALITION_LEDGER_SET_LOGICAL_WRITES_LIMIT operation in bsd/kern/sys_coalition.c
lands on a coalition_ledger_set_logical_writes_limit(coal, limit) call. So a
user-space component supplies the starting value, and that component isn't open
source. One thing
worth keeping separate: the same repository records a per-process I/O monitor default
of 20480 MB as IOMON_DEFAULT_LIMIT — but that is a different ledger. The numbers in
my reports come from the group ledger, and both the report files and the kernel log say
2048 MB for the starting value. I use the number because two sources agree on it; I am
not claiming it is "the default."

One more thing stayed open. I assumed none of these records leave the machine, and the
settings file both agrees and doesn't.
/Library/Application Support/CrashReporter/DiagnosticMessagesHistory.plist has
AutoSubmit and ThirdPartyDataSubmit set to false — sharing is off. The same
file also carries LastFullSubmissionCalled = 2026-11-10 07:14:43 +0000 and, one
second later, LastFullSubmissionSuccess = 2026-11-10 07:14:44 +0000. I could not establish
from primary sources what that submission carries, so I am not claiming anything went
anywhere. I am only reporting that the file writes both lines.

If you want to look at your own machine

These files are readable by the _analyticsusers group, which means your own user can
read them. Four steps:

ls /Library/Logs/DiagnosticReports — how many files, how many are genuine crashes (.ips), how many are resource records (.diag).
In one .diag file, read the Event, Resource Coalition and Action taken lines. Don't stop at Command; the budget belongs to Resource Coalition.
Note the Writes limit. If it is larger than 2147.48 MB, that group has violated before and had its limit multiplied; divide by four repeatedly to count how often — assuming the start was 2 GiB. The count belongs to that coalition instance, not to the application's lifetime: on my machine com.anthropic.claudefordesktop shows up under four separate coalition numbers (773, 11637, 17488, 71862), each starting again at 2147.48 MB. Quitting and relaunching the app resets the counter, and coalition.c restarts the 24-hour period on every re-arm too.
When grouping by day, use End time, not Date/Time and not the filename.
If the group shows only a number — 15 of my 33 write reports do — the one practical route is matching the kernel log's Triggered by process [pid] against ps at the moment it happens. Look later and that pid belongs to somebody else.

The conclusion I drew isn't a settings change. All 55 of my reports say
Action taken: none — but the field being called Action taken says other actions
are possible; task_policy.c calls proc_apply_resource_actions on a CPU violation,
and per-thread limits can raise an exception. None of that fired on my machine. What
changed isn't a switch, it's how I read these files.

There is nothing to blame on the CPU side: my machine is doing its own background work
and measuring it against a half-a-core line over a three-minute window. On the write
side, ownership matters more than the number. A report doesn't say "git wrote too
much," it says "the daily budget of the app group git was running in is spent" — and
that group exists because I started it. I learned that the things wearing my disk out
most are the tools working on my behalf, and I learned it from a record my own machine
keeps. And that record is tuned to grow a little more generous after every
violation, so whatever makes enough noise gradually stops being seen.

I have looked into a limit ledger twice more on this blog in the past month, and
honestly both started somewhere close to here: once on
blowing past CommitLimit with nobody stopping me,
once on
only one of five processes under a single limit getting killed.
The "the limit warns and then does nothing" part was in those pieces too; that isn't
what's new here. Two things are: the owner of the budget is not the process named on
the report, and the budget doesn't shrink when it is broken — it grows. Without the
first you read Command and chase the wrong process; without the second you take a
137 GB limit for a choice Apple made, when that number was written by that group's
own history.

Reading a number without asking whose account it was charged to is from the same
family as looking at an average and missing the stall.

Official Sources

Not Even Root Can Move a Managed Interrupt

Mustafa ERBAY — Tue, 10 Nov 2026 11:35:36 +0000

Back in April I wrote a SoftIRQ saturation and IRQ affinity runbook on this blog. Its third section covers what to do when interrupts pile onto one core: choose cores according to the card's queue count, map the IRQs onto those cores, verify the distribution with /proc/interrupts. There's also a sentence saying deliberate pinning is the safer option on critical production nodes.

Seven months later I ran that step end to end for the first time on my own machine, the server that hosts this blog. On twenty of the forty interrupt lines the mapping step isn't possible at all: the mask I write as root is refused, and the error that comes back says "I/O error" — as though something were wrong with a cable. On the other twenty, pinning works, and it works instantly. What doesn't work is the move people reach for first: widening the mask.

This article is the measurement of those two halves. In an October piece about rq_affinity I dropped a side observation that "all the interrupts land on one core" — that experiment ran on a different VM with 14 cores and a single hardware queue, and the subject there was which ledger completions get written into. I never looked at why the interrupt itself wouldn't move. I'm looking now.

The machine and the method

The measurements were taken on 10 November 2026 on the server hosting this blog: Ubuntu 24.04.4 LTS, kernel 6.8.0-147-generic, 18 vCPUs, QEMU/KVM (i440FX), storage on virtio-scsi (PCI 00:05.0), network on virtio-net (PCI 00:12.0), up for three weeks and one day. The irqbalance service is inactive — so there's no background process quietly balancing interrupts.

Since I didn't want to play with routing on a live server, the work started with a side-effect-free probe: write each line's own current mask back into itself. The value is identical, so nothing changes; if the write is refused, the errno comes back.

for d in /proc/irq/[0-9]*; do
  cur=$(cat "$d/smp_affinity") || continue
  if err=$( (printf '%s' "$cur" > "$d/smp_affinity") 2>&1 ); then
    printf '%-4s WRITE_OK\n' "${d##*/}"
  else
    printf '%-4s WRITE_FAIL %s\n' "${d##*/}" "$err"
  fi
done

Distributions were counted by hand too; subtracting the columns of a /proc/interrupts line between two snapshots is enough:

a=$(grep -E '^ *46:' /proc/interrupts); sleep 20; b=$(grep -E '^ *46:' /proc/interrupts)
awk -v A="$a" -v B="$b" 'BEGIN{ split(A,x," "); split(B,y," ");
  for(i=2;i<=19;i++){ d=y[i]-x[i]; if(d>0) printf "CPU%d=%d ", i-2, d } print "" }'

Two side effects, stated up front. First: on four old ISA lines with no driver attached (3, 5, 7, 13), effective_affinity_list read 0 before the probe and 0-17 after it. The reason sits in kernel/irq/manage.c, two lines above the if block quoted later on: irq_set_affinity_deactivated() copies the mask and updates the effective mask with it, without calling the chip driver at all, for an interrupt that isn't activated yet. Second: during the migration experiment further down, the NIC's receive interrupt spent a while on CPU3; when the experiment finished I put it back where it was, on CPU9, and verified the final state.

Twenty of forty refused the write

The result came out surprisingly sharp:

Result	Lines	Which ones
`WRITE_FAIL` (I/O error)	20	0, 2, and the eighteen `virtio1-request` lines from 27 to 44
`WRITE_OK`	20	1, 3-15, 24, 25, 26, 45, 46, 47

Eighteen of the twenty refusals belong to a single device: virtio1-request, the request queues of the virtio-scsi controller. Each one's mask is a single CPU — IRQ 27 only CPU0, IRQ 28 only CPU1, and so on to IRQ 44 on CPU17.

My first instinct was that the refusal might be because the value was identical. It isn't: I wrote 3 (CPU0 and CPU1) to IRQ 27 instead of its current mask, got the same error, and smp_affinity_list still read 0. The refusal is about the line's identity, not the value.

The remaining two lines are 0 and 2, the timer and the cascade. The gate in the code actually checks three things at once — that the line can be balanced, that its affinity isn't kernel-managed, and that noirqaffinity wasn't passed at boot — and I could not determine by measurement which of them these two tripped. Why I couldn't comes later.

Why eighteen lines refuse

The answer is in the block layer. Under sda's mq directory there are eighteen hardware queues, each with a cpu_list of exactly one core:

/sys/block/sda/mq/0/cpu_list  → 0    (nr_tags=256)
/sys/block/sda/mq/1/cpu_list  → 1    (nr_tags=256)
...
/sys/block/sda/mq/17/cpu_list → 17   (nr_tags=256)

Every core has its own queue and every queue its own interrupt vector. The driver asked for that arrangement: when drivers/scsi/virtio_scsi.c looks up its queues it hands the kernel a placement descriptor (struct irq_affinity desc = { .pre_vectors = 2 }), and in response the virtio-pci layer adds the PCI_IRQ_AFFINITY flag and calls pci_alloc_irq_vectors_affinity(). The kernel's PCI documentation describes what that flag does in one sentence: it spreads the interrupts around the available CPUs.

Interrupts distributed this way are what the kernel calls managed, and it deliberately blocks user space from touching them. The write handler in kernel/irq/proc.c closes the door on its first line:

    if (!irq_can_set_affinity_usr(irq) || no_irq_affinity)
        return -EIO;

And irq_can_set_affinity_usr() in kernel/irq/manage.c combines two conditions:

bool irq_can_set_affinity_usr(unsigned int irq)
{
    struct irq_desc *desc = irq_to_desc(irq);

    return __irq_can_set_affinity(desc) &&
        !irqd_affinity_is_managed(&desc->irq_data);
}

This also explains the most confusing part of the table: on the same machine the virtio1 lines refuse while the virtio2 lines accept, even though both are virtio. The difference is in the driver. The network driver, drivers/net/virtio_net.c, hands over no such descriptor when it looks up its queues — the irq_affinity struct doesn't appear anywhere in the file — and writes the placement itself through virtnet_set_affinity(). With no descriptor there's no PCI_IRQ_AFFINITY, the interrupt isn't considered managed, and the procfs door stays open. So there's no rule saying "modern device, therefore managed"; what decides is whether the driver handed over that descriptor.

That has a further practical consequence: a mask you write by hand onto the network interrupts can be rewritten by the driver's own virtnet_set_affinity() call when the queue count changes or a core goes offline. Your pinning isn't a lasting contract.

Why managed interrupts exist, and how that reaches you

I could have waved this away as "don't break the design," but the kernel's own rationale is more concrete. managed_irq.rst, added to the documentation in April 2026, explains it: an NVMe device with 128 queues typically requests one interrupt per queue, two such devices make 256 interrupts, and on x86 there are only 256 vectors per CPU with some already reserved. Normally this isn't a problem, because the interrupts are spread across cores — but when the system suspends, every secondary CPU is taken offline and all interrupts migrate to the one core still running. That's where the vector space runs out and the suspend fails. Managed interrupts exist precisely to prevent that migration.

The operational face of this is in the same document: when a core in the mask goes offline the interrupt moves to the next core in the mask, and when the last core in the mask goes down the interrupt is shut down, with the driver required to have quiesced the matching queue. On my machine each queue's mask is a single core, so that "last core" case is the same thing as taking that core offline.

The same page also gives the supported way to keep managed interrupts away from particular cores: isolcpus=managed_irq,<mask> at boot. It's best-effort — if the automatically computed mask has no online core outside the avoided set, the setting has no effect. So when a managed interrupt lands on a hot spot you aren't empty-handed; the lever just isn't in /proc, it's on the boot line.

"I/O error" was the wrong word, and the kernel agreed

EIO — "Input/output error" — makes an administrator think of hardware trouble. What's actually happening is a permission matter. Others noticed before me.

First, irqbalance. It's the software that actually writes these files, and it translates the error internally; in activate.c the first line of the write-error branch reads, verbatim:

error:
    /* Use EPERM as the explaination for EIO */
    errsave = (errsave == EIO) ? EPERM : errsave;

There's more: if the error is in the permanent class, irqbalance drops that interrupt to BALANCE_NONE, logs "IRQ %i affinity is now unmanaged", and never bothers with it again. On my server it's off anyway; had it been running, it would have quietly given up on those twenty lines.

Second, the kernel itself. Two patches landed a day apart in August 2024, and both shipped in v6.12:

eb29369fa543 changed the returned error from EIO to EPERM; the commit message states the rationale plainly.
c7718e5c76d4 is the more radical one: the kernel already knows at allocation time whether an interrupt can be set from user space, so it should create the procfs files for the ones that can't as read-only (0444).

The consequence of the second patch: on a newer kernel a single ls -l shows which interrupts you can't move. On my 6.8 it doesn't, because all forty files are 0644. The twenty files that refuse every write look exactly as writable as the rest.

Rather than assume the version map, I scanned it. The value returned in kernel/irq/proc.c is EIO on v6.9, v6.10 and v6.11, and EPERM from v6.12 through v7.3 and on master. Then I saw the fixed behaviour in a lab: an Ubuntu 25.10 VM on my Mac running kernel 6.17.0-40-generic, where 16 of its 36 interrupt lines are mode 0444, the block devices' request queues among them. Writing a line's own mask to one of those as root gets you Operation not permitted.

A small aside: that fix also made irqbalance's translation line redundant. The kernel now returns EPERM directly, and since the file is read-only the write usually isn't even attempted.

The diagnosis — the answer to "why won't this line move?" — is two years and one LTS generation away from my server's kernel. On the same machine apt-cache policy linux-generic-hwe-24.04 currently offers 7.0.0-42 as the candidate. One package, but jumping a kernel generation on a production server isn't a decision you make for the convenience of ls -l, and I didn't make it for this article.

The twenty that accept: a narrow mask moves them, a wide mask does nothing

Now the other half. On all twenty lines that accept the write, smp_affinity_list reads 0-17 — on paper, "this interrupt may land on any of eighteen cores." But effective_affinity_list on those lines shows a single core, and the counters back the effective mask. Three 20-second runs on the two busiest lines, the network card's receive and transmit queues:

Run	IRQ 46 (`virtio2-input.0`)	IRQ 47 (`virtio2-output.0`)
1	5,152 interrupts → CPU9 (100%)	4,895 interrupts → CPU10 (100%)
2	7,133 interrupts → CPU9 (100%)	7,677 interrupts → CPU10 (100%)
3	9,855 interrupts → CPU9 (100%)	10,281 interrupts → CPU10 (100%)

44,993 interrupts in total, with seventeen columns reading exactly zero for each line. In the same minute the local timer interrupt LOC was spread across all eighteen cores, so the machine is running multi-core.

So what happens if I narrow the mask? I didn't want to write this up without measuring it, because that's the real question here. I wrote 8 (CPU3 only) to IRQ 46, then 200 (CPU9 only), counting for ten seconds at each step:

Step	`smp_affinity_list`	`effective_affinity_list`	interrupts in 10 s
A — wide mask	`0-17`	`3`	2,093, all on CPU3
B — wrote `200`	`9`	`9`	1,694, all on CPU9
C — back to `3ffff`	`0-17`	`9`	2,565, all on CPU9

Three rows, three separate statements. B: a narrow mask works, and it works instantly — by the first read after writing the file, the effective core had already changed. On a line taking a few hundred interrupts a second there's nothing to wait for. C: reopening the mask to eighteen cores does nothing at all; the interrupt stays on CPU9, neither spreading nor being re-picked. A: it was already sitting on CPU3 with a wide mask, because I had put it there in the previous step and nobody moved it back — irqbalance is off, and the kernel has no business redistributing on its own.

That lets me rename the mask properly: smp_affinity is a candidate set, not a distribution order. An interrupt delivered over MSI-X goes to a single target; the kernel picks one core out of the set you gave it. Narrow the set and you force the choice — that works. Widen it and you've asked for nothing. April's runbook saying "deliberate pinning" turns out to have been right for exactly this reason; what's wrong is the instinct that opening the mask splits the load.

The exception the floppy drive taught me

There is a case where the migration really doesn't happen, and hardware this machine doesn't have showed it to me. IRQ 6 belongs to floppy: there's no floppy drive, the line is there, and since boot — three weeks and a day — it has counted three interrupts, all on CPU16.

start:       smp_affinity_list=0-17   effective_affinity_list=16
echo f  → OK smp_affinity_list=0-3    effective_affinity_list=16
echo 2  → OK smp_affinity_list=1      effective_affinity_list=16
restore → OK smp_affinity_list=0-17   effective_affinity_list=16

All three writes succeed, and the mask file reads back exactly what I wrote each time. The effective core doesn't move: CPU16, CPU16, CPU16. What happened instantly on IRQ 46 never happens here, because the difference isn't in the write, it's in the interrupt.

The mechanism is written down in kernel/irq/manage.c. The write request lands in irq_set_affinity_locked(), and if the chip cannot move the interrupt in process context, the change isn't applied immediately, it's put pending:

    if (irq_can_move_pcntxt(data) && !irqd_is_setaffinity_pending(data)) {
        ret = irq_try_set_affinity(data, mask, force);
    } else {
        irqd_set_move_pending(data);
        irq_copy_pending(desc, mask);
    }

The place where a pending move is applied is the path the interrupt itself travels: apic_ack_edge() in arch/x86/kernel/apic/vector.c calls irq_complete_move() and then goes on to apic_ack_irq(); the irq_move_irq() call lives there, and the pending mask finally takes effect in irq_move_masked_irq() in kernel/irq/migration.c. In other words, an interrupt migrates to its new core only the next time it fires. On a network queue taking hundreds of interrupts a second that takes microseconds; on a line that fired three times in three weeks, when "next time" happens is anybody's guess.

So why did the mask file show what I wrote? That's in proc.c: if a move is pending, procfs prints the pending mask instead of the real one.

        mask = desc->irq_common_data.affinity;
#ifdef CONFIG_GENERIC_PENDING_IRQ
        if (irqd_is_setaffinity_pending(&desc->irq_data))
            mask = desc->pending_mask;
#endif

That display is still in today's master, only the helper names have changed. The decision gate itself, though, has not stayed put: v6.14 introduced the IRQCHIP_MOVE_DEFERRED flag (a648eb3a3f79) and x86 was converted to that model (7d04319a05ab). Deferral is now a property of the chip rather than a per-interrupt marking, and on master the IO-APIC and MSI chips carry the flag. The "applied immediately if remapping is present" distinction that holds on my 6.8 doesn't hold in the same shape on newer kernels.

I can't ask the kernel this question

Earlier I said I couldn't determine which condition made those two lines refuse. Here's why. The kernel has a debug interface that exposes per-interrupt flags, but it depends on a build option, and my server's configuration has it off:

# CONFIG_GENERIC_IRQ_DEBUGFS is not set

That's why there's no /sys/kernel/debug/irq directory — debugfs is mounted, there are forty-odd other directories inside it, this one isn't there. So on 6.8 the only answer I can get to "is this interrupt managed?" is the errno of a write attempt. The question isn't free: to get the answer you have to touch the thing. That's why the probe had to be side-effect-free. And the kernel's own documentation recommends exactly this debugfs file for confirming a managed interrupt's state — a recipe that doesn't run on my box as long as the distribution doesn't compile it in.

The documentation: one page from 2011, one from this April

The page that comes to mind first for this knob is Documentation/core-api/irq/irq-affinity.rst. Seventy lines, and reading it surprised me twice.

First, its example: on an eight-CPU box the IO-APIC-level eth1 line spreads interrupts across CPU0-3 once the mask is set to 0f (1068, 1785, 1785, 1783), and moves to the other four with f0. On my machine, opening the mask spreads nothing. Both are correct — one describes an IO-APIC level-triggered card, the other a virtio device using MSI-X — but the document doesn't help you tell which situation you're in. Second, the omissions: neither the effective_affinity file nor managed interrupts appear on that page. The only sentence saying writes aren't entirely free is the warning that you're not allowed to switch off all CPUs.

The page's stillness got measured too: the difference between v6.8 and today's master is four lines — "cpu" capitalised to "CPU", and a missing grep restored in the example. The last addition to its content was the smp_affinity_list paragraph in 2011; all five touches since then were spelling, formatting and a move.

But ending the story there would be unfair, because the kernel closed this gap seven months ago. On 1 April 2026, commit bb6a85b4b652 added a new page in the same section: managed_irq.rst, listed directly beneath irq-affinity. Both of this article's findings are in it: that the affinity of managed interrupts is handled entirely in the kernel and cannot be modified through the /proc interfaces, and example output where smp_affinity_list and effective_affinity_list diverge. The example hardware is too familiar to be a coincidence: a QEMU instance booted with -device virtio-scsi-pci, three management interrupts and one interrupt per queue.

The only problem is that the page doesn't exist where I work. It's absent in v7.0 and present from v7.1 on. The official account of managed interrupts is three generations ahead of the kernel on this server. The knowledge exists; it just isn't on the page I was reading or in the kernel I'm running.

What the real lever is on this machine

The measurement showed that the "open the mask, let the load spread" move is empty on this box. The right doors got measured as well.

The network side. ethtool -l eth0 says one thing: Combined: 1, with a pre-set maximum of 1. The device has a single queue pair; there's no second interrupt vector to spread. Fiddling with interrupt affinity on a card like that is like looking for a second cashier in a shop with one till. If the load genuinely saturates one core, the lever is on the software side: RPS. That's off too — /sys/class/net/eth0/queues/rx-0/rps_cpus reads 00000. I haven't touched it, because I have not measured a bottleneck right now.

The block side. Nothing to do there, because it's already done. The sixty-second delta for the eighteen virtio1-request lines:

line	interrupts	landed on	line	interrupts	landed on
IRQ27	1,350	CPU0	IRQ36	934	CPU9
IRQ28	1,259	CPU1	IRQ37	883	CPU10
IRQ29	1,122	CPU2	IRQ38	1,095	CPU11
IRQ30	1,111	CPU3	IRQ39	1,128	CPU12
IRQ31	1,396	CPU4	IRQ40	1,131	CPU13
IRQ32	670	CPU5	IRQ41	1,120	CPU14
IRQ33	1,320	CPU6	IRQ42	1,058	CPU15
IRQ34	1,364	CPU7	IRQ43	640	CPU16
IRQ35	1,123	CPU8	IRQ44	676	CPU17

19,380 interrupts in total, not one of the eighteen lines saw a second core, and the core each landed on matches its cpu_list exactly. It's the mirror image of the network table's "mask of eighteen, reality of one": here the mask is one and reality is one, and because the two agree, nobody gets confused.

The list I took away for anyone who has to look at this knob in production:

Read effective_affinity first. smp_affinity is intent, effective_affinity is state. Without seeing the gap it's very easy to conclude "the mask is wide, so it must be spreading."
Verify with the counters. Learn how many cores a line lands on from the /proc/interrupts delta, not from the mask.
On kernel 6.12 or newer, run ls -l /proc/irq/*/smp_affinity. The 0444 ones can't be moved, and asking costs nothing.
On anything older, probe with each line's own mask. The value is identical so nothing changes, and the errno still arrives.
If you want spreading, widening the mask won't give it to you. The measurement is unambiguous: a wide mask neither distributes nor re-picks. Spreading comes from queue count (ethtool -L, the device's num_queues) or from RPS/XPS.
Don't fight the lines that refuse; the right lever is the boot line. isolcpus=managed_irq,<mask> tries to keep managed interrupts off specific cores, and irqaffinity= sets the default for the management interrupts.
Know the cost of offlining a core under managed interrupts. When the last core in the mask goes offline, the interrupt is shut down.
A change on a quiet line may never be applied. The pending move waits for the next interrupt; if it doesn't come, nothing happens.

Conclusion

What I take from this is bigger than one knob. Under /proc and /sys there are two kinds of file: the ones holding what I asked for, and the ones holding what the system is doing. Most of the time they show the same number, so we treat them as one file. The gap between smp_affinity and effective_affinity is exactly this, and it only surfaces when things aren't ordinary — while a move is pending, when the interrupt never fires, when the device distributed its own queues. Which is precisely when you reached for the setting.

As for April's runbook: the "deliberate pinning" advice came out of the measurement intact — a narrow mask genuinely works, and works instantly. What was missing is where that advice cannot be applied — on the interrupts of a driver that asks for a vector per queue, the step never starts, and it tells you so with an "I/O error". The me of seven months ago didn't know that, because I hadn't run that article all the way through on my own server.

And third: the kernel wasn't wrong here, it had just picked the wrong word; it fixed the word two years ago and documented the gap this April. Both are ahead of the version I'm running. Getting the name of an error right does as much work as adding a feature — provided you can reach the version where that name appears.