DEV Community: Mustafa ERBAY

CPU Quota Does Not Look at Your Average

Mustafa ERBAY — Sun, 04 Oct 2026 15:00:33 +0000

It is very easy to look at a container's CPU graph and conclude the thing is loafing. Average
consumption: a fifth of its quota. Flat line. No alerts.

That same container was stopped 509,912 times over the past ten days.

These two sentences do not contradict each other. The contradiction is in what I was looking at.
Linux's CFS quota mechanism has no interest whatsoever in your average; it cares about 100
millisecond windows. And for years, every time I typed --cpus on a container, I was not saying
"half a core per second" — I was saying "50 ms in every 100 ms, no matter what." Those are not
the same thing. In the resource-hog container guide
I wrote back in May, I handed out the --cpus "0.5" example without a second thought; the advice
in that guide is not wrong, but this measurement showed me it was incomplete.

I swept the fleet and found 16 quota'd cgroups

On VPS3 (Ubuntu, 6.8.0-142-generic, 18 cores) I walked every cgroup that had a quota and read
its cpu.stat. In all 16 cgroups where I had set a quota, cpu.max.burst is 0 and the
nr_bursts counter is 0. I will get to why that surprised me shortly.

The resulting table gave me pause, because the two most heavily throttled services were the two
with the least tolerance for latency. Top five by throttle count:

cgroup	quota	periods	throttled	throttle rate	throttled time	avg. quota use
`vpsman-etcd`	0.5 CPU	9,297,557	509,912	5.48%	14.24 hours	17.55%
`licman-etcd`	0.5 CPU	5,195,009	308,177	5.93%	8.45 hours	19.82%
`messageman-api-1`	1 CPU	3,336,709	7,270	0.22%	18 minutes	3.46%
`burncpu-app`	2 CPU	125,783	1,725	1.37%	22 minutes	12.24%
`messageman-pg-1`	1 CPU	201,165	813	0.40%	41 seconds	5.16%

etcd. The consensus ledger. The thing that triggers a leader election when heartbeats are late,
the thing every write waits on. Because its average load is 17% of quota, I had assumed for years
that "half a core is more than enough." Technically true. In practice it stopped etcd half a
million times over ten days.

Cumulative counters tell you about the past, not about today. So I read cpu.stat twice, 120
seconds apart:

vpsman-etcd, 120-second window
  periods     : 1191   (= 119.1 s — the timer almost never goes idle)
  throttled   : 107 times (8.98% of periods)
  throttled time : 12.42 s
  CPU used       : 13.60 s = 22.84% of its quota

One hundred and seven times in two minutes. While using a quarter of its quota. No average-based
dashboard will show you this, because the average genuinely is low — the problem is that the
average is the wrong question.

The window is 100 ms, and there is no average in there

The mechanism is stated plainly in the kernel docs: within each period the group is allocated
quota microseconds of CPU, and once that is exhausted its threads are throttled until the
period refreshes. The default period is 100 ms, the default quota is unconstrained, the minimum
quota and period are 1 ms, and the maximum period is 1 second.

Wondering whether quota carries across periods, I went to the "Caveats" section of the docs —
because I was about to write "it never carries," and I would have been wrong. What it says is
that once a slice is assigned to a CPU it does not expire; if every thread on that CPU
becomes unrunnable, all but 1 ms of the slice may be returned to the global pool. So there is a
small carryover — "typically at most 1 ms per cpu or as defined by min_cfs_rq_runtime." In the
kernel source that constant sits there as 1 * NSEC_PER_MSEC.

The nuance matters, and so does its scale: on an 18-core machine that is a few milliseconds of
residue at best. Rounding error next to an 80 ms peak demand. The same paragraph also names what
this carryover is meant to eliminate: "the propensity to throttle these applications while
simultaneously using less than quota amounts of cpu." The exact phenomenon these counters record,
named in the kernel's own documentation — and still happening on my server.

It is worth doing the arithmetic once. A half-core quota grants 50 ms per period. If your job
needs 80 ms of CPU in one go, it must be split no matter what your average consumption is —
because what is missing is not the budget but its distribution. vpsman-etcd uses 17% of its
quota on average; across 10.76 days of periods it never spent 82% of its entitlement, close to
nine days' worth.

Which is exactly what happened to etcd. For a read-only range request — and for a request this
small, at that:

{"level":"warn","caller":"txn/util.go:93","msg":"apply request took too long",
 "took":"220.804807ms","expected-duration":"100ms",
 "prefix":"read-only range ","request":"key:\"health\" "}

Reading the health key took 220 ms. The work itself takes microseconds. The rest is waiting.

Four ways to misread cpu.stat

I tripped over these counters four times. I could not be confident until I opened the kernel
source (kernel/sched/fair.c, v6.10) and found the lines — and all four run against intuition.

nr_periods is not wall-clock time. The period timer is deactivated when the group goes
idle; the comment at the head of do_sched_cfs_period_timer says so explicitly. So
nr_periods × 100 ms is not elapsed time, it is "time during which there was work." In a lab run
that lasted 124 seconds, the counter showed 750 periods (75 seconds). Inverted, it makes a nice
diagnostic: for vpsman-etcd, 9,297,557 periods works out to 10.76 days against an uptime of
10.80 days — across ten days there was only about an hour in which it had no runnable work.
licman-etcd passes the same test independently: 6.01 days against 6.04.

throttled_usec can exceed wall-clock time. The counter is kept per cgroup but accumulated
separately as each CPU's queue is released: cfs_b->throttled_time += rq_clock(rq) - cfs_rq->throttled_clock. If the group is throttled on four cores simultaneously, 100 ms of real
waiting adds 400 ms to the counter. The measurement itself shows this: dividing 51,261 seconds by
509,912 throttle events gives 100.5 ms per event, which is above the period length — and since no
single period can hold more waiting than that, the excess comes from summing across cores. So the
"14.24 hours" in the table is not real waiting, it is an upper bound.

The counters are not hierarchical. The docs cover this in a single sentence, but the
consequence is heavy: those five CFS fields account only for throttling caused by that cgroup's
own bandwidth limit. If the quota lives on a parent slice (kubepods.slice under Kubernetes, a
parent unit under systemd), the container's own cpu.stat looks spotless. In September, when
I wrote 0 to cgroup.pressure
the counter went quiet; this is the same blindness wearing a different file. There is a remedy,
and it was news to me: cpu.stat.local also reports throttling inherited from ancestors, and it
has been there since Linux 6.6. I found the file on VPS3 (6.8); for my containers the two files
hold identical values, meaning the quota really does sit at the leaf. But in a setup that puts
the quota on a parent unit, that is the file to read. Note that cpu.stat.local is also summed
across CPUs, so it does not solve the second trap.

The threshold in an application log is the application's, not yours. etcd's 24-hour log holds
891 "apply request took too long" lines for vpsman-etcd and 12,395 for licman-etcd. For a
moment I got excited that the distribution never dipped below 100 ms — "look, the exact period
boundary!" No. etcd's expected-duration threshold is 100 ms; it never logs anything below that.
The floor belongs to the logger, not the scheduler. For licman-etcd the actual distribution is
a smooth tail from 100 ms out to a second: p50 141 ms, p90 278 ms, p99 601 ms, worst case 996 ms.

The fix has been in the kernel for four years: cpu.max.burst

Banking unused quota and spending it later is not a new idea. Commit f4183717b370
("sched/fair: Introduce the burstable CFS controller") is dated 21 June 2021 and landed in Linux
5.14; the cpu.max.burst interface arrived in the same release. In v5.13 there is no such thing
as cfs_burst.

The logic is ten lines inside __refill_cfs_bandwidth_runtime:

(The nr_burst and burst_time in the diagram are the kernel's internal field names; in the
cgroup v2 interface you see the same things as nr_bursts and burst_usec.)

The critical line is the ceiling at the bottom: runtime = min(runtime, quota + burst). The most
you can spend in a single period is quota plus burst. The docs give the permitted range for
cpu.max.burst as [0, $MAX], default 0. So if you set burst equal to quota, you can spend at
most twice the quota in one period, and not a microsecond more.

That ceiling helped me design the experiment — because it tells you in advance what should happen.

The experiment: half a burst buys you nothing

In Docker Desktop's linuxkit VM (6.10.14-linuxkit, cgroup v2) I created my own cgroup and wrote
a deliberately spiky workload: each round, idle for 400 ms, then burn 80 ms of CPU, 250 rounds.
Quota 50000 100000. Average demand is only 33% of the quota. The quantity under measurement is
the wall-clock duration of the work — since the burn loop spends exactly 80 ms of CPU as counted
by CLOCK_PROCESS_CPUTIME_ID, the difference is waiting.

I ran three arms:

quota 50000/100000, work: 400ms idle + 80ms CPU x 250 rounds

burst=0       p50  97.3 ms   throttled 250/250 rounds   throt 4.298 s   nr_bursts   0
burst=25000   p50  98.0 ms   throttled 250/250 rounds   throt 4.416 s   nr_bursts 250  burst 6.250 s
burst=50000   p50  80.0 ms   throttled   0/250 rounds   throt 0     s   nr_bursts 208  burst 3.765 s

The third row was what I expected: with burst equal to quota there is not a single throttle event
and latency settles at 80.0 ms — the pure CPU cost of the work. A 22% latency tax, gone.

The second row is what stopped me. With burst set to half the quota, nr_bursts came out at 250:
the mechanism fired on every single round. burst_usec is 6.250 seconds, that is exactly
25,000 µs per round — the budget spent down to the last drop. And latency did not improve by one
millisecond.

The ceiling formula explains why: 50 ms quota + 25 ms burst = 75 ms, and the job needs 80 ms.
Five milliseconds short. Waiting for a period boundary over five milliseconds produces the same
delay as waiting over fifty. Here a half measure arrives at full cost and zero benefit — while
the counters report back that "burst is working!" Green on the dashboard, waiting for the user.

If it were me, I would take exactly one rule away from this: there is no such thing as enabling
burst "a little." If you do not know the peak your workload demands in one go, you do not know
whether the burst value you picked will do anything at all.

The dial nobody turns: period

Here is what gets lost in the burst conversation: cpu.max is two numbers. Everyone tunes the
first one and leaves the second at its default for four years. The same 0.5 CPU ratio, run at
three different periods with burst disabled:

same work (400ms idle + 80ms CPU), 0.5 CPU ratio throughout, burst=0

5000/10000      (10 ms period)   p50 151.0 ms   throttled 2892/3414 periods
50000/100000    (100 ms period)  p50  96.4 ms   throttled  197/600  periods
250000/500000   (500 ms period)  p50  80.0 ms   throttled    0/194  periods

Shortening the period made latency 57% worse. That may read as counterintuitive — I had
assumed "refreshing more often means waiting less" — but the opposite happens: as the window
narrows, the slice you can take in one go shrinks, and an 80 ms job gets cut into sixteen pieces.
Every wait is short, but there are a lot of waits.

Lengthening the period gave the same result as burst: zero throttling, 80.0 ms. And this dial
works today — you set it with docker run --cpu-period --cpu-quota, it is part of the
container configuration so it survives a restart, and it needs no new tooling. The cost is
symmetric: once you genuinely exhaust the quota, a single stall can now last up to 500 ms, and
the group holds the machine for longer while it does. The documented ceiling is 1 second.

So a solution that needs no burst at all was sitting there, and for two weeks I never looked at
the second number in cpu.max.

Where the chain breaks

As for burst: the reason it is off in all 16 of my cgroups is that nothing in my stack turns it
on. I checked the chain link by link:

Kernel: present since v5.14. ✓
OCI runtime-spec: a burst field is defined, with the constraint that it must not exceed a positive quota. ✓
runc: specconv maps r.CPU.Burst to c.Resources.CpuBurst; the actual write happens in fs2/cpu.go inside opencontainers/cgroups, to the cpu.max.burst file. ✓
Podman: settable via --cgroup-conf=cpu.max.burst=50000 — documented behavior, writing to an arbitrary cgroup v2 file as part of the container configuration. (I am taking this one from the docs; I did not measure it, as podman is not installed on my machine.) ✓
Docker CLI: absent. On my 27.4.0 install, the word "burst" appears zero times in docker run --help; the current docker run reference lists --cpu-period, --cpu-quota and --cpu-shares, and lists neither burst nor any --cgroup-conf-style passthrough. ✗
Compose spec: zero hits in deploy.md. ✗
systemd: systemd.resource-control has CPUQuota and CPUQuotaPeriodSec, with no burst equivalent (VPS3 runs systemd 255). ✗
Kubernetes: no burst field in the Pod spec. On the CRI side there is a generic cgroup v2 passthrough map called LinuxContainerResources.unified — so the pipe is not technically closed, there is just no user-facing surface for the kubelet to fill it from. ✗

The chain is not broken end to end; it breaks at the Docker CLI and the kubelet's Pod
interface. If you run podman, the switch is within reach today.

"I will just write it by hand on Docker," I said, and tried. I started a container with
docker run --cpus=0.5, wrote 50000 into its cgroup, verified it, then ran docker restart:

docker --cpus=0.5  -> cpu.max=[50000 100000]  burst=[0]
written by hand    -> burst=[50000]
after restart      -> cpu.max=[50000 100000]  burst=[0]   (same container id)

Docker rewrites cpu.max because that is its configuration; it does not rewrite burst because
it knows no such thing exists. Any value you write by hand evaporates on the first restart. This
is not configuration, it is a poke.

What to ask about your own setup

First ask why the quota is there at all. My etcd instances are squeezed into half a core on an 18-core machine. The quota was put there to stop them starving neighboring projects, but etcd is not the profile that starves anyone. The right answer here may be to drop the quota and give them a share of the contention via cpu.weight — but that is not free: a weight only divides the spoils during contention, it sets no ceiling. A runaway compaction can still take all 18 cores. A quota is a hard wall; a weight is a negotiation.
Consider the period; you have probably never touched it. The measurement above shows that lengthening the period at the same ratio achieves what burst does, and it works in Docker today. Shortening it backfires. Limits: the documented maximum period is 1 second, and a longer period means a longer single stall.
Raising the quota is boring, and it works. The difference from burst is this: doubling the quota also doubles your entitlement on average, whereas burst only gives back entitlement you previously did not spend.
Burst is not free — the docs say so outright. The burst section of the kernel documentation introduces the mechanism and its price in the same sentence: borrowing against a future underrun happens "at the cost of increased interference against the other system users." The consolation is that the tardiness is bounded, and that per the docs the interference stays limited when there are many cgroups or the CPU is under-utilized. Still, do not read it as "protects the neighbors": it protects the average and raises the instantaneous interference.
If you do choose burst, measure peak demand rather than guessing it — and work out up front how to make it stick. If it does not come through Docker, you need something that reapplies it on every restart.
At minimum, put the counter on a dashboard. Even if you do none of the above, reading nr_throttled and throttled_usec is free — or cpu.stat.local if the quota lives on a parent unit. An average CPU graph structurally cannot show you this event.

The last one is the cheapest and the most useful. A pause you do not measure has not been
cancelled, only made invisible. And if you have an appetite for changing the scheduler more
fundamentally, the sched_ext side
is worth a look — though the answer to the quota problem is not there.

What I did not prove

I need to draw a line here, because the story looks too clean and for a while I believed it too
much myself.

In the lab there is causation: same work, same quota, burst the only variable, result 250 throttle
events down to zero. But the lab workload is single-threaded. The docs state that burst is not
transferred between cores; in a multi-threaded group like etcd, per-CPU slice distribution enters
the picture and the behavior may not be identical. What I proved is the single-threaded form of
the mechanism.

In production, all I have is a strong coincidence. I know etcd is slow, I know it is being
throttled, I know the requests that slow down are things like key:"health" with no CPU cost at
all, and I know storage is not the culprit (against 891 and 12,395 apply warnings in 24 hours,
slow fdatasync appears only 1 and 3 times). I wanted to test it with a measurement: I took eight
one-minute readings on licman-etcd and lined up throttle counts against slow-request counts
across the seven deltas.

The result did not support me: Pearson correlation 0.437, n=7. That does not mean "no
relationship" — it is moderately positive but not significant at this sample size, so the
measurement neither supports nor refutes; it is underpowered. The reason is visible inside it:
against an average of 61.2 throttle events per minute there are only 11.3 slow-request warnings.
Most throttles never turn into a warning on the request path — they land on background threads,
and one slow request can span several throttles besides. At one-minute resolution the signal
drowns in noise, and the way to fix that is to enable burst in production and run an A/B, which I
am not going to do on a live consensus ledger.

So what I hold is this: exposure measured, mechanism proven in its single-threaded form,
production causation unproven. Keeping those three apart matters as much as the measurement itself.

What is certain, though, is that the question I had been asking since the day I set that quota was
the wrong one. I was asking "how much of its quota is this service using?" and for ten days I
proudly collected the answer "17%." What I should have asked was "how many times has this service
hit its quota?" The first question measures an average. The second measures what the user is
waiting for.

Official Sources

I Could Not Turn On Journal Sealing; Only the Debug Log Said Why

Mustafa ERBAY — Sun, 04 Oct 2026 12:46:48 +0000

Logs are the most trusted and least questioned files on a machine. After an incident you open them and believe what they say. systemd has an answer to that trust: Forward Secure Sealing. journald seals the file cryptographically at regular intervals, and if somebody tampers with it afterwards, journalctl --verify tells you.

The journald.conf documentation puts it this way for Seal=: "If enabled (the default), and a sealing key is available (as created by journalctl's --setup-keys command), Forward Secure Sealing (FSS) for all persistent journal files is enabled." So what ships enabled by default is the mechanism, not the key. The key is yours to create. Creating it is what I sat down to do, and the first command hit a wall at the first step:

$ journalctl --setup-keys --interval=1m
Generating seed...
Generating key pair...
Failed to generate key pair: Operation not supported

The error message names nothing. Finding the cause took half an hour; fixing it took one package. Then sealing actually engaged, and the real half of this article began: what sealing catches, what it does not, and what that PASS line really promises.

The lab

Every number below comes from two privileged containers with systemd genuinely running as PID 1:

$ journalctl --version | head -1
systemd 257 (257.13-1~deb13u1)
$ . /etc/os-release; echo "$PRETTY_NAME"
Debian GNU/Linux 13 (trixie)
$ uname -r
6.10.14-linuxkit

On the comparison side sits Debian 12: systemd 252 (252.39-1~deb12u2). One disclosure up front: that linuxkit in uname -r is the kernel of a Docker Desktop virtual machine on a macOS host. Both Debians share the same non-Debian kernel. What gets measured here is Debian userspace: journald, journalctl, packaging. No claim in this article rests on the kernel.

Sealing needs a persistent journal. Without /var/log/journal, journald keeps everything under /run/log/journal in memory and there is no file to seal. The directory exists on both.

What "Operation not supported" does not say

The setting is right where it should be:

$ systemd-analyze cat-config systemd/journald.conf | grep -i seal
#Seal=yes

The man page still documents --setup-keys. The file header shows no sign of a gap either, only the sealing flag is missing:

$ journalctl --header | grep -E "File path|Compatible flags"
File path: /var/log/journal/a452a00b.../system.journal
Compatible flags: TAIL_ENTRY_BOOT_ID

Raising the log level gave up the answer:

$ SYSTEMD_LOG_LEVEL=debug journalctl --setup-keys --force 2>&1 | head -3
libgcrypt.so.20 is not installed: libgcrypt.so.20: cannot open shared object
file: No such file or directory
Failed to generate key pair: Operation not supported

That line is printed at LOG_DEBUG, so it never shows up in a normal run. Sealing depends on libgcrypt, and since systemd v256 the library is no longer an ordinary shared-library dependency; it is loaded at runtime through dlopen(). The upstream NEWS file announced the change along with its warning: those libraries might no longer be pulled in automatically when ELF dependencies are resolved. On the Debian side the result looks like this:

$ apt-cache show systemd | grep -iE "^(Depends|Recommends|Suggests)" \
    | tr ',' '\n' | grep -ci gcrypt
0

The systemd package does not ask for libgcrypt20, neither as a dependency nor as a recommendation. The fix is one line and it works instantly:

$ apt-get install -y libgcrypt20
$ journalctl --setup-keys --force --interval=1m >/dev/null; echo "rc=$?"
rc=0

The scope of the trap deserves stating. On a desktop or a full server install, libgcrypt20 is usually already there because something else pulled it in; apt-cache rdepends libgcrypt20 lists libxslt1.1, the webkit libraries, strongSwan plugins. This silent loss mostly hits minimal images, containers and thin servers built with debootstrap. So the answer to "can sealing be turned on here" belongs to the machine, not to the distribution.

Once sealing really engages

With the key pair in place, a /var/log/journal/<machine-id>/fss file appears: the sealing key, 482 bytes, owned by root:systemd-journal. Watch the mode, because on two separate installs it came out as 0640. That means accounts in the systemd-journal group can read the sealing key. The verification key is printed to the screen exactly once; keeping it off the machine is your job, because you will not see it a second time.

Second point: journald cannot add a sealing flag to the header of a file it already has open. Generating the key and restarting the service was not enough, a new file was needed:

$ journalctl --rotate
$ journalctl --header | grep "Compatible flags"
Compatible flags: SEALED SEALED_CONTINUOUS TAIL_ENTRY_BOOT_ID

Now verification says something meaningful:

$ journalctl --file=.../system.journal --verify --verify-key=$VK
PASS: /var/log/journal/a452a00b.../system.journal
=> Validated from Sun 2026-10-04 11:51:50 UTC to Sun 2026-10-04 11:53:00 UTC,
   final 2.758424s entries not sealed.

That last line is my favourite output in this article. The tool announces its own blind spot: the final 2.76 seconds of entries are unsealed. Sealing reaches as far as the next seal, and the window in between stays open. The man page says the same thing while explaining --interval=: a shorter interval raises CPU consumption and shortens the time range of undetectable alterations. The default is 15 minutes. In the lab it was pulled down to one minute, which is why the window here is a matter of seconds.

Verification without a key, on the other hand, did not behave the way I assumed. Expecting at least a structural check on a sealed file, this came back:

$ journalctl --file=.../system.journal --verify
Journal file ... has sealing enabled but verification key has not been passed
using --verify-key=.
FAIL: ... (Required key not available)

The source is unambiguous about it: on a sealed file with no key, journal_file_verify() returns -ENOKEY before it even opens a temporary directory. Not a single structural check runs. The operational translation is just as blunt: lose the verification key and you cannot form even a structural opinion about your journal's integrity.

How wide does the tail get

For five minutes nothing was written to the machine by hand. The container's own services kept writing, which matters, because a seal only lands while an entry is being appended. The result:

=> Validated from Sun 2026-10-04 12:18:58 UTC to Sun 2026-10-04 12:23:00 UTC,
   final 59.603160s entries not sealed.

Six seals had accumulated in the file, at epochs 0, 0, 1, 2, 3, 4. With a one-minute interval the tail reaches a full minute. At the 15-minute default that same window is a quarter of an hour.

The source holds a more uncomfortable detail here. In v257 there are only two places that write a seal: journal_file_append_first_tag() when the file is created, and journal_file_maybe_append_tag() called from the entry-append path. No entry, no seal. On a quiet machine the tail is therefore not bounded by the interval at all; it stays unsealed indefinitely.

My own assumption had been that running journalctl --rotate after a critical event would seal the tail. In the v257 source there is no path that appends a seal while closing a file, and no run of mine showed rotation sealing the old file's tail either. Do not lean on it. What you can lean on is that --verify prints the window on every run.

What sealing looks like inside the file

Seeing where the seal lands is the shortest path to understanding its reach. The file-format document gives the TAG object's layout: an object header, a sequence number, an epoch number and a 32-byte SHA-256 HMAC, 64 bytes in total. I walked my own file raw, stepping through the objects from start to end and counting their types:

object types: ENTRY 73, DATA 258, FIELD 40, ENTRY_ARRAY 90, TAG 3
TAG (seqnum, epoch): (1, 0)  (2, 0)  (3, 1)

Three seals in a file spanning roughly 70 seconds, with the interval set to one minute. The first seal lands as the file is created: journal_file_append_first_tag(), called from journal_file_open() for newly created files only, folding the header plus the two hash-table objects into the HMAC. The second sits in the same epoch. Hold on to that detail, I will come back to it.

Each seal is computed over the objects that arrived since the previous one. The document adds an important footnote: while computing the HMAC, volatile header fields of objects are skipped, such as linked-list pointers added later. What validates the consistency of those fields is not the seal but the structural checks in --verify.

I did not take the cost side, but it can be derived from the structure that was measured: a 64-byte TAG at the default 15-minute interval means 96 seals a day, roughly 6 KB. Pull it down to one minute and you get 1,440 seals, about 92 KB. The disk side is irrelevant. The cost the man page warns about is on the CPU, because the key is derived forward at every seal. That figure is not one I took, so I will not quote one; if you plan to shorten the interval, run it on your own hardware and look.

What `--verify` checks independently of sealing

Sealing is half the story. The journal_file_verify() source holds a long checklist that runs even without a key: hashes of data and field objects, object sizes and alignment, consistency of compression flags, strictly increasing entry sequence numbers, timestamps ordered monotonically within the same boot, hash-table chains free of cycles, entry-array chains that never jump backwards, and header counters that reconcile with the actual content.

That list catches corruption well. What it does not catch is a deliberate edit that leaves all of those fields consistent. The hash on data objects is keyed, and my file's header carries the KEYED-HASH flag, but the key sits inside the file itself. It is protection against hash flooding, not authentication. Whoever holds the file can recompute the hash. Authentication needs a secret the attacker does not have, and sealing is exactly what brings one.

There is one more wrinkle: a sealed file will not even open on a machine without sealing support. Copying the same sealed file to a Debian 13 host with no libgcrypt20 and trying to verify it:

$ journalctl --file=/root/t/muhurlu.journal --verify
Failed to open files: Operation not supported

Verification does not fail; the file cannot be read at all. If you plan to open sealed archives on a thin recovery box later, libgcrypt20 has to be there too.

What sealing catches

Three kinds of tampering, each on a copy of the file, at single-byte or single-bit granularity.

A message inside the sealed region. I changed one character inside a log line:

392be0: Invalid object contents: Bad message
File corruption detected at .../t13-mesaj.journal:3746784
  (of 8388608 bytes, 44%).
FAIL: ... (Bad message)

Caught, but sealing contributed nothing here. Every data object carries its own hash and changing the text breaks it. The same edit would be caught in an unsealed file. A corruption check and authentication are not the same thing, and the first one is what this test demonstrates.

The seal itself. To isolate sealing, I flipped a single bit in the HMAC field of a TAG object:

3955d8: Tag failed verification
FAIL: ... (Bad message)

This is the proof that sealing works. According to the file-format document, every TAG carries a SHA-256 HMAC over the objects before it, with the key derived forward at each epoch.

Dropping the flag. Can an attacker pretend the file was never sealed? I zeroed the compatibility flags in the header:

38f980: Tag object in file without sealing
FAIL: ... (Bad message)

No. With TAG objects still in the file, clearing the flag does not silence verification, it makes more noise.

Then I deleted the logs

A real attacker does not flip bytes, they delete. The sealing key lives on the very machine it protects, so anybody with root can read it. I saved the harshest test for last: stop journald, delete every .journal file, leave the fss key alone, start the service, write a clean history.

$ journalctl --no-pager -q | wc -l
1384
$ systemctl stop systemd-journald
$ rm -f /var/log/journal/*/*.journal
$ systemctl start systemd-journald
$ for i in $(seq 1 6); do logger -t audit "FORGE-$i"; done
$ journalctl --no-pager -q | wc -l
11

One thousand three hundred and eighty-four lines gone, eleven left. The new file opened with SEALED SEALED_CONTINUOUS flags, so on paper I have a sealed history. Here is what verification said:

38f980: Tag failed verification
FAIL: ... (Bad message)

The wipe left a mark. The same result came back three times across three separate containers, and there is a control run to go with it: without any deletion, a new file produced by a plain rotation passes the same verification. So this is not a blanket false alarm.

On Debian 12 the same move gives a different answer:

PASS: /var/log/journal/41a11bc4.../system.journal
=> No sealing yet, 2.020730s of entries not sealed.

There, 1,543 lines dropped to 19 and verification was not bothered in the slightest.

This is where I owe you honesty. I did not trace the Debian 13 FAIL down to which byte broke and why, so I am not pinning the mechanism on a patch or a flag. What I hold is a behaviour reproduced three times and a control run that does not refute it. I claim the behaviour, not the mechanism.

The patch that seals empty epochs

Debian 13's files carry an extra flag, SEALED_CONTINUOUS, that Debian 12's do not. The flag arrived with Felix Dörre's patch, merged on 8 November 2023 in the v255 cycle, and the patch states what it closes: "Currently empty epochs are not sealed. This allows an attacker to truncate a sealed log and continue it without any problems showing when verifying the log."

The same gap is on record as CVE-2023-31438. The NVD description: a sealed log file can be truncated and log sealing resumed such that the integrity check shows no error, despite modifications. The record also notes the vendor replying that none of the findings was a security vulnerability, and the entry is tagged disputed.

One sentence in the patch text sits where most summaries skip: "This partially addresses CVE-2023-31438." It also says what closing it completely would take: verifying that there is exactly one seal per epoch, and not sealing before the epoch has ended. The author left the premature sealing in place, having found it deliberate but not understood its purpose.

That open door is sitting in my own measurement. Look at the TAG dump above: (1, 0) and (2, 0), two seals in epoch 0. The premature sealing the patch says it did not remove is exactly this. The verifier's continuity check accepts it too; in the source, a file's first tag and a second tag in the same epoch are explicitly exempt.

The flag has one more practical side effect. Hand a sealed file produced by Debian 12 to Debian 13's journalctl and you get this:

000000: This log file was sealed with an old journald version where the
sequence of seals might not be continuous. We cannot guarantee completeness.
PASS: /root/t/bw.journal
=> Validated from Sun 2026-10-04 11:45:41 UTC to Sun 2026-10-04 11:48:00 UTC,
   final 4.528312s entries not sealed.

A PASS, but not a complete one. If you keep archives produced by older releases, make a habit of looking for that line in your verification report.

Delete the key too, and verification goes quiet

So what if the attacker deletes the key file as well? Same scenario, one extra step:

$ systemctl stop systemd-journald
$ rm -f /var/log/journal/*/*.journal /var/log/journal/*/fss
$ systemctl start systemd-journald
$ journalctl --header | grep "Compatible flags"
Compatible flags: TAIL_ENTRY_BOOT_ID
$ journalctl --verify --verify-key=$VK
PASS: /var/log/journal/a452a00b.../system.journal

PASS, even with the verification key handed over, and not a word of complaint. The logic is consistent: the file is not sealed, and an unsealed file has no seal to verify. From a security standpoint the consequence is this: journalctl --verify never tells you "this file should have been sealed". You have to hold that expectation somewhere the machine cannot reach.

The bottom-left corner of that diagram is the real lesson here. PASS has two separate meanings and the output says both with the same word.

How v262 changes this picture

systemd v262 shipped on 22 September 2026 and sealing moved from libgcrypt to OpenSSL. Debian 13's systemd package already carries a libssl3t64 dependency, so the missing-package trap in this article does not change shape when 262 lands; it disappears.

What replaces it is sneakier. In the new source, verifying a sealed file on a machine without sealing support no longer returns -ENOKEY:

if (journal_auth_supported())
        return -ENOKEY;
else
        log_notice("Journal file is sealed, but journal sealing support is "
                   "disabled. Skipping seal verification.");

So the situation that produces a hard FAIL on 257 will, from 262 onward, leave a notice line and carry on toward PASS. It is another variant of the quiet PASS this whole article is about, this time inside upstream itself. When your distribution moves to 262, that notice line is what your verification reports will need you to look for.

What I do in practice

Check the flag, do not trust the default. If journalctl --header | grep "Compatible flags" shows no SEALED, there is no sealing. On minimal images this is the first place to look.
Do not leave the check to a human. Advice is not a control. A small timer that greps journalctl --header for SEALED and alerts when it is absent is the "hold the expectation off the machine" argument turned into something operational.
Put libgcrypt20 on the install list. When it is missing the error message hides the cause, and the diagnostic path is SYSTEMD_LOG_LEVEL=debug. Recovery boxes that will open sealed archives need it too.
Get the verification key off the machine. A key kept on the same disk is a convenience for whoever takes that disk, not evidence. It is printed once.
Know the price of rotating keys. --setup-keys --force mints a new pair. The verification key belongs to a key pair, not to a machine, so the moment you rotate, your existing sealed files can no longer be verified with the key in your hand. Do not rotate without archiving the old one.
Remember that early boot sits outside the seal. Only persistent files are sealed. Everything journald writes under /run/log/journal before systemd-journal-flush.service is unsealed by design, so the first seconds of boot are structurally out of scope.
Pick the interval deliberately. The default is 15 minutes, and so is the length of the undetectable window. The "entries not sealed" line in --verify reports it on every run.
Do not confuse sealing with staying put. Sealing shows that existing bytes have not changed, not that the records still exist. Detecting a missing message is a different problem: signed syslog (RFC 5848) numbers messages so the receiver can see which ones are absent. A copy shipped to a central collector does the same job, and the setup in central logging with systemd-journal-remote exists for exactly that. What gets deleted locally survives remotely.
Remember what rotating logs costs. How copytruncate silently eats records when it truncates a file in place is something I took the measurements for earlier; in a sealed world such truncations cost you not only data but evidence.
Detection and repair are separate things. The distinction from the dm-integrity article holds here too: sealing tells you about damage, it does not undo it. What undoes it is a backup or a remote copy.

Conclusion

Sealing works wherever you can turn it on. What tripped me up was not its quality but the silent assumption that it existed. The Seal=yes line sits in the documentation as a default, anyone skipping the condition next to it can believe their sealing is on, and nothing announces that it is not: no service log, no startup error, and certainly not the PASS from --verify.

Journal integrity is not a setting but a chain. The library has to be installed, the key generated, the file rotated, the verification key kept outside, and somebody has to hold the knowledge that "this file should have been sealed" off the machine. Break any link in that chain and journalctl --verify still prints PASS. A quiet PASS is no different from an alarm that stays silent.

Official Sources

I Trust 158 Root Certificates. I Installed Three of Them

Mustafa ERBAY — Sun, 04 Oct 2026 06:06:09 +0000

If someone asked me "how many certificate authorities do you trust?", my honest answer for years would have been "no idea, whatever Apple says." I see the padlock in the browser, I move on. I assume somebody back there has thought about it.

This morning I sat down and counted that assumption. Trust isn't a feeling, after all; it's a list sitting on my machine. And despite two decades in this line of work, I had never once opened that list.

What surprised me when I did wasn't how long it was. It was that the number turned out not to be a single number.

I asked the same question in four places and got four answers

The macOS root store is a keychain file on disk. Counting what's inside is one command:

security find-certificate -a -p \
  /System/Library/Keychains/SystemRootCertificates.keychain \
  | grep -c 'BEGIN CERTIFICATE'
# 158

A nice, clean number. Then I asked the trust settings the same question:

security dump-trust-settings -s 2>/dev/null | head -1
# Number of trusted certs = 157

One short. A certificate that lives in the store but doesn't make it into the "trusted" count. I put the third question to the names: deduplicating the labels of those 158 certificates left me with 155. And the fourth question was this — how many of the 158 did I put there? Answer: none. The ones I added aren't even in this file; they live in entirely separate places, and there are three of them.

So: same question, and the answers are 158, 157, 155 and 3. None of them is wrong. Each measures something different. Ever since the day I asked five different tools how much free disk space I had and found a 13.5 GiB spread, this no longer surprises me — but when the subject is trust, it does make you wince a little.

That extra certificate in the store isn't a bug

Tracking down the gap between 158 and 157 took some digging, but I liked the answer. I opened all 158 certificates in the store and compared the subject field against the issuer field. In 157 of them the two matched — self-signed. In exactly one, they didn't:

subject= CN=Developer ID Certification Authority, O=Apple Inc., C=US
issuer = CN=Apple Root CA, O=Apple Inc., C=US
notAfter= Feb  1 22:12:15 2027 GMT

This is Apple's developer signing intermediate. It lives in the root store but it isn't a root; it has a parent, and that parent — Apple Root CA — sits in the same store. Its name never appears in the security dump-trust-settings -s output, because that command lists anchors, not residents of the store.

There's a subtle but important distinction here. RFC 5914 defines a trust anchor as "an authoritative entity represented by a public key and associated data"; the anchor is an input to path validation, not a link in the chain. An intermediate's trustworthiness doesn't come from itself, it comes from Apple Root CA. Sure enough, validation passes cleanly:

security verify-cert -c devid.pem -p basic
# ...certificate verification successful.

That's why 157 is the more honest number: it counts the points this machine accepts without question. The intermediate is parked there for convenience.

The naming side has a similar trap. There are four separate certificates whose label is exactly GlobalSign; the distinguishing information isn't in the CN but tucked away in the OU field (Root CA - R3, Root CA - R6, ECC Root CA - R5, ECC Root CA - R4). And if you search for certificates with "GlobalSign" anywhere in the CN, you get nine. Four or nine? Both are correct; one is an exact match, the other a substring search. All 158 certificates have distinct SHA-256 fingerprints — what repeats isn't the certificate, it's the name shown to the human.

While comparing the two lists, by the way, I walked straight into a trap of my own making. Subtracting the trust list from the store list showed two "missing" names, and for a moment I was genuinely excited. The second one was real (the intermediate above); the first was entirely my own fault. NetLock Arany, a Hungarian root, has non-ASCII characters in its name, and the two commands print the same certificate two different ways — one as a hexadecimal blob, the other as proper UTF-8. Same certificate, two spellings, one phantom finding. When you're measuring, the most expensive mistakes always surface where the tools disagree with each other.

Counting certificates is not counting trust

After finishing with the system store I looked at the other keychains on the machine, and that's where I got the actual lesson.

/Library/Keychains/System.keychain holds six certificates. Four are familiar: Apple's system identity, its Kerberos KDC, the developer relations certificate, and the AdGuard root we'll get to below. The fifth is a customer's internal root. The sixth one's name tells me nothing at all:

subject= CN=Self-Signed-7C9F7AE3
notBefore= Dec  2 08:06:00 2020 GMT
notAfter = Nov 30 08:06:00 2030 GMT
X509v3 Basic Constraints:
    CA:FALSE

December 2020. Nearly six years ago. It may predate this machine; it may have been carried over in a migration. I don't know what it is, and writing that I don't know beats inventing a guess.

But here's the part that matters: CA:FALSE. This isn't a root, it's a leaf certificate. Its name doesn't appear in any of the three trust domains — I checked all three, zero hits. So it sits on my machine and nothing trusts it. A harmless leftover.

In my own user keychain things get more instructive still. There are 20 certificates there, and 12 of them are marked CA:TRUE — structurally capable of signing certificates. But when I look at the trust settings, the user domain contains only two roots.

I have twelve CA certificates and I trust two of them. The other ten are merely being stored; somebody sent me an organization's root, I dropped it into the keychain, and never trusted it. That's the correct behaviour — a pleasant surprise, even.

What follows from this is simple, and I had been assuming the opposite: you cannot measure trust by counting certificates. The keychain is a drawer; the trust settings are a separate ledger. Something being in the drawer doesn't mean it's written in the ledger. The sentence "I have 184 certificates on my machine" tells you, on its own, nothing at all.

Of the 157 roots, zero carry a decision of mine

Apple documents that it splits this list into three categories: trusted roots, ones that always ask, and blocked ones. Its wording for the blocked category is unambiguous — certificates believed to be compromised, which will never be trusted.

Curious, I went looking for these categories on my own machine. Across all three trust domains (system, admin, user) I couldn't find a single record returning Deny or an always-ask result. What I found instead was more interesting: every one of the 157 anchors in the system domain looks like this.

Cert 5: ISRG Root X1
   Number of trust settings : 0

Zero. No explicit setting — implicit, blanket trust. Not one of the 157 roots carries a decision recorded by me or by any administrator of this machine. That isn't a bad thing in itself; Apple's store is managed through audited, public processes and can be updated remotely. But the subject of the sentence "I trust these" isn't me. I trust a list I inherited.

So are there any records carrying an explicit decision? Three: two in the user domain, one in the admin domain. All three are mine.

But the genuinely uncomfortable part surfaced right here. The AdGuard root, the protagonist of this post, also has an explicit setting count of zero. I carry it exactly the way I carry Apple's 157 — without being asked anything. A root I added with my own hands sits in precisely the same silence as the list I inherited.

The three roots I added myself

Dumping the admin and user domains turned up three distinct roots. I installed all three at some point, and I had forgotten all three.

The first is my own work: kopru-root-ca, the root of the köprü platform I use to manage customer servers. Self-signed, CA:TRUE, pathlen:0, generated on 9 July 2026, valid until 6 July 2036. I put this there deliberately, I know what it does, and I know where the key behind it lives. No complaints.

The second is a customer's internal root CA. I'm not naming it here — the name of an organization's internal infrastructure doesn't need to appear in my blog post. What makes it interesting is this: it's registered in both the admin and the user domain. Same root, two places. I don't remember adding it twice. Most likely it went in once by hand and once with a tool's help during some setup.

The third one actually stopped me: Adguard Personal CA.

I apparently promised an ad blocker until 2045

I installed AdGuard to cut out ads. A reasonable request. During setup it told me I needed to install a certificate, and I said "sure, it has to filter HTTPS" and approved it. I haven't thought about it once since closing that dialog.

Today I looked at that certificate:

subject= C=EN, O=AdGuard, CN=Adguard Personal CA
notBefore= Jan 19 22:27:46 2025 GMT
notAfter = Jan 14 22:27:46 2045 GMT
X509v3 Basic Constraints: critical
    CA:TRUE

Twenty years. 2045. The first thing that struck me reading that line was that this laptop will long since have been scrapped by then. The certificate is written to outlive the computer it sits on. I added this root thinking "for now"; the document says "for life."

And this root is not dormant. AdGuard is running right now, with three processes open including a system extension:

/Applications/Adguard.app/Contents/MacOS/Adguard
com.adguard.mac.adguard.network-extension.systemextension
com.adguard.mac.adguard.helper

A trusted root flagged CA:TRUE can sign a certificate for any domain. Including my bank's. I'm not claiming AdGuard is malicious — I have no evidence of that. What I'm saying is structural: if that key is ever leaked or abused, it carries the same authority on my machine as DigiCert. The difference is that DigiCert is subject to public audits, transparency logs and a revocation mechanism. The only person auditing AdGuard's root is me, and I didn't check it once in twenty months.

CISA's warning on this isn't new — it dates from 2017 and sits in their archive — but the mechanism hasn't changed. The gist is that products performing HTTPS inspection have to install a trusted certificate on the device in order to avoid showing the client warnings, and that many of these products don't properly verify the server's certificate chain. In other words, the layer in the middle may be telling you "this connection is secure" without looking as carefully as you would. From that point on, the padlock describes the intermediary, not the far end.

This is not a call to delete AdGuard. Filtering ads and trackers is a real benefit and I wanted that benefit. But making the trade knowingly isn't the same as approving it on a setup screen and forgetting. I had done the second.

And then there are the expiry dates

Once you start counting it's hard to stop. I pulled the end dates for all 158 certificates: none have expired as of today, and the furthest runs until 9 December 2054. But the nearest is 27 November 2026 — less than eight weeks away. Three roots will lapse before 2028; the fourth certificate expiring in that window is the Apple intermediate we already met, in February 2027.

Those dates aren't my problem; Apple handles them with updates. For the three roots I added myself, nobody is standing behind me. I'm the one who has to track what happens to those keys in 2036 and 2045, and I have hard data on my performance so far: twenty months, zero checks.

If you want to read your own list

These commands are all read-only; none of them changes anything:

# How many certificates are in the store
security find-certificate -a -p \
  /System/Library/Keychains/SystemRootCertificates.keychain \
  | grep -c 'BEGIN CERTIFICATE'

# The anchors the system trusts without question
security dump-trust-settings -s

# THE TWO COMMANDS THAT ACTUALLY MATTER: what was added by hand
security dump-trust-settings -d   # admin domain
security dump-trust-settings      # user domain

The last two are the important ones. The first two show you Apple's work; the last two show you yours. For every name that comes up you should have an answer to three questions: why did I add this, is it still needed, and when does it expire? If you can't answer all three, that root hasn't earned its place on the list.

The distinction that helped me decide what to do with those names was this. If there's somebody other than you behind a root — an audited public CA, your organization's IT team, a process that lands in transparency logs — then there's a structure carrying the risk, and your job is just to know the list. But if you installed the root, you are its only caretaker, and that has three concrete consequences: you have to know where the key lives, you have to put its expiry in your calendar, and you have to delete the root the day you stop using the tool. The third is the most commonly skipped. Removing a tool often doesn't revoke the trust record; the application leaves, the root stays.

The second distinction concerns intermediary layers. Ad blockers, corporate filters, debugging proxies — they all use the same mechanism and they all raise the same question: is this layer actually validating the server's certificate on my behalf? That was the sharp edge of CISA's warning. If you don't know the answer, you need to walk back some of the meaning you've attached to that padlock.

For me, this round came out like this: the köprü root stays, its rationale is clear. I'll deduplicate the customer root's twin records across the two domains. AdGuard's root stays too — but now knowingly; its expiry is going into my calendar.

Trust is the sum of decisions you don't remember

What I noticed while writing this is that the number 157 never bothered me. Those aren't my decisions; they're a list Apple audits, can revoke, and updates. There's risk, but there's an owner.

The three bother me. Because I'm their owner and I was absent. Everyone who likes tinkering with infrastructure has a layer of sediment like this on their machine: a certificate, a proxy, a port opened for a test. Taking root of trust in server infrastructure seriously while not opening the root store on my own laptop for twenty months is a strange inconsistency. I'd made the same kind of slip while writing that identity doesn't come from network location: there I was insisting that "it came from Cloudflare" doesn't mean "it came for me," while here I handed out twenty years of authority because a setup screen asked for it.

Most security writing tells you to add something new. This morning's lesson was the reverse: reading the list once taught me more than installing another tool would have. Four commands, one morning, three forgotten roots.

Go and look at yours. There's probably somebody on your list too, and you probably invited them.

Official Sources

Forty-Four Posts in a Row From One Category, a Reader Caught It

Mustafa ERBAY — Sat, 03 Oct 2026 14:28:05 +0000

This blog's publishing pipeline wakes up three times a day: 08:30, 13:30, 16:30.
Each time it picks a topic, researches it, writes it in Turkish and English,
renders a cover, pushes it through the gates and puts it live. Six months of
that. Where it keeps breaking is something I have already
laid out in numbers.

In the middle of September the pipeline did not miss a single day. Pace on
target, gates green, alarms quiet. And over that same stretch, the 44 posts it
produced all sat in the same category.

Once the count was in, the number was not the uncomfortable part. The
uncomfortable part was that nothing had broken. Had it broken, it would have
told me. It did not break; it just kept writing into the same shelf.

The queue ran out on 3 September

The pipeline has a topic queue: scripts/content-calendar.json. There are 514
entries in it, 500 of them published. Each entry carries a source note —
recording where a topic came from is a habit that started embarrassingly late,
but from 3 September onward the notes are explicit:

"source": "nobetci-uretimi (kuyruk tukendi, taze konu)"

A second record agrees on the date. When the old generation workflow was retired
on 9 September, the note left at the top of the file reads: "Bu cron ise 3 Eyl'den
beri SIFIR makale uretti: takvim kuyrugu tukendi" — this cron has produced zero
articles since 3 September, the calendar queue is empty. Two separate places,
the same day.

So no queued topic was left and the pipeline started finding its own. That part
was no secret to me; I simply did not read it as a problem. I did not want
production to stop when the queue emptied, and the fallback behaviour looked
sound: find a mechanism the blog has never covered, verify it against the primary
source, measure it in a lab, write it up.

The behaviour was sound; the outcome was not.

The fingerprint of a reflex

The window from 3 to 22 September — the day the queue emptied through the day the
warning arrived — is 20 days. Counting from the frontmatter gives 65 posts:

Category	Posts	Share
technology	59	90.8%
tutorials	5	7.7%
life	1	1.5%
career	0	0%

On 15 of those 20 days every single post was technology. The longest unbroken
run is 8–21 September: 14 days, 44 posts, one category, no interruption. The
last career post before the gap went out on 1 September and the next one on 23
September — 22 days apart. For life the gap is 20 days, for tutorials 17.
technology, meanwhile, never went quiet in that window at all: it published on
all 20 of the 20 days.

A table does not quite convey a run like that; a list does. The nine posts
published between 13 and 15 September, in order: HSTS preload, Authenticated
Origin Pulls, fs.protected_regular, kTLS, tcp_ecn, TCP keepalive, dm-verity,
KSM, ssh -J. Nine distinct mechanisms, nine distinct measurements — and all
nine on the same shelf. For three days, a reader of this blog was offered exactly
one kind of writing: a lab note that flips a kernel knob and shows the result
differing from expectation. A good kind on its own. Three days running, it stops
being a genre and becomes a tic.

The tags point the same way. Of those 65 posts, 47 (72.3%) carry the linux tag
and 28 carry kernel. Of those 59 technology posts, 43 carry the linux tag
and 27 carry kernel or sysctl. None of them were bad posts; they
were measured, sourced, tested in my own lab. They were simply all being filed
in one place.

A control group: what came out while the queue was full

A number like that is misleading on its own; it needs a comparison. The window
immediately before, 14 August to 2 September, while the queue was still full:
19 days, 59 posts.

Window	Posts/day	technology	tutorials	career	life	`linux` tag
14 Aug – 2 Sep (queue full)	3.11	39.0%	44.1%	10.2%	6.8%	3.4%
3 – 22 Sep (queue empty)	3.25	90.8%	7.7%	0%	1.5%	72.3%

Exactly one column refuses to move: posts per day. It went from 3.11 to 3.25.
Throughput held; composition collapsed.

While the queue was full, the variety was not a virtue of mine. The balance was
built months earlier, when the topics were written into the list by hand, and the
list carried it. The list ran out, the balance went with it, and what was left was the reflex.

The volume indicator never turned red

The pipeline has gates and they work: the source policy (at least three primary
sources across two distinct hosts), the duplicate scan, word-count and
reading-time consistency, MDX parsing, 93 policy tests, a publication-stalled
alarm, an intraday pace check. Some of them do look at the individual post: the duplicate scan asks "has this
topic run before?", the source policy asks "are the sources in order?" So the
post-scale version of "what did it publish?" is covered. What is never asked is
the scale above it: the ratio between posts. No gate puts two posts side by
side and asks whether they resemble each other.

One of them deserves a closer look, because it is the watchman that counts the
published output every hour: daily-pace-check. The comment at the top of the
file states its job in one line — it checks the maximum-four-posts-per-day rule
once an hour. The step itself does exactly that: it counts the commits on main
since the start of the day whose message begins with feat: yeni makale, and
errors out only if the count exceeds four. GitHub's documentation defines the
schedule event in a single sentence of its own: "The schedule event allows you
to trigger a workflow at a scheduled time." What fires it is not a judgement about
content but a clock — and the same page noting that the trigger can be delayed
under high load, with queued jobs sometimes dropped, is a reminder that it is a
timer rather than an opinion. For two weeks that clock kept finding a number that
did not exceed four — on 10 and 17 September exactly four posts went out, and
neither day broke the rule. It was entirely right to stay silent; that was the
question I had given it.

Google's SRE book names two questions for a monitoring system: "what's broken,
and why?" My indicators answered both correctly — nothing was broken. But the
SLO chapter of the SRE workbook has a sharper sentence: if a disruption is not
captured by any indicator, that is "a strong sign that your SLO lacks coverage".
The gap in my coverage had a name, and the name was composition.

The Prometheus alerting guide lands in the same place: alert on symptoms rather
than causes, and aim for "as few alerts as possible". Sound advice. But defining
the symptom is my job, and I had defined it as "publishing stopped". Publishing
did not stop. The blog narrowed to one subject — a symptom that definition
simply did not contain.

The warning came from a reader

The best evidence of the pipeline's blindness to composition sits in the
calendar's own record. The source note on the topic pool added on 23 September
begins like this:

"KONU HAVUZU (23 Eyl 2026, kullanici talebi: makaleler hep technology
cikiyordu, 7 Eyl'den beri kategori cesitliligi yok)"

Translated: topic pool, 23 September 2026, user request — the articles kept
coming out as technology, no category variety since 7 September. The note is
unambiguous about the source of the signal. What flagged the broken balance was
not an indicator, a test or a report; it was somebody reading the blog. From 7 to
23 September is 16 days. That is the detection latency: sixteen days and zero
automatic signals.

There is no soft way to phrase this. The person who has spent six months
counting and writing up every place this pipeline breaks failed to catch, by his
own measurement, the crudest deviation in its most visible output. And what the
side that did catch it said was not a subtle metric: the articles keep coming out
as technology.

One correction I owe here: "since 7 September" is the note's own phrasing, not my
measurement. One of the four posts published on 7 September is tutorials; the
unbroken run begins the next day. What the record supports is this: 15 days
counting from the start of the run, or 13 days counting from the day the detector
I am about to build would have fired. Either way, the side that produced the
signal does not change.

Back-testing the detector I did not have

"Which indicator would have seen this?" is a question I did not want to leave
hanging, because questions like that usually hang there and quietly die as good
intentions. So here is a simple rule: slide a seven-day window, and if it holds at
least seven posts, compute the category shares; if the dominant category crosses a
threshold, warn. Then apply the rule backwards to every day from 1 April to 3
October — a range covering 1,276 of the archive's 1,308 posts; the remaining 32
carry older dates.

At a 70% threshold: 22 warning days in six months, in three clusters — 4–7 August
(dominant tutorials, 19 of 27 posts), 26–27 August (12 of 17) and 9–24
September (16 days; at first trigger, 19 of 24 posts technology).

At 80% a single cluster survives: 10–24 September. No other day fires at all.
So a threshold exists that drives false alarms to zero, and that threshold would
have caught the deviation on 10 September. The reader spoke on 23 September.
Thirteen days apart, and 41 more posts went out in that gap.

The weakness of this test belongs in the article too: the threshold was found in
data where the deviation was already known. 80% fits this six-month archive well;
a week whose topic distribution narrows by design — a series going deep on one
product, a cluster of posts after an incident — would trip the same threshold
unfairly. So the right shape is not "warn and halt" but "warn and ask": a
question about whether the narrowing is a deliberate choice or a reflex. Still,
to claim the threshold sits roughly in the right place, there is six months of
real trigger history behind it, which beats having none.

There is also a more honest route for anyone picking a threshold without the
benefit of hindsight: instead of writing a fixed number, let your own history set
it. Apply the same sliding window to the past, derive the distribution of the
dominant category's share, and make an upper percentile — p95, say — the
threshold; your normal gets defined by you, rather than borrowing my 80%. Window
length follows the same logic: a window needs at least a dozen units in it, or a
two-item day reads as "monoculture" by chance. For a pipeline producing three
posts a day, seven days clears that bar; for somebody closing two tickets a week,
the same rule needs a thirty-day window or it never fires at all.

The count itself is about as hard as a shell loop. Category and date are already
in every post's frontmatter:

for f in src/content/blog/*/*.mdx; do
  case "$f" in *.en.mdx) continue;; esac
  awk -F': *' '/^publishDate:/{d=$2} /^category:/{gsub(/"/,"",$2); c=$2} END{print d, c}' "$f"
done | sort | awk '$1>="2026-09-03" && $1<="2026-09-22" {n[$2]++; t++}
                   END{for (k in n) printf "%-11s %3d  %%%.1f\n", k, n[k], 100*n[k]/t}'

technology   59  %90.8
tutorials     5  %7.7
life          1  %1.5

That a query this small went unrun for sixteen days is the most expensive detail
in this piece. Measuring was not hard; thinking of measuring was hard — because
every counter was green, and a green counter does not provoke questions.

Why the reflex kept going to the same place

This is the part that interests me most, because this part is mine, not the
pipeline's.

The topic-finding method goes like this: grep the archive, find a mechanism never
covered, verify it against the primary source, measure it. That method behaves
like a cost function — and kernel settings are by far its cheapest candidates.
You flip a sysctl in one session, measure, compare, and put the evidence on
screen. Cheap to verify, fresh evidence, low risk of being wrong.

Career and life pieces demand the same evidentiary bar at a far higher price.
The evidence for a kernel setting is on my own machine, now, in the output of one
command. The evidence for a career piece is scattered across six months of git
history, CI logs, the actual state of servers, or 1,300 published posts. It has
to be searched, extracted, counted, and the count verified twice — the windows,
the control group and the back-test in this article are exactly that expensive
part. The method was picking the most easily verifiable topic; I had been reading
that as the best topic.

A distinction is worth drawing here: the reflex did not produce bad work. I still
stand behind the technical content of those 44 posts; they were measured and
their sources held. What the reflex broke was not individual posts but the ratio
between them. And a ratio is a property no single post contains — it exists only
when you look from above.

The chapter on automation in the SRE book fits precisely here: automation is a
"force multiplier, not a panacea", and much of its value comes from consistency —
"very few of us will ever be as consistent as a machine". The book scopes that
consistency narrowly: the execution of well-scoped, known procedures. What my
pipeline was executing was not a procedure but a choice. The catch is that
consistency never asks whether the decision was right. The pipeline applied my
preference 44 times in a row with flawless fidelity. A human would have grown
tired, grown bored, switched subjects. The machine did not get bored.

I also do not think this failure is specific to automation. Looking at my own
weeks, the same curve shows up: what I take up to learn is usually dictated not
by curiosity but by how provable the thing is. I go where I can measure. Part of
what I call expertise is the sediment of a reflex like that.

Writing the rule into the system: compute, do not choose

Two things happened on 23 September. A new topic pool went into the calendar,
with category balance taken into account. And a step that runs before topic
selection was added to the production recipe: compute the target category from
the last-written date, take the one that has gone longest without a post, and
produce only there on that run.

The detail that matters is that the rule is code rather than a sentence. Writing
"mind the category variety" into the recipe would have achieved nothing, because
minding runs at the same moment as the reflex that finds the cheapest topic, and
it loses that race. The rule is now a step that computes the target category from
the calendar and imposes it. Not a statement of intent; a gate.

The price of that distinction has come due before: there was a line in the deploy
workflow I took for a guard, and months later it turned out to be
blocking nothing at all.
A rule being written down does not mean it is enforced. That post closed on
"being able to count the places the gates do not look"; the difference here is
that this time the counting actually happened instead of the sentence being
repeated.

Eleven days later — and the limits of this measurement

From 23 September to 3 October: 11 days, 34 posts.

Category	Posts	Share
technology	10	29.4%
tutorials	8	23.5%
career	8	23.5%
life	8	23.5%

This table does not count the post you are reading; with it the window holds 35
posts, career rises to 9 (25.7%) and posts per day to 3.18.

On none of those 11 days did a day stay in a single category; in the previous
window that ratio was 15 of 20. The average number of distinct categories per day
went from 1.25 to 2.82, and the share of posts tagged linux fell from 72.3% to
44.1%.

One more look at throughput, this time with the same denominator across all three
windows — per calendar day, counting days with no publication too: 2.95 · 3.25 ·
3.18. A number moving inside a ten-percent band. Composition collapsed and then
recovered while volume did not budge; the balance was not bought by cutting
output.

Now, the way this table should not be read: "the rule worked, here is the proof."
Two things changed on the same day; a topic pool built with category balance also
went in. The pool alone could have produced this table. A measurement separating
the two is not something I have, and not something I set up. Without a
discriminating test there is no telling which intervention is doing the work —
the same trap I fell into
two months earlier with a LinkedIn hypothesis. And 11 days is a short window next to 20.

The honest statement: composition recovered, and the cause cannot be assigned to
a single intervention.

And one more correction. "We will see when the pool runs dry again" is what I was
about to write; the calendar says it has run dry already. Of 514 entries, 500
are generated and the remaining 14 are rejected — zero topics waiting. The 23
September pool lasted eleven days. So the rule is already running on its own, and
the post you are reading is the product of exactly such a run: rotation said
"career", the career queue was empty, and the pipeline found the topic itself.
This time what it found was not a kernel setting but its own publishing record.

The real gain from this piece is not even the rotation rule
— it is that a query which measures composition now exists, and noticing a
fourteen-day run no longer requires waiting for a reader.

The reflex moved up a level

Running the same eye over the draft of this post produced a finding that stung,
because it was right. All nine career posts published since 23 September carry a
counted number in the title: 67 repairs, seven servers, two months, ninety
seconds, three scripts, twenty findings, seventy branches, sixty-eight backups —
and the forty-four posts you are reading about. The category field recovered; the
title shape settled into a single mould.

The rotation rule looks at the category field. It does not look at the shape of
the title, the structure of the piece, or the kind of evidence used. The reflex
did not disappear; it moved outside the field being measured. Put the constraint
anywhere and repetition accumulates one notch above it — and my new composition
query is blind to this by construction, because it too counts only categories.
Starting to measure one monoculture does not mean you have started counting the
dimensions you are not measuring.

This paragraph is not here as a gesture of modesty. If the thesis holds, its first
casualty should be the post itself: your coverage reaches exactly as far as the
dimension you measure, and the next blind spot is waiting one level above wherever
you put the gate.

Applying it to your own work

For anyone reading this without a blog pipeline, the same questions in short
form:

What is the composition of your last 20 units? The last 20 tickets you closed, the last 20 reports you wrote, the last 20 proposals you sent. Split them into categories and count. Not volume — distribution.
Which of your indicators would have seen this drift? If none would, "everything is green" means no more than "the three things I measure are fine".
What is carrying your balance? A list, a plan, a client portfolio — or your appetite on the day? When the list ends, the balance ends with it.
Is cost making your choices? Easily verified work is not the same as the right work. Your reflex goes to the cheapest evidence, and the shape of your expertise comes out of that.
Did you write a rule or a gate? "I will be careful" is not a rule. A rule is computed, imposed, and visible when skipped.
Who is your detector? If the answer is a person — a reader, a client, a teammate — that person is your monitoring system, and it does not scale.

Conclusion

A system looking healthy does not mean it is doing the right thing; it means it
answers the question I gave it well. An indicator that measures volume cannot, by
definition, see composition collapse: with 44 posts out of one category, the
counter still writes the same three.

Variety is not a by-product of productivity. What arrives as a by-product is
repetition — because repetition is cheaper, faster, and its evidence is already
at hand. If balance is wanted, it has to be written somewhere as a constraint,
into the system or into the day. Otherwise the reflex does the choosing, and a
reflex always reaches for the nearest shelf.

Official Sources

Who Inherits the Queue of a Closing Socket?

Mustafa ERBAY — Sat, 03 Oct 2026 13:28:57 +0000

The favourite use of SO_REUSEPORT goes like this: N worker processes listen on the same port, the kernel spreads incoming connections among them, and when you roll out a new version you replace the workers one at a time. The new worker comes up, the old one closes, the port is never idle. Wherever it is described, it sounds like zero downtime.

It is not. The connections waiting in the closing worker's accept() queue — connections that have finished their handshake and that the application has not yet picked up — die with that socket. On the client side this is an RST; on the server side it is nothing at all. It never reaches the access log, because the application never saw that connection.

The kernel has a switch for this: net.ipv4.tcp_migrate_req. It has been there since Linux 5.14, it defaults to off, and its documentation drifts from the code in at least two places. So I sat down and poked at it.

A note on the terminal output below: the measurement harness is mine and prints Turkish labels. B kabul etti means "B accepted", gocen means "migrated", dagilim means "distribution", istemci kaybi means "clients lost", and sunucu tarafi means "server side". The numbers and labels are the harness's own; where a block is long I trimmed trailing fields, but nothing was retyped or translated.

The lab, and an honest disclaimer

Let me say this first: this is not an outage story. I did not hit this problem on my own server, because there is no reuseport group there. Here is the reading:

$ ssh vps3 'ss -ltnH | awk "{print \$4}" | sort | uniq -c | awk "\$1>1"'
$ ssh vps3 'ss -ltnH | wc -l'
90

Ninety listening sockets, and not a single address:port pair shared by two of them. nginx 1.30.0 is running, but neither reuseport nor backlog appears anywhere in its configuration — one shared socket, the default queue of 511. tcp_migrate_req is off too, and both counters sit at zero:

$ ssh vps3 'uname -r; cat /proc/sys/net/ipv4/tcp_migrate_req; nstat -az | grep -i migrate'
6.8.0-142-generic
0
TcpExtTCPMigrateReqSuccess      0                  0.0
TcpExtTCPMigrateReqFailure      0                  0.0

Rather than flipping this on a live server to see what happens, I took it to the lab. The environment is Docker Desktop's linuxkit VM:

$ uname -r
6.10.14-linuxkit
$ python3 -V
Python 3.12.3
$ ss -V
ss utility, iproute2-6.1.0

Using containers has a technical reason: tcp_migrate_req is network-namespaced (net->ipv4.sysctl_tcp_migrate_req), so docker run --sysctl sets it per container and leaves the host's value untouched. I could run one round off and one round on with nothing leaking between them. The harness is two Python scripts, 311 lines together: they open listeners with SO_REUSEPORT, establish client connections, count who accepts what, and read counter deltas from /proc/net/netstat.

First measurement: is the loss real?

The setup is simple. Listener A is up alone with a backlog of 128. Twenty clients connect, and each one sends a few bytes as soon as the connection is established. A accepts none of them; all twenty sit in the queue. Then listener B joins the same port, and A closes. B tries to drain the queue.

Before A closes, ss shows the state clearly — the Recv-Q column is the number of connections waiting in the queue:

LISTEN 20     128    127.0.0.1:18081 0.0.0.0:*

Here is the result with both settings side by side:

	`tcp_migrate_req=0`	`tcp_migrate_req=1`
Accepted by B	0 / 20	20 / 20
Clients that got their data back	0	20
Clients reset	20	0
`TCPMigrateReqSuccess`	0	20
`TCPMigrateReqFailure`	0	0

A small caveat on that measurement: my harness drops both ECONNRESET and a clean end-of-file read into the same "reset" bucket, so it does not prove an RST at packet level. The source makes clear it is an RST — the inet_child_forget path leads to tcp_send_active_reset — but I know that from the code, not from my own measurement.

With the switch off, all twenty connections were gone. With it on, all twenty moved to B — and note this: the data the clients sent before the migration moved with them, so B was able to answer all twenty requests. The socket is not handed over, it is moved, receive buffer included.

There is one more thing in that table, and it is the part that bothered me: in the "off" round, twenty connections died while both counters stayed at zero.

The mechanism: when does the socket leave the group

To understand the silent counter I had to read the close path. __tcp_close runs a special branch for listening sockets, and the order is this:

if (sk->sk_state == TCP_LISTEN) {
        tcp_set_state(sk, TCP_CLOSE);

        /* Special case. */
        inet_csk_listen_stop(sk);

So the state change comes before the queue is drained. For TCP_CLOSE, tcp_set_state removes the socket from the hash table (sk->sk_prot->unhash(sk)), which calls reuseport_stop_listen_sock() by way of inet_unhash. That is where the fork in the road is:

if (READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_migrate_req) ||
    (prog && prog->expected_attach_type == BPF_SK_REUSEPORT_SELECT_OR_MIGRATE)) {
        /* Migration capable, move sk from the listening section
         * to the closed section.
         */

If the switch is on, the closing socket is not thrown out of the group; it moves from the group's "listening" section to its "closed" section, and the sk->sk_reuseport_cb pointer survives. If the switch is off, it is, as the comment puts it, "detach immediately" — the pointer becomes NULL.

Then inet_csk_listen_stop calls reuseport_migrate_sock() for every child in the queue. The first thing that function does is look up the group:

reuse = rcu_dereference(sk->sk_reuseport_cb);
if (!reuse)
        goto out;

The out label does not touch any counter. The label that increments is failure, and you can only reach it after the group has been found. So with the switch off, the kernel does not even attempt the migration — and because it does not attempt it, it does not count it.

To confirm this I closed a listener with no other member left in the group: ten connections in the queue, the only listener closed, switch on.

=== G: grupta BASKA dinleyici yokken kapatma (K=10) ===
  MIB delta: {'TCPMigrateReqSuccess': 0, 'TCPMigrateReqFailure': 10}
  istemci kaybi: 10/10

This is where the counter speaks. Running the same experiment with the switch off also killed ten connections, but the counters stayed at zero. The practical conclusion is easy to read backwards: TCPMigrateReqFailure is not a "how many connections did I lose" counter; it is a "I tried to migrate and found nowhere to go" counter. Looking at it with the feature disabled and concluding "zero, so I have no problem" is like checking that a closed door's bell is not ringing and deciding nobody is home.

Who picks the target: the docs say random, the code says hash

The kernel documentation looks clear on this:

Otherwise, the kernel will randomly pick an alive listener only if this option is enabled.

There is no randomness in the source. reuseport_migrate_sock takes the migrating socket's own hash (hash = migrating_sk->sk_hash) and picks the target with it:

i = j = reciprocal_scale(hash, num_socks);

reciprocal_scale is a multiply-and-shift that stands in for a modulo: (u32)(((u64) val * ep_ro) >> 32). Same input, same output. Measuring that turned out to be easy — I bound the clients to fixed source ports and built the same 4-tuples twice, in the same namespace, against the same destination port:

tur 1: 41000->B3, 41001->B1, 41002->B3, 41003->B3, 41004->B3, 41005->B1, 41006->B4, 41007->B4
tur 1: gocen=8/8  dagilim=B1:2 B2:0 B3:4 B4:2
tur 2: 41000->B3, 41001->B1, 41002->B3, 41003->B3, 41004->B3, 41005->B1, 41006->B4, 41007->B4
tur 2: gocen=8/8  dagilim=B1:2 B2:0 B3:4 B4:2
  ortak kaynak port: 8, AYNI hedefe gidenler: 8

Eight out of eight landed on the same listener in both rounds. You have to read "randomly" in the documentation as "unmanaged, without a policy"; if you read it as "statistically random", you will be wrong.

The second half of that output pulled me in the wrong direction for a while: the distribution is B1:2 B2:0 B3:4 B4:2, so one of the four listeners received none of the eight connections. My first reaction was "the hash does not spread evenly". Wrong reaction. For eight draws into four buckets that skew has a chi-square of 4.0, and the chance of seeing something that uneven or worse from a uniform hash is roughly 0.30 — indistinguishable from noise. Simulating the selection function itself (reciprocal_scale over uniform 32-bit input) shows the opposite at scale: with 8,000 draws across four buckets, the max-to-min ratio over 2,000 trials has a median of 1.045 and a worst case of 1.15. There is no two-to-one load gap.

The real issue is not the count but what the selection ignores. reuseport_select_sock_by_hash does not look at queue depth or current load when it picks a target; if some member of the group uses SO_INCOMING_CPU a CPU match enters the picture as well, and beyond that the only input is the connection's identity. So even when the number of connections evens out over time, their cost does not: a long-lived WebSocket and a 20-millisecond health check look identical to this selector.

That determinism is not specific to migration, either. Normal distribution came out the same way under the same harness — four listeners up from the start, no migration, fixed source ports — and eight out of eight landed on the same listener in both rounds. But the mapping changed between two separate containers, and that has a counterpart in the source as well: inet_ehashfn computes the hash using inet_ehash_secret + net_hash_mix(net). The mapping therefore carries a per-boot random secret and a per-namespace mixer. It is reproducible on your machine, and different on mine.

Migration does not respect the target's backlog

While reading the source I ran into something I was not looking for. The last step of a migration is inet_csk_reqsk_queue_add, and that function never asks about the target's queue capacity:

spin_lock(&queue->rskq_lock);
if (unlikely(sk->sk_state != TCP_LISTEN)) {
        inet_child_forget(sk, req, child);
        child = NULL;
} else {
        ...
        sk_acceptq_added(sk);
}

The only check is "is the target still listening". There is no sk_acceptq_is_full() call. On the normal path, accepting a new connection does make that check; on the migration path it does not.

So it went to the lab. Twenty connections in A's queue, and B opened with listen(1) — that is, having told the kernel "I want exactly one pending connection on this socket":

B'nin ilan ettigi backlog: 1 | ss (A+B):
LISTEN 0      1      127.0.0.1:18085 0.0.0.0:*
LISTEN 20     128    127.0.0.1:18085 0.0.0.0:*

A kapandiktan SONRA, accept'ten ONCE ss:
LISTEN 20     1      127.0.0.1:18085 0.0.0.0:*
B kabul etti: 20/20  MIB={'TCPMigrateReqSuccess': 20, 'TCPMigrateReqFailure': 0}

Recv-Q 20, Send-Q 1. A socket that asked for one has twenty waiting — twenty times the limit it declared. All of them were accepted cleanly afterwards, so this is not an accounting glitch; the queue really is that full.

For most setups this is good news: migration does not give up halfway because the target's queue is tight. For a setup that deliberately keeps its queue short, the news is mixed — and here I had to correct an assumption of my own. I thought a short backlog was a fail-fast fuse: when the queue overflows the client fails quickly and the load balancer in front moves on to another server. Default Linux does not work that way. In the experiment in the next section, when the queue overflowed, not a single client got an error; the kernel dropped the SYNs silently and the clients retried. Setting tcp_abort_on_overflow to 1 and repeating the experiment changed nothing, because that setting only engages when there is a request socket to reset, and here no request socket was ever created.

What remains is a resource limit rather than a fuse: a low backlog bounds how many ready connections the application has to deal with at once, and migration does not honour that bound. If several workers close at the same time, the one still standing wakes up with a queue many times the limit it declared. I saw this in the source first, then measured it; I could not find it mentioned in the documentation.

Which connections recover on their own

To draw the boundary of the risk I ran one more experiment. I opened A with listen(1) and threw twelve clients at it using non-blocking connect(). The queue overflowed immediately, and ss told me what the server side actually looked like:

A'da accept kuyrugu (Recv-Q/Send-Q): LISTEN 2      1      127.0.0.1:19004 0.0.0.0:*
sunucu tarafi: SYN-RECV=0  ESTABLISHED=2

Two connections finished their handshake and entered the queue. The SYNs of the remaining ten were never accepted — and rather than guess at that, I read it off the counters: ListenOverflows and ListenDrops both climb while SYN-RECV stays at zero. The kernel did not even start building a request socket, because the queue was already full; it dropped the SYN.

Then I added B (backlog 128) and closed A:

	`tcp_migrate_req=0`	`tcp_migrate_req=1`
Accepted by B	10 / 12	12 / 12
Arrived by migration (instantly)	0	2
Arrived by SYN retransmission	10	10
`TCPMigrateReqSuccess`	0	2

What migration saved was exactly two connections: the ones that had finished the handshake and had not been accepted. The counter says two as well. The other ten survived under both settings, but not thanks to migration.

In the first round I waved at what saved them — "the client retries" — and that guess needed to become a measurement, because the ~0.86 seconds I saw read like a constant. Sampling ListenOverflows once per second made the mechanism plain: the counter rises by exactly 10 every second, for four seconds running. Each of the ten blocked clients retransmits its SYN once a second. Recovery then happens at the first retransmission after the new listener exists — so the delay depends on where the swap lands in that cycle. Shifting the swap moment gives 0.74, 0.86 and 0.58 seconds, and under 0.1 seconds when the swap coincides with a retransmission. Not a fixed delay budget: a wait that ranges from zero to one second.

So the set tcp_migrate_req protects is not "everything in flight at the moment of close". If the client is still trying, TCP does its own job. What cannot be recovered is the connection that the client considers established and is now waiting on, while the server has not yet picked it up. That also happens to be the most annoying failure mode on the user's side: request sent, no answer, connection reset. Whether it is safe to retry is something the client does not know.

One more boundary, for honesty's sake: requests still mid-handshake (TCP_NEW_SYN_RECV) have a separate migration path in the source, and that path runs not at close() but when the SYN+ACK retransmission timer fires (inside reqsk_timer_handler). I could not trigger that path in this experiment — the queue overflowed, so the requests were never created, and SYN-RECV=0. Its behaviour here is therefore read from the source, not measured.

Who is allowed into the group

For migration to be on the table at all, the new worker has to have joined the same reuseport group. That group's door has two locks; these outputs come from two separate runs, with the harness's own labels:

=== SENARYO D (port 18086) gruba katilma kapilari ===
  SO_REUSEPORT'suz bind: OSError errno=98 (EADDRINUSE)

=== F: gruba farkli UID ile katilma ===
  uid=65534 + SO_REUSEPORT bind: OSError errno=98 (EADDRINUSE)
  uid=0 (ayni kullanici) bind: BASARILI

The first is expected. The second is the one that can catch you in production — and it should not really be a surprise, because socket(7) states it outright: "To prevent port hijacking, all of the processes binding to the same address must have the same effective UID." Being written in the manual is not the same as being remembered on deployment day. A procedure that starts the new version under a different service account cannot bind to that port at all; the migration question never even comes up. And EADDRINUSE will not tell you why — it says "port busy", not "different user".

The ordering inside the group is not arbitrary either. socket(7) defines the numbering: "Sockets are numbered in the order in which they are added to the group (that is, the order of bind(2) calls for UDP sockets or the order of listen(2) calls for TCP sockets)." That is the array the hash is scaled into. In other words, the order in which you start your workers is one of the inputs that decides which connection goes to which worker.

One more run, because the documentation mentions shutdown() alongside close(). I called shutdown(SHUT_RDWR) on the listening socket, leaving the file descriptor open:

A.shutdown(SHUT_RDWR) cagrildi, fd ACIK
B kabul etti: 10/10  MIB={'TCPMigrateReqSuccess': 10, 'TCPMigrateReqFailure': 0}

Ten out of ten migrated with the switch on, zero out of ten with it off. If you are writing a graceful drain, that is useful: you can stop listening and hand off the queue without closing the descriptor.

Before you turn it on

The part of the documentation that deserves the most attention is at the end, and it is a warning:

Note that migration between listeners with different settings may crash applications. Let's say migration happens from listener A to B, and only B has TCP_SAVE_SYN enabled. B cannot read SYN data from the requests migrated from A.

Migration assumes the two listeners have identical socket options. If they differ, the target ends up reading a connection it did not set up using its own assumptions. During a version rollout, where the old and new worker's socket options can drift apart, that is a real risk — and that is precisely the moment you would most want migration. The kernel's suggested answer is to pick the target with a BPF_SK_REUSEPORT_SELECT_OR_MIGRATE eBPF program and return SK_DROP to cancel the migration when no suitable target exists. Flipping the bare sysctl means leaving that policy to "whatever the hash says".

The decision framework I ended up with:

If several processes listen on the same port and you close the old one during a rollout, turn it on. nginx's reuseport parameter does exactly that: per the documentation, "an individual listening socket for each worker process". A reload closes the old workers' sockets.
If you have a single listening socket, nothing changes — but which kind of "single" you have shows up in the counters. If the socket was opened with SO_REUSEPORT there is still a group: migration is attempted, finds nowhere to go, Failure rises, and the connection dies anyway. If SO_REUSEPORT is absent — which is the case for all 90 listeners on my own server — the if (rcu_access_pointer(sk->sk_reuseport_cb)) gate inside inet_unhash never opens, the migration code never runs, and both counters stay at zero. Either way the fix lives elsewhere: an approach like systemd socket activation, which leaves behind a socket that can be handed over.
If you deliberately keep your queue short, think twice. Migration does not honour that limit.
If your workers' socket options are not identical, avoid the bare sysctl: use an eBPF policy, or do not turn it on at all.

Three commands are enough to check. First, do you even have a group:

ss -ltnH | awk '{print $4}' | sort | uniq -c | awk '$1>1'

Then the setting and the counters:

cat /proc/sys/net/ipv4/tcp_migrate_req
nstat -az | grep -i migrate

Both flags are needed, for different reasons: without -a, nstat prints the delta against its own history file (which is kept per user, so running it under sudo reads a different ledger), and without -z, counters sitting at zero drop out of the listing entirely — so if no migration has been attempted yet, you will not see the line at all. And as the runs above show, while the feature is off these counters will not report your losses.

Where the counter goes quiet

This is a long story for a single byte. Looking back, what stayed with me is not the setting but the two gaps I ran into while chasing it.

One is on the instrumentation side. With migration off, the kernel threw away twenty connections and not one counter moved — because that counter counts migrations that were attempted and failed, not the ones that were never attempted. A gauge reading zero does not mean the thing it measures is not happening; it does not even mean the gauge is connected. That is exactly why both counters sitting at zero on my own server tell me nothing.

The other is whose point of view "zero downtime" is spoken from. The port was never idle, there was no gap in the process table, there is no error line in nginx's access log. Looked at from the server, there really was no downtime. The downtime happened on the side of twenty clients that had finished their handshake, and that side does not write into our ledger. If the client is still retrying, TCP repairs itself; if it has stopped, we are the ones creating the loss. It helps to think of this setting not as a performance knob but as where you draw the line between those two.

A last note on currency. The feature arrived in 2021 with Linux 5.14 (f9ac779f881c, "net: Introduce net.ipv4.tcp_migrate_req", Kuniyuki Iwashima) and it still sits in today's mainline in the same place with the same default. It has not been removed, renamed or marked discouraged; it has simply been off by default for five years and rarely discussed. The kernel's own test suite has a case called migrate_reuseport.c — the right place to read if you want to see how they exercise the migration.

Official Sources

I Enabled casefold, and the Directory Stayed Case-Sensitive in Turkish

Mustafa ERBAY — Sat, 03 Oct 2026 06:27:43 +0000

Anyone who has moved a file server from Windows to Linux hits the same wall: Rapor.docx and rapor.docx are one file on Windows and two on ext4. The casefold feature that arrived in ext4 with Linux 5.2 exists precisely to fix that — per-directory case insensitivity, implemented inside the filesystem itself.

This week I sat down and turned it on. It worked: I found Rapor.txt by asking for RAPOR.TXT, and by asking for RaPoR.TxT. Then I tried Turkish, and saw this — in the same directory, at the same time, IŞIK.TXT and işik.txt are one file, while IŞIK.TXT and ışık.txt are two.

So my directory became case-insensitive in English and case-sensitive in Turkish. Not half-broken; perfectly consistent, just not with the rule I expected. The reason is a single line of kernel source, and I found it.

The second finding bothers me more: cp, rsync and tar — all three — copy two files into that directory, return zero, and leave one file behind. No error, no warning, clean exit code.

A note on the terminal blocks below: they are the output of the runs as they happened, on a Turkish-locale shell, so some labels are Turkish (BULDU = found, YOK = missing, ESLESTI = matched, kalan_ad = surviving name, icerik = content). I have left machine output exactly as it came out rather than retyping it in English.

The lab

Every measurement below comes from this environment, on fresh ext4 images on loop devices:

$ . /etc/os-release; echo "$PRETTY_NAME"
Ubuntu 26.04 LTS
$ uname -r
7.0.0-34-generic
$ mke2fs -V
mke2fs 1.47.2 (1-Jan-2025)

Setup is two commands:

sudo mkfs.ext4 -q -O casefold -E encoding=utf8 /var/tmp/cf.img
sudo mount -o loop /var/tmp/cf.img /mnt/cf

casefold is a filesystem feature, but on its own it makes nothing insensitive. All it does is record in the superblock how text is encoded on this filesystem:

$ sudo tune2fs -l /var/tmp/cf.img | grep -i "character encoding"
Character encoding:       utf8-12.1

Insensitivity is per directory, switched on with the +F inode flag. The kernel documentation says it in one sentence: "It is enabled by flipping the +F inode attribute of an empty directory." The empty-directory requirement is a real requirement, not advice:

$ mkdir /mnt/cf/dolu && touch /mnt/cf/dolu/x
$ chattr +F /mnt/cf/dolu
chattr: Directory not empty while setting flags on /mnt/cf/dolu

On an empty directory it goes through, and once set, every directory you create underneath inherits the flag:

$ mkdir /mnt/cf/ci && chattr +F /mnt/cf/ci
$ mkdir /mnt/cf/ci/alt && lsattr -d /mnt/cf/ci/alt
--------------e--F---- /mnt/cf/ci/alt

In practice those two rules combine into this: you have to catch the top of the tree you want insensitive while it is still empty. You cannot convert a populated share after the fact — you have to create a new tree and move things into it. The move itself is its own problem, and I will get to it.

The mount point itself can never be insensitive, because the root of a fresh ext4 is never empty:

$ ls -a /mnt/cf
.  ..  ci  dolu  lost+found
$ sudo chattr +F /mnt/cf
chattr: Directory not empty while setting flags on /mnt/cf

lost+found closes that door from the start. Your insensitive tree will always be a subdirectory of the mount point.

First, the portability trap: mke2fs does not consult your kernel

I walked into this one myself. I created the casefold image inside Docker Desktop's VM, mkfs ran happily, and then it would not mount:

# uname -r
6.10.14-linuxkit
# mkfs.ext4 -q -O casefold -E encoding=utf8 /tmp/t.img && echo BASARILI
BASARILI
# mount -o loop /tmp/t.img /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error.
# dmesg | tail -1
[309036.421209] EXT4-fs (loop0): Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE

The cause is plain:

# zcat /proc/config.gz | grep -i CONFIG_UNICODE
# CONFIG_UNICODE is not set

mke2fs does not check whether your kernel supports the feature; it writes the flag into the superblock and calls it a day. So when you move a casefold disk to a machine whose kernel was built without Unicode support, the disk is not corrupted — it simply never mounts. Rescue environments, embedded devices, minimal VM images: all candidates.

The picture across my own fleet looks like this. Docker Desktop's linuxkit kernel does not support it; VPS3 does, but nothing uses it:

# VPS3
$ uname -r; grep -i "^CONFIG_UNICODE" /boot/config-$(uname -r)
6.8.0-142-generic
CONFIG_UNICODE=y
$ for d in sda1 sda16; do printf "%s: " $d; \
    sudo dumpe2fs -h /dev/$d 2>/dev/null | grep -o casefold || echo "casefold YOK"; done
sda1: casefold YOK
sda16: casefold YOK

Make dumpe2fs -h | grep casefold a habit before you take a backup image. Ever since the evening df reported zero while root kept writing, I stopped doing filesystem work without looking at the feature flags.

Baseline behaviour: names are preserved, lookups flex

Here is the part that works the way I expected — and it works well:

$ touch /mnt/cf/ci/Rapor.txt
$ for n in Rapor.txt rapor.txt RAPOR.TXT RaPoR.TxT; do
    [ -e "/mnt/cf/ci/$n" ] && echo "BULDU  $n" || echo "YOK    $n"; done
BULDU  Rapor.txt
BULDU  rapor.txt
BULDU  RAPOR.TXT
BULDU  RaPoR.TxT
$ ls /mnt/cf/ci
Rapor.txt
alt

All four resolve to the same file, yet ls still says Rapor.txt (alt is the inherited subdirectory from the previous step). In the kernel documentation's words, the behaviour is "name-preserving on the disk" — the name written to disk is a byte-per-byte match of what the user supplied. Only the comparison is flexible, not the storage. Windows and NTFS do the same thing: keep the name as given, flex the comparison.

This is where I saw the first silence:

$ touch /mnt/cf/ci/RAPOR.TXT; echo "exit=$?"
exit=0
$ ls /mnt/cf/ci
Rapor.txt
alt

touch did not create a new file; it opened the existing one and updated its mtime. Exit code zero. If you want to state that your intent was "a new file", you need O_EXCL — and then you learn the truth:

>>> os.open('/mnt/cf/ci/RAPOR.TXT', os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0o644)
FileExistsError: [Errno 17] File exists: '/mnt/cf/ci/RAPOR.TXT'

Keep that in mind: only the caller who asks gets to see EEXIST.

Turkish: one directory, two different rules

Now the part that surprised me. I created a file called IŞIK.TXT and looked for its variants:

Name on disk	Lookup	Result
`IŞIK.TXT`	`IŞIK.txt`	match
`IŞIK.TXT`	`işik.txt`	match
`IŞIK.TXT`	`ışık.txt`	no match
`IŞIK.TXT`	`Işık.txt`	no match
`İŞLEM.TXT`	`İŞLEM.txt`	match
`İŞLEM.TXT`	`işlem.txt`	no match
`İŞLEM.TXT`	`i̇şlem.txt` (`i` + U+0307)	match

Read that again. As far as this directory is concerned, the lowercase of IŞIK.TXT is işik — so the spelling that is wrong in Turkish matches, and the correct one does not. The consequence shows up immediately: when I tried to create ışık.txt, I got a second file.

>>> os.open(b+'ışık.txt', os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0o644)   # created
>>> os.open(b+'işik.txt', os.O_CREAT|os.O_EXCL|os.O_WRONLY, 0o644)   # EEXIST
>>> sorted(os.listdir(b))
['IŞIK.TXT', 'İŞLEM.TXT', 'ışık.txt']

The directory now holds, to a Turkish reader, the same word twice in two cases — and the filesystem counts them as two different things.

Why: the kernel drops the T rows

This is not a bug; it is a deliberate choice. The kernel's Unicode tables are generated from the Unicode Character Database, and fs/unicode/README.utf8data lists exactly which files feed them — CaseFolding.txt among them. Every row in that file carries a status code: C (common), F (full), S (simple), T (Turkish/Azeri special case).

Inside fs/unicode/mkutf8data.c, which generates the tables, a single line tells the whole story:

/* Use the C+F casefold. */
if (status != 'C' && status != 'F')
        continue;

The T rows are discarded. Here are the CaseFolding.txt entries for the letters involved — the annotations on the right are mine:

0049; C; 0069;       # LATIN CAPITAL LETTER I                 -> kept
0049; T; 0131;       # LATIN CAPITAL LETTER I                 -> DROPPED
0130; F; 0069 0307;  # LATIN CAPITAL LETTER I WITH DOT ABOVE  -> kept
0130; T; 0069;       # LATIN CAPITAL LETTER I WITH DOT ABOVE  -> DROPPED
015E; C; 015F;       # LATIN CAPITAL LETTER S WITH CEDILLA    -> kept

The detail that closes the table is the one not in that list: CaseFolding.txt has no row at all for ı (U+0131) or ş (U+015F), so both fold to themselves. UnicodeData.txt does record I as the uppercase of ı, but folding and uppercasing are different operations, and the filesystem uses folding.

From there: IŞIK folds to işik (I → i, Ş → ş) while ışık folds to ışık. Different. İŞLEM folds to i̇şlem, that is i plus a combining dot, whereas işlem is a plain i. Different again. All seven rows of the table come out of these entries.

The İ row has a subtlety. The canonical decomposition of U+0130 is already 0049 0307, that is capital I plus the dot — and the C row folds that I to i. So even without the 0130; F; row the result would land in the same place; two paths meet at one point.

The reasoning behind the choice is defensible: a filesystem has no locale. When you mount the same disk from a Turkish machine and an English one, the answer to "which file is which" cannot change. Had they kept the T rows, the file that IŞIK.TXT resolves to would depend on the machine's LANG. Between correctness and stability, they picked stability.

This is not an ext4 quirk

I ran the identical test on my own Mac (macOS 26.6.2, APFS, the default insensitive volume):

dosya 'IŞIK.TXT' varken 'ışık.txt' -> -
dosya 'IŞIK.TXT' varken 'işik.txt' -> ESLESTI
dosya 'İŞLEM.TXT' varken 'işlem.txt' -> -
dosya 'İŞLEM.TXT' varken 'i̇şlem.txt' -> ESLESTI
NFC 'rapor-é.txt' varken NFD arama -> ESLESTI

Line for line the same. So there is no story here about Linux not knowing Turkish; this is the shared stance of case-insensitive filesystems. It has worked this way on your Mac for years and you probably never noticed — because nobody wakes up wanting to put both IŞIK and ışık in the same folder. On a migrated file server with 200,000 files, it happens for you.

casefold does normalization too, not just letters

The sentence that caught my eye in the documentation: "The comparison algorithm is implemented by normalizing the strings to the Canonical decomposition form, as defined by Unicode, followed by a byte per byte comparison." So NFD normalization is part of the comparison. I tested it — wrote é as a single code point (NFC, U+00E9) and looked it up as two (NFD, e + U+0301):

olusturuldu NFC 'rapor-é.txt' (12 bayt)
  NFD ile arama 'rapor-é.txt'  -> BULDU
  NFD buyuk harf 'RAPOR-É.TXT' -> BULDU
  NFC buyuk harf 'RAPOR-É.TXT' -> BULDU

The control group is a directory on the same filesystem without +F — there, all four are separate files:

>>> # /mnt/cf/plain (plain directory)
>>> os.path.exists(b+'RAPOR.TXT')          # with 'Rapor.txt' present
False
>>> os.path.exists(b+'rapor-é.txt')  # with the NFC version present
False
>>> sorted(os.listdir(b))
['RAPOR.TXT', 'Rapor.txt', 'rapor-é.txt', 'rapor-é.txt']

Look at that last line: the directory holds two names that look identical. One is NFC, the other NFD. ls prints both as rapor-é.txt. This bites you with archives coming from macOS, quite independently of case, and it drives people up the wall — this may well be the more valuable problem casefold solves.

The full lookup path works like this. The kernel runs the folding and the decomposition from one combined table; the split below is for readability:

Three tools, zero exit codes, one file

Everything so far was "interesting". This section is "be careful in production".

On a plain ext4 I prepared two files whose names differ only in case, with different contents:

$ cd /var/tmp/kaynak && ls -1 && md5sum Rapor.txt RAPOR.TXT
RAPOR.TXT
Rapor.txt
a336ddc0133b9f2219b73d9fbdc43239  Rapor.txt
2b40728022b6f578b4f6a93288dce045  RAPOR.TXT

Then I copied them into three separate, freshly created casefold targets:

[cp]    exit=0  kalan_ad=Rapor.txt  icerik=YENI SURUM
[rsync] exit=0  kalan_ad=RAPOR.TXT  icerik=ESKI SURUM
[tar]   exit=0  kalan_ad=RAPOR.TXT  icerik=YENI SURUM

All three returned zero. All three left one file in the target. rsync went further and printed >f+++++++++ on two separate lines — it claimed it had transferred two new files.

Which file survives depends on the tool, and that is a trap of its own. cp and rsync keep the name they created first and overwrite the content from the second file; tar unlinks and recreates on a collision, so it takes both the name and the content from the last entry. Run the same source into the same kind of target with three tools and you get three different results — and with cp and rsync you are left with a hybrid whose name came from one file and whose content came from the other.

The one thing that does not change: one of your two files is gone, and no exit code said so.

Before migrating: scan, but scan correctly

My first instinct was a tolower-based awk one-liner. I tested it against deliberate collisions and it came up short. There are six names in the source directory; three of them collide:

$ ls -1 /var/tmp/tara
IŞIK.TXT
RAPOR.TXT
Rapor.txt
işik.txt
rapor-é.txt
rapor-é.txt

The awk/tolower script saw only two of them — it missed the NFC/NFD pair. What the kernel actually does came out of copying the same names into a casefold directory: six names collapsed to three in the target. So the pair it missed is real data loss.

The right key is the one the kernel uses: casefold first, then NFD.

python3 - <<'PY'
import os, unicodedata, collections, sys
kok = sys.argv[1] if len(sys.argv) > 1 else '.'
g = collections.defaultdict(list)
for d, _, dosyalar in os.walk(kok):
    for f in dosyalar:
        g[(d, unicodedata.normalize('NFD', f.casefold()))].append(f)
for (d, _), v in sorted(g.items()):
    if len(v) > 1:
        print('CAKISMA:', d, sorted(v))
PY

In the same directory it finds all three:

CAKISMA: ['IŞIK.TXT', 'işik.txt']
CAKISMA: ['rapor-é.txt', 'rapor-é.txt']
CAKISMA: ['RAPOR.TXT', 'Rapor.txt']

Do not use this script without planting a deliberate collision first, or you will not be able to tell "found nothing" apart from "does not work". What it costs to over-trust a single measurement is laid out in that "very small time slice" piece.

One more caveat: Python's casefold uses the Unicode tables of your installed interpreter, not 12.1, and it may not line up with the filesystem on ı/İ. The script gives you a candidate list, not a verdict — make the call by looking at the names.

strict: keeping invalid UTF-8 at the door

casefold has another quiet side. In the default configuration a filename that is not valid UTF-8 is accepted — but insensitivity does not work for it. The documentation says so outright: when invalid strings are encountered, "it falls back to considering the entire string as an opaque byte sequence, which still allows the user to operate on that file, but the case-insensitive lookups won't work."

Measured, exactly so:

>>> os.open(b'/mnt/cf/op/bozuk-\xff\xfe.txt', os.O_CREAT|os.O_WRONLY, 0o644)  # created
>>> os.path.exists(b'/mnt/cf/op/bozuk-\xff\xfe.txt')
True
>>> os.path.exists(b'/mnt/cf/op/BOZUK-\xff\xfe.TXT')
False

The result is a directory in which some names are insensitive and some are not. You cannot tell which is which without inspecting the bytes of the name. Closing that door is only possible at mkfs time, with the strict flag — and the mke2fs man page notes the default is off:

sudo mkfs.ext4 -q -O casefold -E encoding=utf8,encoding_flags=strict /var/tmp/st.img

The outcome splits in two, and the split matters:

strict +F dizin             : EINVAL (Invalid argument)
strict ama +F OLMAYAN dizin : olusturuldu (EVET)

strict does not cover the whole filesystem; it only applies inside insensitive directories.

And here a gap turned up that I had not expected: you cannot read back whether strict is on. I compared a strict and a non-strict image with three different tools, and all three said the same thing:

$ sudo tune2fs -l  /var/tmp/s2.img | grep -i "encoding"
Character encoding:       utf8-12.1
$ sudo dumpe2fs -h /var/tmp/s2.img | grep -i "encoding"
Character encoding:       utf8-12.1
$ sudo debugfs -R "features" /var/tmp/s2.img
Filesystem features: has_journal ext_attr ... casefold sparse_super ...

No Encoding flags line is ever printed. The only way to answer "is strict on for this disk" is to try writing an invalid UTF-8 name into a +F directory and see whether you get EINVAL. If you are building a new share, turn encoding_flags=strict on from the start and write it down somewhere — you can neither add it later nor ask for it later.

Cost: the first measurement was wrong

I was curious about lookup cost. I created two directories of 20,000 files — one +F, one plain — and timed 20,000 random stat calls. The first result was this:

casefold (+F)  tam yazim     6.93 us/arama
duz dizin      tam yazim     1.65 us/arama

A factor of 4.2. I was about to write "folding is expensive", except in the same run a lookup against the casefold directory with a different spelling came out at 1.86 µs. That would mean the case that requires folding is faster than the case that does not. Impossible; the measurement order had leaked in — whichever directory was measured first was paying for a cold dentry cache.

I added warm-up passes, interleaved the scenarios, and took the median of three rounds:

casefold (+F)  tam yazim     medyan   1.25 us/arama  (1.36, 1.16, 1.25)  isabet=20000
casefold (+F)  farkli yazim  medyan   1.63 us/arama  (1.63, 1.54, 1.69)  isabet=20000
duz dizin      tam yazim     medyan   1.19 us/arama  (1.08, 1.19, 1.21)  isabet=20000
duz dizin      farkli yazim  medyan   0.87 us/arama  (0.87, 1.04, 0.86)  isabet=0

That is the real table, and even the comparison in its first two rows is weaker than I thought. For exact-spelling lookups, the casefold directory's best round (1.16) beats the plain directory's worst (1.21); with three rounds I cannot claim to have measured that difference. The honest statement: there is no measurable difference for exact spellings.

When the name genuinely has to be folded, the gap does clear the rounds: 1.63 µs against 1.19, roughly 37%. The last row is not comparable, and let me underline that — all 20,000 lookups there are misses, and a negative dentry is cheaper than a hit.

Directory metadata grew negligibly too at 20,000 files: 708,608 bytes against 696,320, that is 1.8%. Do not go looking for a performance argument to reject this feature; the arguments live elsewhere.

Nobody would have noticed if I had published the wrong measurement, so let me note it: a micro-benchmark taken in a single round and a single order tells you about your ordering, not about the thing you measured.

Your backup does not carry `+F`

This was nowhere in my plan, and I think it is the most expensive finding in the article. +F is an inode flag, and archive formats have no place for it:

$ lsattr -d /mnt/y/canli
--------------e--F---- /mnt/y/canli

$ tar -C /mnt/y -cf /var/tmp/yedek.tar canli
$ tar -C /mnt/y/geri1 -xf /var/tmp/yedek.tar
$ lsattr -d /mnt/y/geri1/canli
--------------e-------

$ rsync -aAX /mnt/y/canli /mnt/y/geri2/
$ lsattr -d /mnt/y/geri2/canli
--------------e-------

Not even rsync's -aAX carries it — those flags are for ACLs and extended attributes, not inode flags. So when you back up your insensitive share and restore it, what you get back is a sensitive directory, and no command reports an error.

What follows is worse: in the restored tree, Rapor.txt and RAPOR.TXT can sit side by side again. If the next restore target is +F, the "three tools ate a file" trap from earlier in this article fires during recovery — that is, while you are coming back from a disaster.

The remedy is simple but has to be remembered: your restore procedure must create the empty +F directories first and unpack into them. If you have not measured this in a recovery drill, you do not know it.

Security: anyone who can choose a name can choose a collision

Insensitivity changes the answer to "which file is which". Anywhere an attacker controls a filename, that becomes a question about authority. Measured:

$ chattr +F /mnt/y/guv
$ printf 'ssh-ed25519 AAAA...SAHIBI\n'   > /mnt/y/guv/authorized_keys
$ printf 'ssh-ed25519 AAAA...BASKASI\n'  > /mnt/y/guv/AUTHORIZED_KEYS
$ ls -1 /mnt/y/guv
authorized_keys
$ cat /mnt/y/guv/authorized_keys
ssh-ed25519 AAAA...BASKASI

Someone with permission to write AUTHORIZED_KEYS changed the contents of authorized_keys. The filename stayed the same, sshd reads the same path, and the content belongs to someone else. The same logic applies to every fixed-name file: .htaccess, .env, index.php.

The rule that falls out: enable casefold on shares that hold user data, not on trees that hold configuration and credentials. If they live in the same tree, give +F to the data subtree rather than to a common ancestor.

Backing out: two separate doors

If your plan is "I will turn it off if I don't like it", both doors are tighter than you think.

The directory flag only comes off while the directory is empty — the same condition as putting it on:

$ mkdir /mnt/cf/bos && chattr +F /mnt/cf/bos && chattr -F /mnt/cf/bos
$ lsattr -d /mnt/cf/bos
--------------e------- /mnt/cf/bos          # cleared

$ mkdir /mnt/cf/dolu2 && chattr +F /mnt/cf/dolu2 && touch /mnt/cf/dolu2/a
$ chattr -F /mnt/cf/dolu2
chattr: Directory not empty while setting flags on /mnt/cf/dolu2

The filesystem feature itself is more stubborn. On a fresh filesystem where no +F directory was ever created, it clears without complaint. But once you have created one — even if you deleted it — tune2fs refuses:

$ mkdir /mnt/m/ci && chattr +F /mnt/m/ci && touch /mnt/m/ci/a
$ rm -rf /mnt/m/ci && sudo umount /mnt/m
$ sudo tune2fs -O ^casefold /var/tmp/m.img
The casefold feature can't be cleared when there are inodes with +F flag.

No directory, no file, and the flag still counts as present. The fix is e2fsck:

$ sudo e2fsck -fy /var/tmp/m.img >/dev/null
$ sudo tune2fs -O ^casefold /var/tmp/m.img
tune2fs 1.47.2 (1-Jan-2025)
$ sudo tune2fs -l /var/tmp/m.img | grep "^Filesystem features"
Filesystem features:      has_journal ext_attr resize_inode dir_index orphan_file
filetype extent 64bit flex_bg metadata_csum_seed sparse_super large_file huge_file
dir_nlink extra_isize metadata_csum

casefold is off the list. The interesting part is that e2fsck does not clear the flag. I read the deleted directory's inode before and after:

-- after deletion --   Inode: 13   Type: directory   Flags: 0x40080000
-- after e2fsck --     Inode: 13   Type: directory   Flags: 0x40080000

0x40000000 is exactly FS_CASEFOLD_FL. The flag byte sits in the inode table and stays there after e2fsck too; what changes is that the inode no longer counts as in use, so tune2fs's scan skips it from then on. I did not read what e2fsck thought it repaired; the behaviour is what the lab recorded.

The error message is correct, just incomplete — it does not say "run e2fsck -f". I hit that same wall on four separate attempts; I am writing it down so you do not hit it once.

A decision frame

If I am building a new insensitive share:

At mkfs time, -O casefold -E encoding=utf8,encoding_flags=strict. You can neither add strict later nor read it back; write the decision down.
chattr +F the root of the tree while it is empty; subdirectories inherit it.
The mount point is never empty because of lost+found, so you will always work with a subdirectory.
Verify CONFIG_UNICODE=y on every kernel that will mount this disk. Count the rescue environment.
Keep configuration and credential files outside the insensitive tree.

If I am migrating an existing tree:

Scan with a casefold + NFD key; plain tolower misses normalization collisions.
Prove the scanning script with a deliberate collision.
Do not trust the exit codes of cp/rsync/tar. Compare the file count between source and target afterwards; that is the only check that works.
Know that I/ı and İ/i pairs in Turkish names do not count as collisions as far as the filesystem is concerned.

For backup and recovery:

The restore procedure has to create the +F directories empty first. The archive does not carry the flag.
In a drill, check lsattr on the restored tree, and the file count too.
Backing out with ^casefold needs umount plus e2fsck -f; in production that means planned downtime.

Places I would not touch at all:

Applications that compare filenames themselves. readdir returns the name on disk; insensitivity stays in the kernel.
Backup targets under continuous sync. Every sync from a sensitive source into an insensitive target carries a silent-merge risk.

Conclusion

casefold is a good feature and it works. My mistake was reading the phrase "case-insensitive" with the meaning it has in my own language. The filesystem never promised me insensitivity; it promised an equivalence defined by Unicode, independent of locale. The two coincide in English and diverge in Turkish, and the filesystem does not regard the second case as an exception — that is simply the rule.

The lesson I take from this is bigger than filesystems: when a system says two things are "the same", do not build on top of it before asking whose "same" it means. tolower has a locale; casefold does not. cp's exit code says "I transferred", not "I transferred all of them". rsync -aAX reads like "preserve everything" but does not cover inode flags. The gaps between those are silent — right up until one file is missing from a 200,000-file migration.

Two current notes. casefold is spreading: tmpfs now supports casefold=utf8-12.1.0 and strict_encoding as mount options, with the same +F and inheritance rules. And the "your application does not know" gap above is closing — FS_XFLAG_CASEFOLD and FS_XFLAG_CASENONPRESERVING have landed in the kernel's userspace interface, so an application can now ask FS_IOC_FSGETXATTR whether a directory is insensitive. The comment above the definition carries its own warning: do not assume the bit belongs to directories only.

One more: the kernel's Unicode tables are still generated from 12.1.0, the release from 2019. For something like filename equivalence, a table frozen for seven years is, I suspect, a choice rather than an oversight.

Official Sources

Every New Command I Type Deletes an Older One

Mustafa ERBAY — Fri, 02 Oct 2026 14:23:42 +0000

It started with a simple question: how far back does my shell history go? How much of
the years spent in a terminal is still sitting there? Answering it looked like a
single-file job.

$ wc -l ~/.zsh_history
    1218 /Users/mustafaerbay/.zsh_history

One thousand two hundred eighteen lines. But how many months is that? That is where I
stalled, because the file contains not a single date — neither the oldest nor the newest
line says when it was written. Looking elsewhere turned up /etc/zshrc in the same
neighbourhood, with these three lines inside:

HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history
HISTSIZE=2000
SAVEHIST=1000

The file is owned by root, mode r--r--r--, last modified on 13 August. This is not a
setting of mine; it is a decision Apple shipped with the machine. And it says: at most
one thousand commands go into that file.

When I finally counted the file properly, that was exactly the number: 1,000. Not
one below the ceiling — precisely on it. Which means this file filled up months ago, and
every command I have typed since then has silently deleted the oldest one.

Every number below was measured on this machine on 2 October 2026: macOS 26.6.2
(25G83), zsh 5.9, arm64. The file shifted underneath me mid-measurement — a terminal
window I had forgotten about closed, and the size went from 52,982 to 53,283 bytes — so
a copy was frozen at 16:47:35, and every figure comes from that single copy.

First I counted it wrong, and the wrong way of counting was the lesson

My first attempt said 939. Not a rounding error but a thesis-inverting one: 939 means
"sixty-one commands left before the ceiling"; 1,000 means "the ceiling is full and the
losses started long ago."

Here is the mistake. zsh records events, not lines; a multi-line command is one event,
with the line breaks marked by backslashes. The correct rule is a single sentence: an
event ends at the first line that does not end in a backslash.

total lines                       1218
lines ending in a backslash        218
=> event count                    1000

My first script threw away blank lines as "not data." But none of the 61 blank lines in
that file is an independent entry; every one of them sits in the middle of a multi-line
command that happens to contain an empty line. Dropping them removed 61 lines and
stitched the surrounding commands together at the wrong seams.

I verified it with zsh itself rather than with arithmetic alone. Under a temporary
ZDOTDIR, the shell read the file:

$ fc -R hist-snapshot.txt
$ print ${#history}; print $HISTCMD
999
1000

The history array excludes the current event slot, so 999 + 1 = 1,000. The independent
arithmetic agrees. The lesson is cheap to state and expensive to learn: when counting a
file, use the rules of the program that writes it, not your own intuition.

What happens once the ceiling is full?

This one went to the lab, reproduced without touching my real history file, inside a
temporary ZDOTDIR. A file of 995 fake entries, then 20 more commands:

start:  995 entries, first=eski_0001  last=eski_0995
after: 1000 entries, first=eski_0016  last=yeni_20

The total should have been 1,015; the file stopped at 1,000 and the first fifteen
entries were gone. Twenty more commands, and eski_0016 through eski_0035 went the
same way. No error, no warning, no log line. My own file is in exactly this state.

The single-session version is starker. A shell took 1,500 commands into memory, fc -l
counted all of them while the session ran, and then the shell closed:

in memory (fc -l): 1500
on disk:           1000
first line:        lab_komut_501

Five hundred never reached the disk. In the zsh documentation HISTSIZE is "the maximum
number of events stored in the internal history list" and SAVEHIST is "the maximum
number of history events to save in the history file." Two numbers, two memories: one is
what you remember during the session, the other is what you still have the next morning.

No entry has a time — but the shell will show you one

How many of the thousand entries carry a timestamp? Zero. zsh writes timestamps in the
: <beginning time>:<elapsed seconds>;<command> form, and only when EXTENDED_HISTORY
is on. The documented default is off, and so is the state on this machine. There is not
a single line starting with HIST in my ~/.zshrc, so this too is an inherited default.

So far this is annoying but honest: if the information is absent, it is absent. The sly
part begins when you ask the shell. I listed a timestamp-free file from two separate
shells four seconds apart, using fc -l -t '%Y-%m-%d %H:%M:%S' so the seconds show:

read 1 — wall clock 16:40:46
    1  2026-10-02 16:40:46  eski_komut_bir
    2  2026-10-02 16:40:46  eski_komut_iki

read 2 — wall clock 16:40:50
    1  2026-10-02 16:40:50  eski_komut_bir
    2  2026-10-02 16:40:50  eski_komut_iki

The same two commands, two different times. The value shown is not when the command ran
but when the file was read — not even the file's modification time. I went looking for
this in the documentation and could not find it: fc only defines the format, not what
gets printed for an undated file. So the sentence below rests on measurement, not on the
manual: it fills an empty field and shows it to you.

The rule that falls out has now caught me twice on this blog: a tool showing you a number
does not mean it knows that number. While
counting my laptop's sleep records
the uptime counter offered a reassuring figure, but what it measured was not "how long
I stayed awake," it was "how long since the last reboot."

The order is not chronological either

If there are no timestamps, is the ordering at least right? Two shells. A started first,
wrote three commands, waited three seconds. B started half a second later, wrote three
commands and exited immediately.

[t=0.5] B exited ->     [t=3.5] A exited too ->
kabukB_1                kabukB_1
kabukB_2                kabukB_2
kabukB_3                kabukB_3
                        kabukA_1
                        kabukA_2
                        kabukA_3

A's commands were typed first and sit last in the file. Because the definition of
HISTFILE is literally this: "the file to save the history in when an interactive shell
exits." The moment of writing is not when you type the command, it is when you close the
window. Thanks to the default APPEND_HISTORY no session overwrites another; but the
file's order is the order in which I closed terminal tabs.

For anyone working across three screens that means the bottom line is not "what did I do
last," it is "which window did I close last."

A crashed shell remembers nothing

If the definition says "when an interactive shell exits," what does a shell that never
exits do? Five commands, then kill -9:

exit code: 137
ls: .../.zsh_history: No such file or directory

The file was never created. Closing the same shell cleanly wrote three lines properly.
So every time you force-restart the machine without closing your terminal, whatever you
typed in that session is gone — and since the loss leaves no record, I cannot even count
how often it has happened.

The remedy is in the documentation: INC_APPEND_HISTORY adds new lines to $HISTFILE
"incrementally (as soon as they are entered)." Default: off.

Commands the agent runs never arrive at all

One number nagged at me. The claude command appears in only 12 of the thousand
entries, while most of the work on this machine happens inside those twelve sessions.

$ zsh -c 'echo etkilesimsiz_komut_calisti'
etkilesimsiz_komut_calisti
$ zsh s.zsh
betik_icinden
--- history file ---
0 lines

Zero. The definition is explicit again: the file is written only when an interactive
shell exits. Nothing an agent, a script or a cron job runs ever lands in this ledger.

Twelve lines, twelve sessions, and no way to learn from this file how many commands ran
inside them. Shell history is a record not of "what I did" but of "what I typed with my
own hands." Those used to be the same thing.

What do the surviving thousand commands say?

After accounting for the losses, the survivors describe a narrower landscape than I
expected:

Measurement	Value
Event count	1,000
Multi-line commands	90
Distinct command names	90
Share of the top 10 commands	70.6%
Unique command texts	373
Entries eaten by repetition	627 (62.7%)
Command names used exactly once	39

The ranking: ssh 158, clear 126, bstart 82, cd 80, fstart 71, python 66,
yes 36, git 31, npm 30, sudo 26, seray 22, ping 22, ls 20, scp 20.
Ninety different tools, and a tenth of them cover roughly three quarters of the work.
Thirty-nine commands appear exactly once. All 36 yes lines are copies of a single text
— one load test's invocation — so even the list flatters the variety.

Three of them stand out: bstart 82, fstart 71, seray 22, all shortcuts from my
.zshrc. bstart enters a project directory, activates a virtual environment and starts
a development server; fstart only enters the front-end directory and calls
npm run dev; seray starts nothing at all, it just enters the directory and activates
the virtual environment. Together 175 entries, 17.5% of my history. Three words.

Those 175 lines teach me nothing. Six months from now the history cannot tell me which
command bstart ran, because it is not written there; and if the definition changed, it
will not recall the old one either. A shortcut gives back at reading time what it saved
at typing time.

clear appearing 126 times — exactly an eighth of the ledger — is its own joke: the
command for clearing the screen was recorded as the very noise it was clearing.

Repetition eats two thirds of the ceiling

Of the thousand entries only 373 are distinct text; the remaining 627 lines are
repeats. Think of it as a memory budget: the ceiling is fixed, and 62.7% of it goes to
rewriting something already there. A rough derivation — not measured, but derived from
two measured numbers: with the same distribution, dropping repeats would fit about
2.7 times today's variety into those thousand lines.

zsh offers four switches for this, and all four ship off:

HIST_IGNORE_DUPS — removes only an exact repeat of the previous event ("duplicates of the previous event," in the manual's phrasing). The weakest one.
HIST_IGNORE_ALL_DUPS — "if a new command line being added to the history list duplicates an older one, the older command is removed from the list (even if it is not the previous event)."
HIST_SAVE_NO_DUPS — "when writing out the history file, older commands that duplicate newer ones are omitted." Memory stays as is; only the disk copy is tidied.
HIST_EXPIRE_DUPS_FIRST — when room is needed, it causes "the oldest history event that has a duplicate to be lost before losing a unique event from the list." The documentation pairs this with keeping HISTSIZE larger than SAVEHIST; on macOS that cushion already exists, 2,000 against 1,000. Apple left the pillow and never turned on the switch that lies on it.

The ledger next door has been silent for five months

There is a second ledger in the same directory: ~/.bash_history. 7,207 bytes, 208
lines, 207 entries, 109 of them distinct. Last modified 8 May 2026; not a single line
added in five months.

Its contents belong to a different era: python 52, clear 38, cd 27, rm 11. And 28
lines begin directly with # — instead of deleting a command I had put a hash in front
of it and said "not right now."

That file records nothing any more, because zsh has been the default shell for new
accounts since macOS 10.15, as Apple's own documentation states. But it was never
deleted. Two hundred and seven lines of a frozen period sit there, and unlike their
neighbour they will never be trimmed. The condition for a complete memory, it seems, is
that the memory is no longer in use.

Before enlarging the memory: what is in it?

This file is plain text and it holds everything typed on the command line. Mine is
rw-------, readable only by me. Still, a crude scan was worth running: 23 of the
thousand entries contain one of the words token, secret, password or api key.
Not all 23 carry a real secret — matching a word is enough to be counted — but until the
scan I did not know that either, and asking the question correctly is the point. The good
news: four of them use read -rs, so the secret went into a silent prompt rather than the
command line.

Raising the ceiling also extends the half-life of that content. Three tools help:

HIST_IGNORE_SPACE removes lines whose first character is a space — but the manual adds an important caveat: "the command lingers in the internal history until the next command is entered before it vanishes," so the disappearance is not instant.
HISTORY_IGNORE takes a pattern and, "at the time history files are written… any potential history entry that matches the pattern is skipped." Far stronger than the leading-space trick for secret hygiene.
And a one-line rule: shell history is not an audit trail. Who ran what, and when, is not answered here; that needs a command-auditing pipeline such as auditd.

Choosing the default on purpose

There is a list of settings to change at the end of this, but the list is not the point.
Until today shell history looked like a record to me; what I had was a cache, and
someone else had decided its capacity, its ordering and its expiry.

Four questions for your own setup:

What is the ceiling? echo $SAVEHIST. If you see 1,000, that is your system's choice and not yours — roughly a month of memory for someone typing 30-40 commands a day.
Are timestamps kept? Without setopt EXTENDED_HISTORY there is no answer to "when did I do this," and the shell will show you today's clock instead of saying so.
When does the write happen? On exit, by default. INC_APPEND_HISTORY writes as soon as the command is entered; INC_APPEND_HISTORY_TIME writes after it finishes and records the duration; SHARE_HISTORY additionally imports other sessions' lines. The manual puts a clear warning here: "The three options should be considered mutually exclusive." Pick one; do not turn on all three.
Parallel shells? APPEND_HISTORY is on by default in zsh, so sessions do not overwrite each other. If the file lives on a network share, HIST_FCNTL_LOCK hands locking to the system's fcntl call — "where this method is available" — and the manual says this avoids history corruption on NFS.

There is a more radical route too: atuin, mcfly and hstr keep history in a database
instead of a flat file and remove all four complaints above — no timestamps, tight
ceiling, wrong order, crashed sessions lost — in one move. The price is one more
dependency in your shell.

As this article goes out, the defaults are still running on this machine, because
changing a setting is the job of a decision, not of a measurement. But I now know what
the setting is and what it costs, which I did not before the measurement.

Remembering Is Not a Default

Back to the original question: how far back does this file go? I cannot say. There are no
timestamps, so the age of the oldest line is unknown. The ceiling is full, so anything
before it is already deleted. Crashed sessions were never written. Commands run by the
agent and by scripts never entered. Four separate reasons, and none of them leaves a
record.

The real lesson is not about shell history. Most of the records in our lives work this
way: there is a ceiling, there is a moment of writing, and there is a default handed to
us without being asked. Until we open the ledger we assume it is boundless. Opening it
shows that the ledger reflects not what it holds, but what it was permitted to hold.

Every command typed while writing this article took one of the oldest lines of my history
with it. I will never know which ones.

Official Sources

I Left Sixty-Eight Backups; nginx Read Exactly One

Mustafa ERBAY — Fri, 02 Oct 2026 11:48:13 +0000

Before I touch a live configuration file, my hands type the same line on their
own: cp -p file file.bak-$(date +%Y%m%d-%H%M%S). An old reflex. If the change goes
wrong, the copy is right there, two keystrokes away. On the evening of 28 September, while reworking the blog's nginx
configuration, that line got typed again.

Four days later the copy was still there. Not merely there — inside the
directory nginx reads. Over those four days nginx restarted once
(30 September, 21:09) and reloaded its configuration seven times; every one of
those times it parsed that backup too, took its server block into memory, and
told me so, in sixteen lines of warnings. That nobody read those warnings for four days is an inference
from the backup still being there: once the "test is successful" line at the
bottom of nginx -t shows up, the job counts as done and the sixteen lines
above it go unread.

The backup did not take over the site. Nothing I did prevented it. This piece
is about what that "nothing" actually was: the safety of the habit comes not
from me but from two conventions I never chose.

First, the count: how widespread is the habit

Wondering about one file is no reason to look at only one file. A scan of /etc
down to three levels, for every name that smells like a backup:

sudo find /etc -maxdepth 3 \
  \( -name "*.bak" -o -name "*.bak-*" -o -name "*.old" -o -name "*.disabled" \
     -o -name "*.orig" -o -name "*.save" -o -name "*~" -o -name "*.pre-*" \
     -o -name "*.before-*" -o -name "*.next" \) -type f | wc -l

The answer: 68. Broken down by directory:

Directory	Copies	How does the system read this directory?
`/etc/nginx/sites-available`	16	Never read (link targets only)
`/etc/systemd/system`	10	Only recognised unit suffixes
`/etc` (root)	7	Files are called by exact name
`/etc/nginx/snippets`	4	Only via `include snippets/<name>`
`/etc/nginx/conf.d`	2	`include conf.d/*.conf` — suffix filtered
`/etc/nginx/sites-enabled`	1	`include sites-enabled/` — everything*

Those six rows account for 40 files. The remaining 28 are scattered two and
three at a time across /etc: application config directories, ssh, apt,
fail2ban — and five of them in the two directories already set aside for the
purpose, /etc/nginx/backups and /etc/nginx/disabled-backups. The blog's deploy
directory holds five more; those come later.

Those three system directories got a separate check, because the claim in the
title depends on them too: the line in sshd_config reads
Include /etc/ssh/sshd_config.d/*.conf, so it is suffix filtered, and apt and
fail2ban go by extension as well. In all three, my backups are not read. The
bare wildcard exists in exactly one place.

The table was a relief, and immediately afterwards an irritation. The relief:
67 of the 68 copies sit where the system does not care. The irritation comes
from the same place — which ones the system ignores was not something I knew
while dropping the copy. If sixty-seven of sixty-eight shots land, that is not
marksmanship; that is the shape of the distribution.

Which directory gets read, and which does not

What draws the line is not my attention. It is two lines inside nginx.conf:

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

The first imposes a suffix. The file
mustafaerbay-cache.conf.bak-20260928-191601 does not end in .conf, so it
never enters the conf.d sweep — my backup sits there like an invisible stone.
The second imposes nothing. A bare * makes everything in that directory part
of the configuration: .bak, .disabled, the ~ file an editor left behind,
all of it.

Debian's sites-available / sites-enabled pair exists precisely for this. The
real file lives in sites-available, and sites-enabled holds a symlink to it.
In that arrangement, a copy dropped "beside" the file lands automatically on the
sites-available side — the side nginx does not look at. The convention itself
is a guardrail.

On the server, sites-enabled holds 45 entries. Of those, 41 are symlinks and
4 are real files. The blog's configuration is one of those four. Its backup is
the second. So the copy went into one of the four places where the guardrail is
switched off, and that is not a surprise either — the reason a real file lives
there is also me. The live configuration now ships from the repository; the file
on the server is byte-for-byte identical to the 222-line version under
deploy/nginx-live/. Automation wrote a plain file where the distribution
expected a symlink. And then a copy went next to the plain file.

nginx did not pass over this in silence. Its nginx -t output carries sixteen
lines:

nginx: [warn] conflicting server name "mustafaerbay.com.tr" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "www.mustafaerbay.com" on [::]:443, ignored
...

Four hostnames (mustafaerbay.com.tr, mustafaerbay.com and the www form of
each) times four listening sockets (IPv4/IPv6 over 80/443) equals sixteen
warnings. Each says the same thing: you defined this name twice, and the second
one is being ignored.

So which one does it ignore?

The word "second" is the crux here, because the ordering is not mine. Rather
than touch production, the question went to a lab: nginx 1.29.8 in Docker, two
configuration files, the same server_name, the same listening socket. One
returns GERCEK-KONFIG, the other YEDEK-KONFIG. The files sit under conf.d,
but the include line was switched to the server's risky pattern —
include /etc/nginx/conf.d/*;, a bare wildcard. Otherwise conf.d's own suffix
filter would have cancelled the whole experiment.

In the first run the backup carries a suffix, as on the server: site and
site.bak-20260928.

# nginx -T load order
/etc/nginx/conf.d/site
/etc/nginx/conf.d/site.bak-20260928

$ curl -H 'Host: lab.example.com' ...   → GERCEK-KONFIG
$ curl -H 'Host: baska.example.com' ... → GERCEK-KONFIG

In the second run a single thing changed — the backup was renamed to bak-site.
Same content, same line count, same permissions. Only the name now sorts ahead
of the real one:

# nginx -T load order
/etc/nginx/conf.d/bak-site
/etc/nginx/conf.d/site

$ curl -H 'Host: lab.example.com' ...   → YEDEK-KONFIG
$ curl -H 'Host: baska.example.com' ... → YEDEK-KONFIG

The backup took over the site. It also answered the non-matching Host header,
because as the nginx documentation puts it, the default server is the first
block listed for that port, and "the default server is a property of the listen
port and not of the server name".

Those lab files are synthetic; their only job was to report which one answers.
The lab ran 1.29.8 while the server runs 1.30.0; nothing changed on the include
side — the glob call in the nginx source still reads the same on today's
master.
The backup on the server, by contrast, holds the 29 July version: 228 lines, 28
of them different from the live file. Had the name sorted the other way, a
two-month-old configuration would have been serving the site.

The ordering comes from POSIX. nginx expands the pattern with glob() and passes
zero as the second argument:

n = glob((char *) gl->pattern, 0, NULL, &gl->pglob);

Since GLOB_NOSORT is not set, sorting stays on. The glibc
manual on the server is
terse: "By default, the returned pathnames are sorted."
POSIX
says what that order actually is: "Ordinarily, glob() sorts the matching
pathnames according to the current setting of the LC_COLLATE category." So file
names decide which configuration serves the site — though "alphabetical" is not
quite the right word, since the server's locale sets the order.

This is where it became clear that luck was not the operative force, and that is
the part that nags: the file.bak-timestamp shape is structurally safe.
Because it appends a suffix rather than a prefix, the copy always falls behind
the original name; the string site is a prefix of site.bak-20260928, and the
shorter one sorts first — a relation that holds whatever the collation setting
is. Had the reflex settled on bak-file, 00-file or
old-file instead, this would be an incident report. The
safety of the habit comes from the direction of the naming, not from its purpose.

One habit, three other directories, three different verdicts

The interesting part is that the same cp reflex produces entirely different
outcomes elsewhere.

All three verdicts got measured.

The systemd unit directory. /etc/systemd/system holds ten in-place
backups: alert-monitor.service.bak-20260703-131021,
kopru-gateway.service.bak-20260730-115225,
vpsman-agent.service.bak-20260911-1550 and their siblings. systemd loads none
of them, because none carries a recognised unit suffix. Asked directly, it is
unambiguous:

$ systemctl status alert-monitor.service.bak-20260703-131021
Unit alert-monitor.service.bak-20260703-131021.service could not be found.

It appended .service to the name and found nothing. The list of loaded unit
files matches zero of these patterns. All ten copies are entirely harmless — and
entirely invisible. Anyone opening that directory has to read timestamps to work
out which file is live.

Drop-in directories. Here the rule inverts. In the words of the
systemd.unit(5) documentation, files with the .conf suffix "will be merged in
the alphanumeric order and parsed after the main unit file itself has been
parsed", and multiple drop-ins with different names "are applied in lexicographic
order". Whatever is applied later overrides the same setting from earlier. In
nginx the first wins; here the last wins. Two directories, two opposite rules,
one habit.

Two units on the server have two drop-ins each. Under kopru-web.service.d sit
20-live-readiness.conf and order-key.conf, and systemctl cat prints them in
that order. They do not collide, because each defines a differently named
credential and that setting is the kind that appends to a list. But that is
luck: had both written the same single-valued setting, order-key.conf would
have won, and my assumption that the numbered file takes precedence because of
its number would have survived untested.

cron.d. Here the trap runs in reverse. Debian's cron(8) manual states that
names in /etc/cron.d "must consist solely of upper- and lower-case letters,
digits, underscores, and hyphens"; a file containing dots is ignored. Copy a
cron.d job to job.bak, delete the original, and what remains is a file that
never runs and never complains. My own cron.d holds ten entries: nine ordinary
jobs and a tenth called .placeholder — a file silently ignored exactly as that
rule intends.

The real cost of a copy is not being read

Three of the seven copies at the root of /etc are, as their names make plain,
backups of files that carry credentials: one access-token file and two env
files. Their permissions checked out — all three are -rw------- and
root:root, same as the originals. There is no exposure here.

The risk sits elsewhere: these copies freeze the old value of a token. Rotate the
token and the copy does not rotate with it; what remains inside is either a
string that no longer works or — worse — one that still does, because it was
never revoked. Sitting in /etc as a date-stamped file, in no inventory.

On all three, the name and the timestamp disagree. Two of them carry 20260914
in the name while their content was last written on 25 June — eighty-one days
apart; on the third the gap is twenty-one days. The copies were made in a way
that preserves timestamps, so the name records when the copy was taken and the
timestamp records when the content was written.
Anyone reading that directory without knowing the convention trusts the wrong
file.

The alarm that rings about the wrong thing

While counting backups, the blog's own working tree came under the same look,
because the live site is served from it: the blog-serve container mounts
/opt/mustafaerbay read-only as /app and runs dist/server/entry.mjs from
inside it. In that directory git status reports six modified tracked files and
a HEAD stuck on 16 May. Today is 2 October; 139 days sit in between.

On first read that looked alarming. A four-and-a-half-month-old HEAD on a live
tree, six dirty files — as though edits nobody knows about had piled up on the
server. Then each file went head to head with today's origin/main:

File reported as "modified"	Against today's `origin/main`
`deploy/backup.sh`	Identical (84 lines)
`scripts/notify-subscribers.mjs`	Identical (278 lines)
`package.json`	Identical
`package-lock.json`	Identical
`deploy/pipeline-health.sh`	One line differs (line 207)
`deploy/update.sh`	Not in the repository at all

So four of the six files git status calls "modified" have not changed at all;
they merely look different against an old HEAD. The dirt is not drift, it is a
stale reference. The alarm rings, but points at the wrong thing — and because it
has pointed at the wrong thing for four and a half months, the habit of looking
at it is long gone.

The one-line difference turned out to be a familiar face. The health script on
the server still prints the repository's old address in its alert emails:
github.com/mustafa_itwise/mustafaerbay. After the repository moved to the
organisation on 28 September, three scripts still calling the old name got
cleaned up that day;
this is the fourth. It was inside my own alerting machinery — in the very line
meant to warn me when something breaks.

The debt that stays quiet

The last row of that table is the real finding. deploy/update.sh was deleted
from the repository on 26 June, as part of the incremental-build work. The copy
on the server was last modified on 14 September. A script that does not exist
in the repository was touched 80 days after its deletion, and the copy beside it
carries that same day's migrate tag in its name. I am the only person who does
that work on that server. It has three
siblings: update.sh.bak-1780461942 and update.sh.bak-20260914-migrate (both
3 June, 07:45) and update.sh.disabled (11 May).

Worse, a unit file still calls it:

Description=mustafaerbay.com deploy pull job
ExecStart=/bin/bash /opt/mustafaerbay/deploy/update.sh

The good news: the timer is off. For mustafaerbay-deploy.timer, is-enabled
answers disabled; the service is static and inactive, with not one journal
line since 26 June. The bad news: that timer's content reads
OnUnitActiveSec=1min. A single systemctl enable --now would start running,
once a minute, a script no repository knows about. The gun is not loaded, but it
sits next to the trigger with no label saying which repository, which version.

The same directory also holds an 830 MB dist-persistent: 13,445 files, the
newest dated 29 June. Leftovers from a build experiment that never merged.
Next to it, dist-old was refreshed today — that one is not a backup but the
rollback copy the deployment rotates on every run. The two sit side by side, one
part of a live mechanism, one a ninety-five-day-old fossil. In a directory
listing they are indistinguishable.

Where the copy belongs

After the measurements, the rule got sharper, and all of it hangs on one
question: does anything read this directory, and by what criterion?

If the directory is read with a bare wildcard (*), a copy goes there never. For the blog's configuration the right place is sites-available or a separate backup directory. The server already has /etc/nginx/backups and /etc/nginx/disabled-backups — past me did the right thing, then forgot.
If it is read with a suffix filter (*.conf), the copy is safe but invisible. Invisibility is not free: three months later you work out which file is live from its timestamp.
Know the direction of the sort. In nginx a conflicting name goes to whichever loads first — that one is not in the documentation; it is behaviour measured in a lab and in the source. In systemd drop-ins it goes to whatever applies last, and that one is documented. Naming a file 00- moves it to the front in one system and to the back in the other.
Use a suffix, not a prefix. file.bak-timestamp falls behind the original name; bak-file jumps ahead of it. That single structural detail is what makes the habit safe.
If configuration ships from a repository, break the symlink convention deliberately, not by accident. A deploy script that writes a plain file also switches off the protection the distribution handed you.
Build the detection too, because nginx -t returns zero alongside those sixteen warnings — measured in the lab. No deployment gate will ever see them. Three checks are enough: treat warnings as fatal with nginx -t 2>&1 | grep -q "\[warn\]", read the real load order with nginx -T | grep "# configuration file", and count the non-symlinks with find /etc/nginx/sites-enabled -maxdepth 1 -type f (mine returned four).
If you leave a backup of a file that carries credentials, add that backup to the rotation list too. Even a copy with correct permissions can keep an unrevoked old secret alive indefinitely.
For a git working copy sitting on a live tree, either keep HEAD current or stop believing git status in that directory. There is no middle ground: a stale HEAD manufactures fake debt daily, and hides the real debt inside that noise.

Conclusion

Of sixty-eight copies only one mattered, and what kept that one from doing harm
were two conventions I had never thought about: a naming habit that grows by
suffix, and Debian's symlink farm. Neither is my design. One arrived through
muscle memory, the other comes bundled with the distribution.

The finding that stings most is the warning lines. For four days nginx said, in
sixteen lines, "you defined this name twice"; those lines got skipped, and the
one at the bottom — "test is successful" — was taken as the end of the job. The git status output that shows six dirty files has been lying
every day for four and a half months, so I stopped looking at it. An alarm that
rings about the wrong thing is more dangerous than one that stays silent,
because it trains you against your own instrument. The day half-finished work
got counted four ways and
produced four different answers
taught the same lesson; apparently learning it once is not enough.

The copy left inside that directory was not a safety net. It was part of the
configuration. What separated the two was the alphabetical order of its name.

Official Sources

I Tested ispmanager in Three Rounds: One Leak Fixed, Six Findings Retracted

Mustafa ERBAY — Fri, 02 Oct 2026 09:39:18 +0000

Disclosure: ispmanager provided a free test licence for this review; I was not paid. In our correspondence they also mentioned a free month of use and a "longer-term partnership" that could include free ongoing use of the panel; I deferred all of that until after testing and accepted none of it. The vendor has no control over the test scope, the findings, the severity ratings or this text. The text was sent to them for a technical accuracy check only: on 18 September, and in its post-second-round form on 21 September. The ispmanager links in this article exist so readers can reach the documentation; nothing was received in return for them.

On 10 September an email arrived. Aleksandra from ispmanager had been reading the blog; they wanted to grow in the Turkish market and were looking for an honest review from the perspective of someone who normally wouldn't need a panel. My relationship with control panels has never been good. On my own servers I prefer the terminal, because when something breaks I know exactly whom to ask: myself.

That is also what made the offer interesting. The question was never "is the UI easy?" It was what the panel takes on, what it keeps out of sight, and whether it tells the operator the truth when something breaks. In my first reply I set a single condition: independence. I would test in my own environment and reach my own conclusions, and the final word on both positive and negative findings would be mine. They agreed.

Three weeks later I have 23 findings, three rounds of testing and an outcome I did not expect. The vendor first rejected the most serious finding, accepted it after the evidence from my second round, and closed it in version 6.153; on 1 October I verified that fix in my own lab. I, in turn, retracted the substance of six of my own findings, because when I measured again, I turned out to be wrong. This article covers both. In my view, the value of a review is not in its number of findings but in how those findings change when new evidence arrives.

Three rounds, one environment

The test environment was not a production server, and I should say so up front: Docker Desktop on a Mac, running a privileged AlmaLinux 9.8 container booted with systemd. Because ispmanager's AlmaLinux repository ships only x86_64 packages, it ran under x86_64 emulation on Apple Silicon.

Round	Date	Version	Focus
1	14 September	Lite 6.150.4 (core 5.445.2)	13 areas, 23 findings, 15-page report
2	21 September	Lite 6.150.4, clean install	Measuring the vendor's objections
3	1 October	Lite 6.153.0 (core 5.448.0), clean install; stable 6.152.2 for comparison	The SEC-02 fix + re-measuring the findings

The first round used the test licence the vendor sent. In the second and third rounds that licence's activation key was no longer accepted, so I used the trial licence the installer issues automatically to this IP. Round two installed the same packages as round one; round three installed 6.153.0 with the current installer script from the vendor's server.

The method was the same every time. I exercised each feature through the panel's API first, then verified it on the operating system: the filesystem, the process table, iptables, systemd, the generated configuration files. On top of that I added hostile scenarios: symlink escapes, cross-tenant read attempts, killed services, a broken nginx configuration, changes made behind the panel's back.

Running in a container produced three side effects, which I separated out from the start: IP management expects an ifcfg-eth0 file, panel login depends on the sshd PAM profile, and MariaDB fails on first start. This round I finally found the cause of the last one. I'll come back to it in a minute, because it is small but instructive.

What the panel takes ownership of

This was the most important question for me, and the answer was more mature than I expected. The installed stack is a classic split: nginx terminates TLS and serves static content, and PHP runs either under MPM-ITK, one of Apache's third-party MPM modules, with mod_php under a separate UID per site, or in per-site PHP-FPM pools over a Unix socket. The panel itself lives in a separate HTTP server, ihttpd, on port 1500.

Configuration is generated from templates, and the distribution's own nginx.conf only gets include lines added; the file is not overwritten. Every site and user has vhosts-resources/<site>/ and users-resources/<user>/ directories; that is the way to extend things without breaking the panel. In the first round I added a line by hand to a vhost file the panel had generated, and after the panel rewrote the file the line was still there. A good sign for anyone who lives in fear of "don't touch it, the panel will overwrite it".

What held up

This list is long, and all of it was measured. nginx returned 403 for a web-reachable symlink pointing at /etc; the panel writes disable_symlinks if_not_owner into the vhosts. Apache MPM-ITK really does run each site under its owner's UID; the test script under mod_php ran as uid=1010 (usera). A database user created through the panel could create and write tables in its own database and got ERROR 1044 on its neighbour's; I repeated that on 6.153 on 1 October as well.

An archive taken with backup2 contained files with their ownership preserved, a valid mysqldump and metadata. I deleted the site's files and restored them from the archive, and the site returned 200 again. Self-signed certificate generation, assignment to a site and the HTTP-to-HTTPS 301 redirect all worked. I injected a broken nginx configuration: the reload was refused and the running workers were left alone. The history_users table records who did what, from which IP, and which fields changed, with a timestamp. Firewall rules added through the panel show up as commented, persistent iptables entries and are removed cleanly. During testing my own IP got caught by the brute-force lockout, so that works too.

Round three added two items to this list, and both came from places where I had been wrong: the panel's own metadata does go into the full backup, and backups can be managed through the API on Lite. I'll come back to both below.

SEC-02: three rounds of one password

This turned out to be the longest-lived finding of the review, and in my view the most instructive.

Round one. You register the database server in the panel with the root user and a password. Then you call the func=db.server list: the response carries the password you saved, in plain text, in the password= field. I rated it High in the report and wrote "including reseller accounts" under impact.

The vendor's objection (18 September, in writing). The engineering team assessed it as "by design". The reasoning: db.server.edit is available only to the superadmin, the superadmin is equivalent to root on the server, so seeing the password gives them no new capability. The last sentence of their reply left the door open: if I could show a scenario in which someone other than the superadmin obtains the credential, they would reassess.

That reply changed my initial assessment. I withdrew the reseller claim, because I had not measured it. But nobody had measured the "superadmin only" claim either.

Round two (21 September). I created an ordinary administrator with admin.edit, level=29. The superadmin is level=30. This account stayed within the limits the panel documents: the file manager, the shell, creating administrators and db.server.edit were all denied. But it could call the db.server list, and the list carried the password:

authinfo=opadmin:...&func=db.server
id=1 name=MySQL type=mysql host=127.0.0.1 username=root password=MgrRoot#2026 savedver=10.5.29

authinfo=opadmin:...&func=file
ERROR access(function): You have insufficient permissions to execute the function 'file'

(The password is a lab-only test value.) The table in the vendor's account levels documentation shows "DBMS management", "File manager", "Command execution" and "Shell client" as unavailable to the administrator. Everything that was denied matched the documentation exactly; the password in the db.server list, however, leaked out beneath the documented boundary. And it was not a placeholder value: what the panel had stored was the real password of the MariaDB root account, in other words the key to every database on the server. I sent this to the vendor on 21 September, together with the reproduction steps.

The whole weight of the threat-model debate rested on a single detail. The vendor had looked at the db.server.edit form; the password was being returned by the db.server list. Four letters separate the two function names, and one role separates their privilege levels.

The vendor's reply (23 September). It was clear and direct: "You are correct: our previous response referred to the wrong API call." They reproduced the scenario in their own environment. They didn't hide their own perspective either: in their model the administrator is a trusted, high-privilege role, and as a rule they don't hide from an administrator what an administrator can see. But they agreed that a stored root password surfacing for a role that is denied the file manager, the shell and administrator management sits below that line, and wrote that they would remove the password from the db.server response, targeting version 6.153, due on 29 September.

I don't see this often. The vendor put forward a claim in their first reply, I measured it, the measurement refuted it, and the vendor revised its initial assessment in writing. To me that is more valuable than an "I found a vulnerability" story: a disagreement was settled by measurement.

Round three (1 October). According to ispmanager's changelog, 6.153.0 shipped a day late, on 30 September, in the beta channel; the stable channel moved to 6.152.2 the same day and is still there as of 2 October. I passed --release 6.153 to the installer, installed a clean 6.153.0 and repeated the second-round scenario exactly. Same administrator role, same call, three output formats:

# opadmin (level 29), 6.153.0
out=text      id=1 name=MySQL type=mysql remote_access=off host=localhost hostandport=127.0.0.1:3306 savedver=10.5.29
out=JSONdata  [{"id":"1","name":"MySQL","type":"mysql","remote_access":"off","host":"localhost","hostandport":"127.0.0.1:3306","savedver":"10.5.29"}]
out=xml       <elem><id>1</id><name>MySQL</name>...<savedver>10.5.29</savedver></elem>

# root (level 30), 6.153.0
out=text      id=1 name=MySQL ... username=root password=MgrRoot#2026 savedver=10.5.29

The administrator's response no longer contains a password field, nor a username field, in any of the three formats: text, XML and JSONdata. The superadmin's list still carries the password. That is consistent with the threat model the vendor defended from the start: the superadmin is root anyway. If it were up to me, I would mask the password on read paths there too, but that is a defence-in-depth preference, not a boundary violation.

I also checked that the fix did not break anything else. The administrator is still denied db.server.edit, the file manager, the shell, creating administrators and the scheduler. A user-level account cannot call db.server at all. The administrator can create a database on behalf of a user, and the list keeps working without exposing the stored credentials. The test password does not appear anywhere in the panel's log files under /usr/local/mgr5/var. One detail remains: although the documentation table shows "DBMS management" as unavailable to the administrator, the administrator can still call the db.server list, now without the password. Documentation and behaviour still don't fully match, but no secret leaks any more.

The detail that matters most for readers: for now the fix exists only in the beta channel. I ran the same scenario in a second container on 6.152.2, which is what the installer fetches with --release stable. The administrator account still receives the password in plain text:

# opadmin (level 29), 6.152.2 (stable)
out=text  id=1 name=MySQL ... username=root password=MgrRoot#2026 savedver=10.5.29

The vendor says it updates the beta channel every two weeks and the stable channel once a month; it hasn't yet said when 6.153 will reach stable. Until then, anyone on stable has two options: move to 6.153 in the beta channel, or create no administrator accounts other than the superadmin. If level-29 administrator accounts could reach the database server list before 6.153, the password may have been exposed to them; change the MariaDB root password after upgrading as well. The upgrade itself doesn't require this; the reason is that exposure window. How to do it is in the day-one list below, because the wrong method breaks the panel.

One more note. The 6.153.0 changelog entry has three "fixed a security issue" lines; two are rated low and one medium, and none of them is named. I don't know which one is SEC-02, and I won't guess. What I do know is that the behaviour I measured has changed.

SEC-01: open_basedir still does not carry over to FPM

When you create a site with the default settings (mod_php), open_basedir is set correctly: limited to /var/www/<user>/data:., and a test script cannot read /etc/passwd. Switch the same site to the PHP-FPM mode the panel supports, call the same script, and the restriction is gone. The panel keeps showing basedir=on; the generated pool file has no open_basedir, the site-level include file is zero bytes, and the user-level one only carries upload_tmp_dir and session.save_path.

The vendor reproduced this in their first reply, agreed that PHP modes should behave consistently, and put a fix for a future release on their backlog. But they disputed the severity: in the multi-tenant model the real boundary is at the operating-system level, each pool runs under its own system user, so one tenant's script cannot read another tenant's files even without open_basedir.

That objection was measurable, and I had not measured it in the first report. /etc/passwd is world-readable anyway. In round two I tried a direct cross-tenant read, and on 1 October I repeated it on 6.153. User A's FPM site tries to read a file on user B's site:

# 6.153.0, a.local (PHP-FPM, isp-php84)
SAPI=fpm-fcgi user=usera uid=1010 open_basedir=[]
/etc/passwd                                 => OK 1796 bytes
/var/www/userb/data/www/b.local/secret.txt  => FAIL (Permission denied)
/usr/local/mgr5/etc/ispmgr.conf             => FAIL (Permission denied)
/etc/shadow                                 => FAIL (Permission denied)

# 6.153.0, a2.local (mod_php), same user
SAPI=apache2handler user=usera uid=1010 open_basedir=[/var/www/usera/data:.]
/etc/passwd                                 => FAIL (Operation not permitted)

The result goes the vendor's way. The mechanism is worth learning, too. B's data directory is drwx---r-x userb:mgrsecure: "others" can read and traverse it, which is what nginx and Apache need to serve static files. But the mgrsecure group has no rights at all, and the panel adds every tenant to that group. On Linux, permission checking picks the first matching class out of owner, group and others and applies only that class's bits; once the group matches, "others" is never consulted. Tenants cannot enter each other's directories; service users can. A clever layout; I had missed it in the first report.

That is why I lowered SEC-01 from High to Medium in round two, and that is where it stays. A defence layer that the panel shows as enabled silently disappearing on a supported mode switch is a real defect; world-readable system files are exposed. But the tenant boundary held in every measurement. The 6.153 changelog has no line about this, and the behaviour has not changed.

Why I retracted six of my findings

In round three I also re-measured the findings other than SEC-02, or opened and read their documentation. I did not re-measure LIF-01 (uninstall) or LIF-02 (resources outside the panel); for API-02 and PLT-02 I only checked the documentation.

I also need to say one thing plainly. On 23 September the vendor wrote that they had sent feedback on the whole file, and according to my working notes that feedback included design explanations for ROB-04, FUN-04, FUN-05 and API-02. I verified everything below by measuring it myself or opening the documentation, not by trusting those explanations. But for some of these retractions, it was the vendor who pointed me in the right direction.

ROB-03 (monitoring). In the first report I wrote "the panel monitors services but does not bring a failed one back". In round two I stopped nginx and waited: the panel's srvmon monitoring job brought it back with systemctl restart nginx.service after 9 minutes 47 seconds. I had waited only 20 seconds before passing judgement. On 1 October I repeated it on 6.153: 11 minutes 58 seconds. In 6.153's crontab the srvmon job runs every 15 minutes (*/15), and in both measurements the restart landed on the hour. So on this install a failed service can stay down for close to a quarter of an hour. In round two the panel's service list showed nginx as started throughout; on 6.153 it correctly showed stopped from start to finish, which I note as an improvement. One more confession: in this round's first measurement nginx came back after one minute. The cause was not monitoring; another test I was running at the same time had made the panel rewrite a site configuration. I threw that measurement away and repeated the test without touching the machine.

ROB-04 (panel metadata not in the backup). Wrong. On 1 October I ran backup2 -m full and opened the root account's archive. It contains /usr/local/mgr5/etc/ispmgr.db, core_core.db, their -wal and -shm files and ispmgr.conf; 2,203 files in total. In the first round I had looked at the user archives and never opened root's.

FUN-02 (backups cannot be managed via the API). Wrong, but in an interesting way. In the first round I called the backup.plan and backup.storages functions, got module is missing, and wrote "on Lite, backup management is effectively CLI-only". In round three a banner inside one of the responses pointed at a backup2.schedule function. I tried it: backup2.settings, backup2.schedule and backup2.list work. The API reference lists both families; the backup.plan family is absent on Lite, the backup2 family is present. What remains is a documentation clarity note: both families are documented side by side, and the docs don't say which works in which edition.

FUN-03 (no WAF on Lite). Wrong. The WAF is an optional component installed from the Software configuration screen; on 6.153 the form shows a package_nginx_modsecurity=off option. I had seen that it was not installed and concluded that it did not exist.

FUN-04 (no supervision for Node.js apps). Wrong. I had looked for a systemd unit, didn't find one, and wrote "no mechanism that restarts the app after a crash was visible". The panel manages Node.js and Python applications with PM2; /usr/lib/ispnodejs/bin/pm2 is installed, the documentation gives that same path for diagnostics, and the 6.152 changelog contains a fix related to the PM2 interpreter. I had been looking for the wrong tool. I did not separately measure a restart after a crash; this retraction rests on the installed tool and the documentation.

FUN-05 (the system user cannot log in over FTP). Wrong. The FTP server documentation describes virtual FTP users as the design itself: FTP access without creating real system accounts. I had also noted "the docs say password, the API wants passwd"; the API reference says passwd too, so that note was wrong as well.

In three places I corrected a finding without retracting it. In PLT-01 (no ARM packages) the behaviour is unchanged; the aarch64 path in the 6.153 repository also returns 404. But the system requirements page says "use the server version of an operating system with the x64 architecture". I had written up a documented limitation as a Medium defect; I downgraded it to an informational note. In FUN-01 (Python does not work) I had given "Phusion Passenger is not installed" as the cause; in ispmanager 6, Python runs under PM2, and the Python component is disabled in a default install. The remaining defect lies elsewhere, below. In SEC-03 I had written that AlmaLinux 9's default password scheme is "yescrypt"; the container's /etc/login.defs says ENCRYPT_METHOD SHA512. The finding stands; the comparison was wrong.

There is no point hiding any of this. If I value the vendor revising its initial assessment of SEC-02, I have to apply the same standard to myself. The common denominator is clear, too: everywhere I was wrong, I had either not waited long enough or drawn a conclusion without opening the documentation. In the first report my numbers were confident; my evidence was not as strong.

What is still open on 6.153

I reproduced every remaining medium-severity finding on 6.153.0 on 1 October. None of them has changed.

SEC-03. Users created through the panel are written to /etc/shadow with the $1$ prefix, that is, md5crypt. The libxcrypt documentation is explicit about this scheme: MD5 is so cheap on modern hardware that it should not be used for new hashes, and its processing cost is not adjustable. On the same machine login.defs says SHA512; the panel bypasses the system default.

SEC-04. MariaDB listens on *:3306, and the single multiport rule in the default firewall opens 20:22,25,80,443,110,143,465,587,993,995,53,3306,5432,1500 to every address. The two anonymous accounts the install leaves behind (''@localhost and ''@<hostname>) are still in the table. Since every named account is @localhost, remote login should not work at the moment. I did not try it. But the day the first @% account is created, nothing is left at the door.

SEC-05. The panel's root login calls PAM with the sshd service name. On 1 October I temporarily removed /etc/pam.d/sshd, and an API call with the correct password returned ERROR auth(badpassword); once the file was back, the same call worked. Every hardening change to sshd's PAM stack, an MFA module or a deny rule for instance, also affects whether the panel can be reached.

SEC-06. This round I measured it with a control group. You send secure=on for a site owned by a user whose SSL limit is off, the API returns OK, and when you read the site back it says secure=off and nginx has no 443 line. Turn the limit on for the same user and send the request: because I passed the certificate name in the wrong format, I first got an explicit validation error (ERROR value(ssl_cert)), so with the limit on, the panel does tell you what's wrong. With the correct name: secure=on, port 443 is listening, and HTTPS returns 200. I turned the limit off and sent the same correct parameters again: OK again, secure=off again. The operator believes HTTPS is on.

What is left of FUN-01 belongs to the same family. With the Python component not installed, sending python=on for a site still returns OK, and the site stays without Python. I lowered it to Low, because this time it is not a security setting; but the pattern is the same.

ROB-01. The panel connects to MariaDB through root@localhost for its own database operations. After a routine hardening step, ALTER USER 'root'@'localhost' IDENTIFIED BY '...', I tried to create a database: ERROR db(connect): ... Access denied for user 'root'@'localhost'. There is no health indicator and no startup check. Reverting fixes it. There is nothing wrong with unix_socket authentication itself; the problem is that the panel never tells the operator about this dependency.

ROB-02. Here is the small story I promised. In all three rounds MariaDB failed during installation. A .cache subdirectory appears in /var/lib/mysql, and mariadb-prepare-db-dir refuses to initialise a non-empty directory. This round I looked inside: .cache/rosetta. The mysql user's home directory is /var/lib/mysql. Under emulation, running any process as a user creates .cache/rosetta in that user's home directory; I reproduced this with a separate test user by running runuser -u rtest -- /bin/true. So the trigger is my environment; you will not see this on a real x86 server.

The product finding is not the trigger but what happens next. The 6.153 install log has Error in POSTIN scriptlet in rpm package coremanager-pkg-mysql, three Job for mariadb.service failed lines, and then seven The server wasn't added. Retrying... lines, backing off from 1 to 64 seconds. The installation still ends with EXIT=0 and the message "Your newly installed ispmanager panel is available". What's left is a panel with no database server registered, and the operator finds out when they try to create their first database and get notconfigured(nodbserver). The retry loop in the coremanager-pkg-mysql package's post-install script and a "pending registration" marker file show that the vendor has been looking at this class of problem; the 6.152.0 changelog also has a fix for the default database server not appearing in the list after installation. The same thing happened in the stable 6.152.2 container. Failure is still reported as success.

The full table

ID	Initial rating	As of 1 October	Note
SEC-01	High	Medium, open	No `open_basedir` under FPM; tenant boundary holds
SEC-02	High	Fixed (6.153.0)	No password in the admin response; independently verified
SEC-03	Medium	Medium, open	md5crypt ( $1$ )
SEC-04	Medium	Medium, open	`*:3306`, world-open rule, anonymous accounts
SEC-05	Medium	Medium, open	Panel login tied to the `sshd` PAM profile
SEC-06	Medium	Medium, open	SSL request swallowed with `OK`
ROB-01	Medium	Medium, open	Panel silently breaks when the root credential changes
ROB-02	Medium	Medium, open	Install "succeeds" without a working DB
FUN-01	Medium	Low	`python=on` → `OK` without the Python component
FUN-02	Medium	Info	`backup2` API exists; docs mix two families
PLT-01	Medium	Info	x64 is a documented requirement
ROB-03	Low	Info	Recovery exists: `srvmon`, `*/15` on 6.153 (measured 11 min 58 s)
ROB-04	Low	Retracted	The root backup includes the panel databases
ROB-05	Low	Low, open	`restore2` date and file options
FUN-03	Low	Retracted	The WAF is an optional component
FUN-04	Low	Retracted	PM2 supervision
FUN-05	Low	Retracted	Virtual FTP users are by design
API-01	Low	Info	Docs say `elid`; `name=` is ignored without an error
LIF-01	Low	Low (not re-measured)	Leftovers after uninstall
API-02	Info	Info	Password in the URL; a session-key alternative is documented
LIF-02	Info	Info (not re-measured)	Resources created outside the panel don't appear in its lists
PLT-02	Info	Info	The installer requires SELinux to be disabled (documented)
OBS-01	Info	Info	13 MB `ispmgr.log` in ~16 min, including install and two test runs

In numbers: as of 1 October, 6.153.0 has 7 Medium, 3 Low and 8 informational items open. One finding has been fixed and four have been fully retracted. When I say I retracted the substance of six findings, I mean those four plus ROB-03 and FUN-02, whose claims collapsed and which were reduced to informational notes. PLT-01 also became an informational note, but I didn't retract it: the behaviour I measured was real, it was just documented. On 6.153 there is no open High-severity finding left; on stable 6.152.2, SEC-02 is still open.

Two rows need more explanation. For API-01 I sent name=nginx to services.restart; the panel returned no error and did not touch nginx. It worked with elid=nginx. The API reference already says elid, so this is less a defect than a note about an API that silently swallows parameters it doesn't recognise. For API-02, carrying credentials in the URL via authinfo= is protected in transit by TLS. The panel masks the value as authinfo=* in its own access log, but the same is not guaranteed for the log of a proxy in between; however, the API guide also describes a session ID valid for one hour, one-time key login and an IP whitelist for authinfo. The choice is the operator's.

Two themes

Put the remaining findings side by side and most of them cluster in two places.

The first is silent failure semantics. A limit rejection is swallowed (SEC-06, FUN-01). The installer "succeeds" with a broken database (ROB-02). The panel silently breaks when the root credential changes (ROB-01). A configured defence layer disappears on a supported mode switch while the UI still shows it as enabled (SEC-01). The product tells the truth, but not loudly enough. On a security-related setting, "OK, but not applied" is the worst answer there is.

The second is a tight coupling to the host's identity infrastructure: the sshd profile in PAM, the root credential in the database, and SELinux, which the installer requires to be disabled. The panel does not carry its own identity; it leans on the host's. Neither requires an architectural rewrite; both call for a clearer definition of the ownership boundary and of the failure contract.

If you are installing ispmanager: a day-one list

I am writing these based on the behaviour I measured on 6.153.0. They are worth re-checking as versions change.

Version. If you are going to give administrators panel accounts, install 6.153.0 or later, or hold off on administrator accounts until you can. As of 2 October, 6.153 is in the beta channel.
Database port. Check the listening address with ss -ltnp | grep 3306. If you don't need remote access, remove 3306 and 5432 from the firewall's multiport rule or narrow the source, and pull the daemon back to localhost with bind-address.
Anonymous accounts. Delete the rows with an empty user name from SELECT user,host FROM mysql.user.
FPM sites. On every site you switch to PHP-FPM, check ini_get('open_basedir') with a small script. If it is empty, there is no restriction, whatever the panel shows.
Whether the install really finished. After the installer finishes, verify that mgrctl -m ispmgr db.server is not empty. The "success" message does not guarantee it.
Credential changes. Before you change the MariaDB root credential or touch sshd's PAM stack, write down in your runbook that the panel depends on them, and try a panel operation after the change.
Panel port. The same multiport rule also opens 1500, the panel itself, to every address. If the panel should only be reachable from a management network, narrow the source.
The password after moving to 6.153. If level-29 administrator accounts could reach the database server list before 6.153, they may have read the stored root password; change the MariaDB root password. The upgrade itself doesn't require this. Keep unix_socket while doing it (ALTER USER 'root'@'localhost' IDENTIFIED VIA unix_socket OR mysql_native_password USING PASSWORD('...')), then save the new password in the panel with db.server.edit. I tried this sequence on 6.153 and the panel kept working; a plain IDENTIFIED BY triggers ROB-01.
Backups. Open the root archive of a full backup yourself once and see that it contains the panel databases. I wrote it up wrong without looking.

What the vendor did

I sent the report on the night of 14 September. On 15 September the reply came: "we've shared it with the team, we'd love a full article". On 18 September the engineering team's written assessment arrived: they reproduced SEC-01 and put it on the backlog, and objected to SEC-02 on threat-model grounds. For the remaining findings the summary was: some make good sense and will be acted on, in a smaller number the intended design wasn't fully accounted for in the report, nothing is urgent, everything goes into the backlog. On 23 September they revised their assessment of SEC-02 and gave a fix date. On 1 October Aleksandra asked whether I had run the test, when the article would be out, and, with a smiley, whether there would be "at least one link".

In round three I saw how right that "intended design wasn't fully accounted for" sentence was: FTP, the WAF, PM2 and the backup API fell squarely into that category. It made me think again that a good vendor response is not saying "you're right" to every finding. A good response reproduces, concedes where it agrees, and gives a testable justification where it doesn't. ispmanager did that. When I tested the testable justification, it turned out to be wrong in one place, and they accepted that. The answer to the link question is in this article too: I linked their documentation generously, because readers should be able to check my claims on the vendor's own pages.

Who it is for, and what I would do

ispmanager's natural territory is web hosting, reseller setups, and servers that are run by several operators or handed over to customers. The panel makes sense wherever someone who doesn't know every corner of Linux has to do routine work from a safe surface. Tenant isolation, template discipline and backup integrity are solid enough for that. For anyone on ARM infrastructure the door is closed, and documented as such; for those who manage configuration heavily outside the panel, LIF-02 is a nuisance. For anyone thinking of building their own PaaS layer instead of a classic hosting panel, what I wrote about Coolify describes a different trade-off.

My default is the terminal, and that hasn't changed. But this test changed my view of where a panel creates value. In a setup where I am not the only operator, and want to hand part of the work to someone who doesn't go deep into the system, I would take ispmanager seriously. That is not the answer I would have given before I started. My conditions are clear too: 6.153 or later, the day-one list above, and checking open_basedir on FPM sites myself until SEC-01 is closed.

Closing

I tested ispmanager Lite at its edges, not on the happy path. Template discipline, OS-level isolation, backup integrity, the TLS lifecycle, the audit trail and atomic reloads genuinely held up. Of the two High findings, one was lowered to Medium by measurement and the other was closed in 6.153, which I verified independently; the fix has not reached the stable channel yet. Seven medium-severity findings remain open on 6.153, and most of them are symptoms of the same habit: failing silently.

Both sides changed their minds during this process. The vendor revised its initial threat-model assessment of SEC-02 after reproducing the privilege-boundary case, and decided to fix the behaviour. I lowered SEC-01's severity, deleted ROB-03's "no recovery" sentence and retracted the substance of six of my findings. Two days ago I wrote about how the patch that closed twenty CodeQL findings turned out to be the source of the twenty-first. This week I learned the same lesson from a different direction. A control panel should make your work easier on a good day and tell you plainly what happened on a bad one. The same standard applies to a review.

Environment: ispmanager Lite 6.150.4 (core 5.445.2) and 6.153.0 (core 5.448.0), AlmaLinux 9.8, a --privileged systemd container on Docker Desktop, x86_64 emulation on Apple Silicon. Round 1: 14 September, vendor test licence. Round 2 (21 September) and Round 3 (1 October): IP-bound trial licence, clean install; in Round 3, 6.152.2 (core 5.447.2) in a separate container for the stable-channel comparison. The three environment-induced conditions (ifcfg, the sshd PAM profile, MariaDB .cache) were not counted in the severity totals. Not tested: Let's Encrypt end to end (there was no real domain), disk quotas (container filesystem), and behaviour on a real virtual machine or physical server. The raw command logs are kept.

Official Sources

My Quarantine Ledger Has 1,945 Rows and Not One Address

Mustafa ERBAY — Fri, 02 Oct 2026 07:25:46 +0000

The question was innocent enough. I wanted to know where a file sitting in my Downloads folder had come from. I couldn't remember its name, my browser history was long, and asking the file itself looked like the shorter path.

macOS has a place for exactly this. Downloaded files get an extended attribute called com.apple.quarantine, and LaunchServices keeps a SQLite ledger alongside it. I opened both. The ledger held 1,945 records, a tidy run from June 2025 to late last night. Then I looked at the address column: all 1,945 rows were empty.

That empty column pulled me into a two-hour measurement. Here is what came out of it: the quarantine mark is not what we think it is. It isn't a record saying "this file came from here"; it's a flag saying "this file needs the user's consent". And whether that flag survives into the next file is decided not by the operating system but by the code of whatever tool touches the file — I tried nine paths, five of them inherited the mark and four lost it silently.

Every measurement below comes from one machine at one moment: macOS 26.6.2 (25G83), arm64, counted on the morning of 2 October 2026. Both the ledger and the folder are live, so tomorrow's numbers will differ — what matters isn't the numbers but the relationships between them.

Reading the Mark: Four Fields, Zero Documentation

Checking whether a file carries the mark takes one command:

xattr -p com.apple.quarantine ~/Downloads/<file>
# 0281;6a8435e5;Chrome;B71407DA-5626-4ACB-8201-7A62855420FA

Four semicolon-separated fields: a flag number, a hexadecimal timestamp, the name of the app that applied the mark, and a UUID. Apple documents this string's format nowhere — I looked, and I didn't find it. But the LaunchServices quarantine properties are documented and map onto the fields one by one. For kLSQuarantineAgentNameKey the documentation says it is the app name of the quarantining agent, and adds that when the key is absent from the caller's dictionary, the agent name is set automatically to the current process name. That is why the third field reads Chrome.

Decoding the timestamp takes a single line:

python3 -c "import datetime;print(datetime.datetime.fromtimestamp(int('6abf42eb',16)))"

So far, familiar territory. What's interesting is whose job the mark actually is. I assumed quarantining a download happened automatically. It doesn't. The documentation for the LSFileQuarantineEnabled key puts it in one line: it is a Boolean saying whether files the app creates get quarantined by default. The mark is something the downloading app volunteers to apply.

Easy enough to test. I pulled the same file with curl:

curl -sS -o astro-logo.svg https://astro.build/favicon.svg
xattr -l astro-logo.svg
# com.apple.provenance:

No quarantine. Only com.apple.provenance, which I'll come back to. Chrome would have left a mark; curl doesn't, because curl isn't a member of that club. That was the first lesson: the absence of a mark doesn't tell you the file is safe, it tells you the tool that fetched it doesn't apply marks.

Volunteering isn't the only door, though. The system carries an exception list of its own that can override an app's Info.plist:

plutil -p /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist \
  | grep -c LSFileQuarantineEnabled
# 18

On this machine, eighteen bundle identifiers are forced to apply marks by the system even without carrying the key themselves — Firefox, Thunderbird, Transmission, Opera, Azureus and a few browsers that no longer exist. The file sits inside root-owned, signed system storage; a user can't edit the list. So you can't tell whether an app applies marks by reading its Info.plist alone. That half the list is decade-old dead software is its own detail: quarantine exceptions have piled up like a museum.

I scanned my own Downloads folder. Including subdirectories, 217 of 306 files carry a mark and 89 do not. The distribution of agent names shows who volunteers: Chrome 77, WhatsApp 45, Claude 42, Outlook 22, Preview 7, Excel 6, Word 2 — plus 16 files where the agent field is empty.

A 1,945-Row Ledger With Zero Addresses

The mark sits on the file, but there's a central ledger too:

sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2 \
  "SELECT count(*) FROM LSQuarantineEvent;"
# 1945

This 425 KB file has stored every quarantine event from 2025-06-26 11:50:06 to 2026-10-01 22:51:21. Broken down by agent, it's a more honest summary than browser history: Chrome 1031, Cyberduck 873, ChatGPT 21, Opera 9, Homebrew Cask 9, Atlas 2.

The schema cheered me up, because the columns I wanted were right there: LSQuarantineDataURLString and LSQuarantineOriginURLString. Their documentation is clear too — kLSQuarantineDataURLKey is the actual URL of the quarantined item, and kLSQuarantineOriginURLKey is the URL of the resource originally hosting it; for web downloads the page on which the user started the download, for attachments the email message or calendar event the item was attached to.

Then I counted:

sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV2 \
  "SELECT sum(LSQuarantineDataURLString IS NOT NULL),
          sum(LSQuarantineOriginURLString IS NOT NULL),
          count(*) FROM LSQuarantineEvent;"
# 0|0|1945

Zero. Zero. One thousand nine hundred forty-five. The columns exist, the schema is unchanged, the documentation still talks about URLs — and on this machine not one of fifteen months' worth of records stores an address. The file side has a similar gap: of the 277 top-level files, only 86 carry the com.apple.metadata:kMDItemWhereFroms attribute, so 69% of them have no "where from" information at all. (The mark ratio was counted over 306 files including subdirectories; I kept these two attribute scans to the top level, which is why the denominators differ.)

Putting the ledger next to the files opened a second crack. The agents marking files on disk include WhatsApp, Claude, Outlook, Preview, Excel and Word; none of those six appears in the ledger at all. The reverse holds too: Cyberduck wrote 873 events into the ledger and has not a single marked file in Downloads. Chrome is the only name on both lists. So writing an xattr onto a file and writing a row into the ledger are not the same job; they happen on separate paths, done by separate implementations. Seeing a mark and concluding "then there must be a ledger row for it" would be wrong for six applications on this machine.

I don't know why, and I'm not going to invent a reason. What I do know is what it means in practice: this ledger is not a forensic record. It tells you which app applied how many marks and when; it does not tell you that a given file came from a given address. If you're investigating a file's origin and leaning on this table, count the column first and confirm it isn't empty.

One note on opening it: don't write to the live database, copy it first and work on the copy. The timestamp is kept in the Core Data epoch, so making it readable needs datetime(LSQuarantineTimeStamp + 978307200, 'unixepoch', 'localtime').

I Tried to Decode the Flag Field, Then Stopped

That first number is very tempting. I had a ready-made sample, so I counted: 0281 on 71 files, 0087 on 42, 0083 on 39, 0081 on 30, 0283 on 18, 0082 on 13, 0287 on 2, 03c1 on one, 0086 on one.

In my own tests I saw a pattern. When I hand-wrote 0083 onto an archive and extracted it, the files that came out read 0283 every single time — the 0x0200 bit added, the agent field cleared, the UUID preserved. (The timestamp sometimes matched the source's and sometimes carried the current moment; I couldn't pin down which is chosen when, so I'm not writing a rule for it.) "Fine," I said, "0x0200 marks an inherited mark." It was a nice hypothesis. Then I tested it in the field: of the 92 files carrying the 0x0200 bit, 76 have a non-empty agent name. If an inherited mark is supposed to clear the agent, those 76 files refute the idea. The other direction held perfectly, though: all 16 files with an empty agent field carry 0x0200. A one-way relationship isn't enough to write a rule on, but it's worth noting.

So I stopped there. Apple doesn't document these bits, and I'm not going to write up a pattern derived from three measurements as if it were a rule. The practical takeaway still holds: don't build policy on the flag field. Whether a file is marked is reliable information; trying to read the mark's subtext means reverse-engineering an undocumented binary format that can change quietly in the next release.

Who Carries the Mark, and Who Drops It

This was my real question. When you extract a marked archive, do the files inside inherit the mark?

I set the experiment up cleanly: packed a small two-file directory as zip, tar.gz and uncompressed tar, applied the same mark to all three by hand, then opened or copied them nine different ways.

Q="0083;$(printf '%x' $(date +%s));SafariTest;$(uuidgen)"
xattr -w com.apple.quarantine "$Q" package.zip
unzip -q package.zip -d A
xattr -p com.apple.quarantine A/package/a.txt
# 0283;6abf42eb;;DCC5A817-FC07-440B-AF95-0BDEEB1405C7

The results came out sharper than I expected:

Path	Mark inherited?
`unzip -q package.zip`	Yes (`0283`)
`ditto -x -k package.zip B`	Yes (`0283`)
`tar -xf package.tar`	Yes (`0283`)
`tar -xf package.zip` (bsdtar reads zip)	Yes (`0283`)
`cp package.zip copy.zip`	Yes (`0283`)
`gzip -dc package.tgz \	tar -xf -`
`cat package.zip > new.zip`	No
`dd if=package.zip of=new.bin`	No
`rsync package.zip new.bin`	No

I tried Python too, in three different orders: open the input before creating the output, do the reverse, create a second file in the same process. The mark survived none of them. I repeated the plain read-then-write case separately with Apple's /usr/bin/python3 and with the python.org 3.13 build — both produced an unmarked file.

That table settles one thing: inheritance is not an automatic behavior of the file system or the kernel. If it were, cat and cp would behave the same way — both read the same file given by name, and one carries the mark while the other doesn't. What separates them is the tools' own code: unzip, ditto, bsdtar and cp implement inheritance, while cat, dd, rsync and Python just move bytes.

Two conditions have to hold at once, and bsdtar demonstrates that by itself: the same tool appears on both sides of the table. tar -xf package.tar carries the mark and gzip -dc package.tgz | tar -xf - doesn't — because in the second case the archive arrives as a stream, so there is no source file and therefore no attribute to read. The tool has to know about inheritance and have a named source to read the mark from. (The rsync row was measured with openrsync, this machine's default, with no extra flags; results may differ with flags that carry xattrs.)

One detail stands out: cp does not copy the mark verbatim. The source reads 0083;…;SafariTest;UUID while the copy reads 0283;…;;UUID. So the attribute isn't being duplicated, it's being rewritten by the system. The same thing happens to files coming out of archives.

Apple's 2026 Patch List Includes These Four Tools

After building the table I went to Apple's security notes, mostly to check whether my measurement was still current. The answer both supported the table and showed its limits.

The macOS Tahoe 26.5 notes (11 May 2026) carry three entries: CVE-2026-28849 in the BOM component, CVE-2026-28900 in libarchive, CVE-2026-28914 in zip. All three describe the same impact: a maliciously crafted ZIP archive may bypass Gatekeeper checks. The descriptions differ, though — the line about a file quarantine bypass being addressed with additional checks appears in only one of the three, the libarchive entry. The same release closes a similar bypass via disk images in the Kernel (CVE-2026-28954), and that entry carries the same description.

The 26.7 notes (14 September 2026) add CVE-2026-65399 in the copyfile component: an archive may be able to bypass Gatekeeper. The same release also closes Gatekeeper or sandbox bypasses in CoreServices, the Kernel, and a component named quarantine outright.

Now lay the two lists on top of each other. The four tools that carried the mark in my measurement: ditto (BOM), bsdtar (libarchive), unzip (zip), cp (copyfile). All four are among the components Apple patched during 2026. That's no coincidence — because inheritance is implemented by the tools, the bugs live in the tools too; every archive format is a separate implementation and a separate bypass surface.

But Apple's list is longer than my table, and that's information in itself: the same two releases patched the Kernel, CoreServices and quarantine as well, and one bypass came through disk images. The quarantine bypass surface doesn't end with archive tools. The table above covers only its most hand-testable part.

What this means in practice: this chain working correctly depends on your OS version. This machine runs 26.6.2, so the copyfile fix isn't in this kernel — which also means the cp row was measured against an unpatched copyfile, and I haven't confirmed it behaves the same on a patched 26.7. If you have a workflow that relies on mark inheritance, staying current isn't a comfort, it's a precondition of the mechanism.

What the Mark Actually Blocks

So far I've looked at how the mark travels. But what does being marked actually do?

The Gatekeeper documentation says it plainly: approval is requested the first time downloaded software is opened, so that nobody is tricked into running executable code they took for a plain data file. The approval path is spelled out too: System Settings, Privacy & Security, "Open Anyway".

I pictured a click-through dialog. Measuring it from the terminal, I met something much harsher. I compiled a three-line C program, left one copy clean and marked the other:

clang -o t1 y.c && cp t1 t2
./t1                      # BINARY RAN, exit=0
xattr -w com.apple.quarantine "$Q" t2
./t2                      # no output, exit=137
xattr -d com.apple.quarantine t2
./t2                      # BINARY RAN, exit=0

137, that is 128+9: SIGKILL. The same binary, the same file system, one attribute of difference. With the mark the process is killed before it starts running; remove the mark and it comes back. No dialog, no warning, no error message — I went looking with log show and found nothing: a three-minute window, --info --debug on, predicates covering process == "kernel", the AMFI sender image and the syspolicy subsystem. There may be lines a non-root log show can't see; if someone can surface them, the method is right here and easy to refute. If anyone has spent hours wondering why a marked binary dies silently when called from a script, the answer is here.

Two more tests, both instructive.

I marked a copy of Apple-signed /bin/echo: it died with exit=137 as well. So the rule isn't "unsigned binaries"; once something stops being a system binary, any unapproved executable is refused while the mark is on it.

Then I put the same mark on a shell script:

printf '#!/bin/sh\necho "SCRIPT RAN"\n' > x.sh && chmod +x x.sh
xattr -w com.apple.quarantine "$Q" x.sh
./x.sh      # SCRIPT RAN
sh x.sh     # SCRIPT RAN

Both ran. A marked script meets no resistance at all, because the thing being exec'd isn't the script, it's /bin/sh. For a plain binary launched straight from the terminal, the gate stands in front of the Mach-O; interpreted code never walks through it. That isn't the same as saying the mark only concerns binaries — app bundles, pkg installers and disk images are separate paths, and the Kernel entry in 26.5 closes the disk image one precisely.

I owe a warning here: the fix for this section is not "delete the mark". For a binary you compiled yourself, xattr -d is reasonable; for one you distribute to other people, the right path is signing it, notarizing it, and attaching the ticket with xcrun stapler staple. Apple's notarization documentation says it plainly: when the user first installs or runs the software, the presence of a ticket tells Gatekeeper that Apple notarized it. Make mark-deletion your distribution strategy and you've asked your users to switch Gatekeeper off.

One more thing worth measuring: what does spctl say? I assessed two copies of a notarized app, one marked and one clean. Both returned accepted source=Notarized Developer ID, and --context kLSDownloadedFileContext didn't change the answer either. So spctl -a reports the state of the signature and the notarization, not the presence of the mark. You have to ask the two questions separately: spctl/codesign for the signature, xattr for the mark.

And the most open spot in the chain: curl … | sh never creates a file. There is no object to mark, so there is no consent to ask for. Gatekeeper has nothing to say on that path — which is not a flaw, it's the scope of the design.

What You're Deleting When You Clean Up

The internet is full of "if the mark is in your way, delete it" advice. Usually with xattr -cr, meaning all attributes at once. So I counted what goes, running xattr -c on a file with three attributes:

before: com.apple.metadata:kMDItemWhereFroms com.apple.provenance com.apple.quarantine
after : com.apple.provenance

The quarantine went, and the "where from" information went with it. So in removing the mark you also remove the only origin record you had — the very record that's already missing on 69% of files. If your goal is to clear the mark, xattr -d com.apple.quarantine takes exactly one attribute; -c sweeps the floor.

The interesting part is that com.apple.provenance isn't removed. It was on the file I pulled with curl, and it's on 193 of the 277 files in Downloads. I couldn't find any Apple documentation for this attribute, so I won't make claims about what it holds; all I'm noting is that it's more widespread than quarantine and that xattr -c can't remove it.

A Half-Hour Check on Your Own Machine

If you want to run the same measurements yourself, here's the order:

Count how many files in your Downloads folder are marked. If the ratio isn't 100%, you have tools that don't apply marks either — the agent field tells you which.
Count the rows in the ledger and check whether the address columns are populated. If you plan to investigate origins, know in advance whether you're leaning on an empty column.
Check your version with sw_vers. The quarantine bypasses in the 26.5 and 26.7 notes concern archive paths; if you rely on mark inheritance, being current is part of the mechanism.
If your build or deployment pipeline extracts archives, choose the tool deliberately. A tar fed through a pipe drops the mark — sometimes that's exactly what you want, but don't let it happen by accident.
If you need to clean a mark, use -d, not -c.
If the machine belongs to a fleet, some of these decisions may not be yours: the Gatekeeper documentation notes that users can override the policy and open any software unless a device management service restricts that. Don't assume the behavior you measure on your own machine will hold on a managed one.

And one habit to avoid: treating the absence of a mark as a trust signal. Eighty-nine of my files are unmarked, and not one of them earned that by passing a check; the tool that fetched them simply didn't apply one.

The Mark Is a Question, Not an Address

I started this measurement wondering where a file came from, and I never got an answer to that question. What I got was more useful: I learned not what the quarantine mechanism is, but what it isn't.

Quarantine is not an origin record. It's a note an app leaves saying "I brought this in, let the user approve it once". Leaving the note is voluntary, carrying it is the tools' business, the ledger keeps no addresses, the only thing it blocks is executable binaries, and when it's removed no trace remains.

What does knowing that change? One general habit around security mechanisms: don't treat the presence of a flag as proof that a check was performed. The mark is a question mark, not a certificate. Black boxes spend their worst nights without telling anyone; this particular box at least confesses what it doesn't do, as long as you count.

Incidentally, there's another example of this machine's own ledgers misleading me: launchd tried 21,881 times and I never looked once. The counters are right there, and nobody reads them.

Official Sources

I Have 91 Backups. None of Them Are This Machine's.

Mustafa ERBAY — Thu, 01 Oct 2026 14:21:41 +0000

I was actually looking at something else. I opened ~/Backups to check whether a side
project's nightly dumps were still landing, saw the listing, and relaxed — 91 files, dates
marching along neatly. Then, in the same terminal, I typed a command I had not planned to
type: tmutil destinationinfo.

The answer was one line.

tmutil: No destinations configured.

What came after made me write this article twice. In the first version I said "I have no
backup," and I believed I had proved it. Then I opened the places I had looked at too
quickly: I did have a backup. It covered exactly my most irreplaceable files, in fact —
and I had forgotten about it.

So the thesis here is not "I neglected my backups." It's the more uncomfortable one:
I didn't know what my backup covered. I had a working copy, a dead copy, and a pile of
things with no copy at all, and the map in my head got all three wrong. Every number below
was measured on this machine on 1 October 2026.

The backup that works: 91 dumps, 760 megabytes, zero surprises

Good news first. Inside ~/Library/LaunchAgents there is a job called
com.burncpu.backup-pull that wakes up every day at 10:00. What it does is modest:

rsync -az --timeout=60 vps3:/opt/burncpu/backups/ "$DEST/files/"
find "$DEST/files" -name "burncpu-*.sql.gz" -mtime +90 -delete

It mirrors the nightly dumps from the server to this Mac; it never deletes to match the
source, it only prunes anything older than 90 days locally. The result: 91 .sql.gz
files, 760 MB in total. The oldest is from 3 July, the newest is
burncpu-2026-10-01T00-00-02Z.sql.gz. Its log is dry and clear:

[2026-09-30 10:00:05] pull basliyor
[2026-09-30 10:00:22] ok — 91 dosya, 767M
[2026-10-01 10:00:05] pull basliyor
[2026-10-01 10:00:15] ok — 91 dosya, 761M

I misread this job's health at first glance. The launchd.err.log next to it was dated 11
June, and I thought I had found a backup that had been stale for four months. But the file
is 0 bytes: it was born in June when the job was installed and has found nothing to write
since. The staleness of the error log was proof of health, not of failure.

That was a preview of the rest of this article: the same silence carries both good and bad
news, and I read it wrong both times on the first pass.

One detail: those 91 dumps are not a backup for this machine. Their original lives on
VPS3; the copy here is the second instance. If the Mac dies, that data lives on.

The machine's own backup: nothing at the system level

tmutil destinationinfo can't find a configured destination. tmutil listlocalsnapshots /
prints an empty heading and falls silent. diskutil list shows a single physical disk:
/dev/disk0, carrying a 460 GiB (494.4 GB) main APFS container; under /Volumes there is
nothing but the system's own boot volume. The data volume is 90% full, 386 GiB in use.

The tooling side says the same: restic, borg, borgmatic, kopia, duplicacy, arq
— none installed, no backup application under /Applications.

I need a parenthesis here, because I fell straight into a hole I dug myself. In my first
pass I ran du on the iCloud folder, saw 60 KB, and wrote "sync is off, the data is
here." Wrong. The folder holds 211 files and every one of them has blocks set to 0 —
they're uploaded to iCloud and evicted locally. du counts blocks, so it reports 60 KB.
The Desktop and Documents entries inside it aren't "empty shells" either; they're
symbolic links to the real folders, and du doesn't follow links, so they show as 0 bytes.
What I took as proof that sync was off was proof that it was working.

A week ago I wrote
a separate article
about how the "free space" number on this very disk changes depending on who you ask. I
forgot that article's lesson in my own measurement one week later. This seems likely to
keep happening, so I'm writing it down here.

I didn't forget to turn Time Machine on — I have no disk to turn it on with

Apple's documentation sets two conditions for Time Machine. Capacity: a device with "at
least twice the storage capacity of your Mac." Dedication: you use that disk only for Time
Machine. Twice a 460 GiB disk means, roughly, a 1 TB external drive — and it means that
drive exists solely for this purpose. I don't own one. So what's missing isn't a setting,
it's hardware.

Local snapshots don't close the gap either. As far as Apple's page says, they're kept on
the Mac's own startup disk, taken roughly every hour, and held for 24 hours. A copy that
lives on the same disk is no help when the disk itself is what goes.

My list is empty. I don't know exactly why, and I won't present my guess as fact: the
destination may simply never have been configured, or Apple's other condition on the same
page may not hold — it says Time Machine "stores snapshots only on disks that have plenty
of free space," and my data volume is 90% full. Two candidate causes; my data isn't enough
to choose between them.

What would I lose: 37 repos, 209 commits

"I have no backup" is dramatic on its own, and empty. What matters is how much
irreplaceable material sits on that single disk. ~/IdeaProjects is 35 GB. I counted.

Measurement	Count
Git repositories (`.git` directory, two levels deep)	37
Repositories with no remote (`git remote`) at all	9
Of those, ones with real history	8
Commits in those 8 repositories	209
Tracked files in those 8 repositories	1,332
Repositories ahead of their remote (unpushed)	4
Unpushed commits	143
Dirty repositories	14
Modified files	275
Untracked entries / actual files	230 / 1,518

I'm writing the counting method down so it can be checked: I searched for .git
directories, which leaves four git worktrees out of the list — their commits already
live in the main repository.

I also have to correct that "230," because my first pass had a units error in it: 230 is
the number of entries git collapses, so most of them are directories, not individual
files. The real untracked file count is 1,518. And that number shouldn't be taken at
face value either: one empty scratch repository alone accounts for 1,118 of them, another
for 196. So most of the 1,518 is noise. The real work is in the places I sampled: a new
cost-rollup module, a barcode handler, six new test files, and database migrations numbered
from the tenth through the twenty-second. Migrations are the kind of file that takes the
schema with it when it disappears.

I'm not writing the repository names down; most of them are client work. But I do need to
mention one: a 916 MB portal, 19 commits, its last two commits made on 29 September, 44
minutes apart. So it's live work from days ago. Remotes configured: zero. One of those
commit messages contains this: "restore the quote package and pricing work that was lost in
production." The code that rescues work lost in production is not itself anywhere it could
be rescued from.

Let me say exactly what the check covers when I say "no remote": git remote returns
empty, meaning there is no configured path to push to. That doesn't prove the code was
never copied anywhere — as we're about to see, some of it was. But on a recovery night, "I
think I copied it somewhere" doesn't count as a plan.

Where git doesn't look — and where it looks by accident

I counted the .env style files across the projects: 18. Then I asked each one, "is git
tracking you?" On my first attempt the answer was "none," and I was pleased with that. My
method was broken: I asked about each file by its bare name instead of its path relative to
the repository root, so git naturally found nothing.

Asked properly: 16 are untracked, 2 are tracked.

The repositories holding those two have an .env rule in their .gitignore. So the rule
was written and the file is inside anyway — added before the rule, or force-added. Their
contents are two lines of domain and API address, not a secret leak; but that isn't the
point.

The point is this: what git protects is what git looks at, and I was wrong about both
sides of that boundary. 16 files sit outside the scope, by my own choice, despite being the
hardest class to reproduce. 2 files sit inside it while I believed they were out. I can
rewrite a lost source file; I cannot rewrite a lost key — I'd have to go to the provider,
generate a new one, and redeploy it everywhere it was used.

The destination I named "backup" is dead — the one I didn't name works

Hoping for better, I looked at the rclone side. Two remotes are configured:

gdrive:
gdrive-yedek:

The second one is named, literally, "yedek" — Turkish for backup. I tried listing it:

CRITICAL: Failed to create file system for "gdrive-yedek:":
couldn't find root directory ID: ... couldn't fetch token:
invalid_grant: maybe token expired? - try refreshing it

In the first version of this article I stopped right here. I wrote "the destination I named
backup cannot authenticate," closed the section, and that was drama built on selective
evidence. I saw two remotes and tested only the broken one.

When I opened the other, seven folders came back. Three of them matter:

A "Projeler" folder dated 22–30 April: ~100,000 objects, 5.6 GiB. Two branches underneath, lokal and harici, with six project folders under lokal.
A folder dated 18 September: Android release signing material and a restore note. A signing key is even less replaceable than an API key.
A key escrow dated 11 July: GPG-encrypted, 1,788 bytes.

So an offsite copy exists, encryption exists, a restore note exists. My
sentence "I have no copy anywhere" was wrong, and I had to delete it.

But the real finding starts here. I compared the names of the six project folders in that
April copy against the list of nine repositories with no remote: none of them match.
Worse, the earliest first commit among those nine is 7 July. So the copy sitting in the
cloud was taken ten weeks before all the work I had flagged as at risk. However
well-intentioned, that folder does not protect me — what it would need to protect hadn't
been born yet.

This is the best evidence for the "silence is the default" thesis. Nobody told me anything
false; I never asked anyone the right question. The destination with "backup" in its name
misled me precisely because it had a name; the one without a name escaped me precisely
because it didn't. An inventory should be kept by measurement, not by label.

I also don't know why the token died. Google's documentation lists several causes for
invalid_grant: access being revoked, a refresh token going unused for six months, a
password change while Gmail scopes are granted, exceeding the limit of 100 tokens per
account — in which case, in the documentation's own words, the oldest are invalidated
without warning — and, for projects whose publishing status is "Testing," expiry in 7
days. I can't say which. What every item shares is that none of them is obliged to tell me.

I've caught this pattern before. A launchd job
tried 21,881 times and failed every time,
and I never looked once.

Today, for the first time, I tested a dump

Last question: can those 91 dumps I called good actually be restored?

I searched the pulling script for restore, verify, gunzip -t, zcat, psql,
pg_restore. Zero hits. The script copies and prunes; it does not test. For ninety-one
days files arrived, and not one was ever opened.

Today I tested the newest. gzip -t passed, and zcat gave up its first lines:

--
-- PostgreSQL database dump
--

That proves one thing: the dump's stream was not truncated. A pg_dump | gzip that
died halfway would fail its checksum. That's all I know, and I won't claim more. gzip -t
verifies the integrity of the compression container, not the consistency of the SQL inside
it; a missing role, a missing extension, a broken ordering would all be invisible.
PostgreSQL's documentation, describing a plain-text dump, says what to do with one: to
restore it, you feed it to psql. A dump becomes a backup only when a database eats it.

CISA's backup guidance asks for three things: 3-2-1 (three copies, two different media, one
offsite), protecting the backups themselves (physical security, encryption, offline
copies), and testing the procedures. Let me draw my own scorecard honestly:

	Copies	Offsite	Encrypted	Tested
Side project database	2	yes	no	never
April project copy	2	yes	no	never
Key escrow	2	yes	yes	never
Work since 7 July	1	no	—	—

Here's what's interesting: the best job I did is in the smallest file. A 1,788-byte GPG
escrow covers two of the three legs at once. The rest was built out of that day's urgency,
not out of habit.

What I'm going to do, ordered by risk

Add remotes to the 9 repositories that have none. 209 commits stop being single-copy in exchange for one evening's work. Client code goes to private repositories. Cheapest action, biggest gain.
Push the 143 unpushed commits, decide about the 275 modified files. Migrations first. Most of the 1,518 untracked files are junk, but I can't know that without sorting them.
Keep the inventory by measurement, not by label. This is the real lesson of this article. I didn't know what the April copy covered; unless I write down what today's copy covers, I won't know that either in three months.
Either refresh the dead token or delete it. A non-working path with "backup" in its name is worse than no path at all — judging by the fact that it made me write this article twice, it did the same to me.
Buy a 1 TB disk and turn Time Machine on. And remember that a backup disk left permanently connected is also exposed to ransomware; CISA's "offline copies" item exists exactly for that.

An honest note: I did not do item 1 today. If I don't do it tomorrow, this paragraph will
be sitting here to embarrass me.

Backup isn't a tool question, it's an inventory question

What I take from this measurement isn't "I neglected my backups" — that's what I thought
when I first wrote it, and I was wrong. I did make backups. There is a daily, automatic, 91-file one; a 5.6 GiB copy of my projects went to the cloud; a key escrow got encrypted with GPG. I couldn't remember what any of them covered.

Because the thing that's easy to automate is not the thing that's expensive to lose. The
database on the server dumps with one command and already exists in two places. A
half-finished migration, an uncommitted test, a key shown exactly once: none of them dumps
with one command. Backup effort ran downhill, like water, along the path of least
resistance — and nobody reminded me where that path ended up.

Back in June I published an article on this blog arguing that backups are everyone's
responsibility. While writing it, I had not run tmutil destinationinfo on my own machine.
The distance between those two things bothers me, so I'm putting it in writing: giving
advice is easier than measuring.

The default sound a system makes is silence. Time Machine didn't report having no
destination, the token didn't announce its death, the April copy didn't mention going
stale. And today I read a 0-byte error log first as a fault and then as proof of health,
read iCloud's evicted files as "sync is off," and wrote the sentence "I have no copy
anywhere" only to delete it. I misread the same silence three times in one day.

So don't ask your own setup "do I have a backup?" You'll answer that one with a clear
conscience. Ask this instead: what exactly does my backup cover, up to what date — and how
do I know? I asked today, and I didn't know half the answer.

Official Sources

I Piled Up Seventy Branches; Six Were Really Unfinished

Mustafa ERBAY — Thu, 01 Oct 2026 11:53:07 +0000

I typed git branch in the blog repo and the screen started scrolling. I counted:
71 lines. One is main; the other 70 are places where I once started something and
walked away.

71 local branches (including main) · 56 remote branches
tip commit by month:  2026-05: 4   06: 15   07: 21   08: 7   09: 23   10: 1

Every number in this article belongs to a single moment: the morning of 1 October 2026,
with origin/main at 4702bc4e. A branch list is a living thing; by the time you read
this it will have changed on my side too.

Four of them date back to May. They have been sitting there for four and a half months
and I cannot remember what any of them are.

What I felt looking at that list was not a technical feeling. It was the "I have this
much unfinished work" feeling — a to-do list nobody wrote, growing on its own. I wanted,
for once, to turn that background hum into a number. This article is the story of that
count, and the count ended differently than I expected: I asked about unfinished work in
four separate ways, got four separate answers, and the real debt turned out to be exactly
where I had not been looking.

Seventy-one lines, zero information

Let's admit something first: the length of git branch output measures nothing. Creating
a branch is free; it amounts to writing a forty-character commit id somewhere. And I
created one in every agent session, every experiment, every "let me just poke at this"
moment all year. Of the 70 side branches, 23 carry the claude/ prefix — working areas
that were opened automatically.

Still, every time I open that list it says the same thing to me: you left a lot of work
half-done. Here is the interesting part — git does not manufacture my guilt, I do; git
merely stores it, and I pay interest on it with every branch call. I was also reluctant
to delete the branches, because of the fear that "there might be something in there." I
had never once checked whether that fear was justified. Not in four and a half months.

To get from a number to information, I had to ask a question. I did not know how many
questions it would take.

First question: ancestor or not

The most familiar question, and the easiest. git branch --merged and --no-merged. The
documentation describes what they do without any embellishment:

"With --merged, only branches merged into the named commit (i.e. the branches whose
tip commits are reachable from the named commit) will be listed."

So the test is one single thing: can the branch's tip commit be found by walking
backwards from the named commit? Reachability. A genealogy query.

git branch --merged origin/main    → 46
git branch --no-merged origin/main → 24

Forty-six branches are closed, provably. That leaves 24, and my first reflex was to read
that as "24 unfinished jobs." So this was the numerical equivalent of four and a half
months of humming.

It was not. Because this command does not know how I work.

Second question: is the content the same

Most of the work I do by hand in this repo lands through pull requests, squashed; the rest
of the commit traffic is the bot's content generation. GitHub's own documentation states
the outcome of a squash plainly:

"Squashing turns all commits in the pull request into one commit on the base branch."

GitLab describes the same behaviour on its side as "Squash and merge combines multiple
small commits into a single meaningful commit." So this is not a GitHub quirk but a common
way of merging.

The result: a new commit, a new identity. My branch's tip commit will never appear in
main's history, because what entered main is not a copy of it but a summary. The
content landed; the genealogy did not. --no-merged cannot distinguish this case, and it
is not supposed to — I asked it a different question.

The command that asks the right question is git cherry. It looks at the diff, not the
genealogy:

"The equivalence test is based on the diff, after removing whitespace and line numbers."

A commit marked - has an equivalent upstream; one marked + does not. The idea
underneath is git patch-id; its documentation defines a patch id as "a sum of SHA-1 of
the file diffs associated with a patch, with line numbers ignored," and it ties the two
commands together explicitly: "git-cherry shows what commits from a branch have patch ID
equivalent commits in some upstream branch." So if a patch does the same work, it yields
the same fingerprint regardless of where it was applied or which commit id it carries.

I ran all 24 branches through this question:

commits on the 24 branches that main lacks : 34
of those, with an equivalent upstream      : 20
without an equivalent                      : 14

branches with no unique commit at all      : 15
branches with unique commits               :  9

Now this number says something. Fifteen of the 24 "unfinished" branches are completely
empty — the work every commit inside them does is already in main, just under a
different identity. fix/pace-date, fix/pace-runner, fix/pace-url-encoding,
fix/devto-mermaid-restore... The output is unambiguous, too; for fix/pace-runner the
command returns a single line, and the mark at the start of that line says everything:

$ git cherry -v origin/main fix/pace-runner
- c36fff1b539c6066e23c02e3d03b2fe99a171f9f fix(ci): pace-check hosted runner yerine self-hosted

Minus. It has an equivalent. The work inside a branch I had been reluctant to delete for
sixteen days had already landed in main. Fifteen of the 24 suspects dissolved at this
step.

I could have stopped here and congratulated myself. Nine instead of 24, how nice. But
there is something git cherry cannot tell you either, and this time it errs in the
opposite direction: it compares diffs, not intentions. If I rewrote the same work, the
two patches look different and the commit gets marked +. So 14 does not mean "open
work"; it means "possibly open work."

That leaves exactly one method, and it cannot be automated: open the nine branches one by
one and read them.

Third question: was this work really not done

I opened the nine branches in order, looked at which files they touched, then searched for
those files in origin/main. For three of them the answer was clear — the work landed, the
branch is redundant:

claude/calendar-atomic-write — the commit message reads "wip: local working session
sync." Inside it are AnimatedPostCover.astro and copy-blog-assets.mjs. Both are sitting
in main. The patch looks different because this commit froze half a session as-is; the
work itself landed through a proper PR.

fix/topic-research-quota-fallback — two commits, 894 added lines; the job of decoupling
topic verification from the Gemini quota. I searched main for the SEARCH_STOPWORDS and
TR_FOLD definitions; both are there. It landed, but then evolved, which is why the diff
no longer matches.

fix/self-host-workflows — on 21 July I opened this to move the keep-alive and lighthouse
jobs onto my own runner. In main today, line 27 of keep-alive.yml and line 17 of
lighthouse-audit.yml say the same thing: runs-on: [self-hosted, itwise-mac]. The goal
was reached — two months later, by a different commit, with a note dated 26 September. The
branch's intention came true; the branch itself became redundant.

Three branches, four commits, zero debt. The remaining six are another story.

Six that are real

For each of the six I looked for the evidence in main and failed to find it:

Branch	Work	Evidence
`claude/elegant-bassi-fa298f`	never-404 hardening, 5 commits, 9 June	`500.astro`, `en/500.astro`, `maintenance.html` absent; zero trace of 500 in the middleware
`fix/callout-structure-gate`	Callout structure gate, 27 July	`mdx-structure.ts` absent; `validateMdxStructure` is imported by neither of the two validators
`fix/accuracy-source-coverage`	widening the source allowlist, 21 August	the allowlist lacks `caddyserver.com`, `libvirt.org`, `linux-kvm.org`, `isc.org`, `letsencrypt.org`; `source-policy.mjs` is 242 lines in `main`, 292 on the branch
`fix/unconfirmed-rejection-not-permanent`	stopping an unverified rejection from eating the queue, 2 September	`topic-research-chain.mjs` in `main` is 283 lines, zero relevant trace
`claude/dreamy-maxwell-17b977`	caption timeout + outage breaker, 18 September	`provider-outage.mjs` absent; `main` still uses 5- and 15-second timeouts
`claude/beautiful-noether-e1c690`	the ispmanager review article, 21 September	file absent from `main`; zero hits across the 1300 records of the live feed; URL returns 404

Seventy came down to six. So 91 percent of the humming was not real.

Where age says nothing

Once the count was done I was left holding an intuition, and I wanted to test that too:
an old branch is a dead branch, a new branch is live work. It sounds reasonable. I sorted
both lists by date.

Of the 15 empty branches, the oldest is 16 May (138 days) and the newest is 23 September.
Of the 6 genuinely open ones, the oldest is 9 June (114 days) and the newest is
21 September. The two ranges almost entirely overlap.

The intuition was wrong — but exactly how it was wrong is the interesting part. One side
of it holds: of the four branches left over from May, one had already been merged and the
remaining three are still on the list, and all three are empty; so the fear that "they
have been sitting there for four and a half months" had precisely zero substance behind
it. But the 114-day-old elegant-bassi is sitting right there too, carrying five commits
of real work. Age indicates neither innocence nor debt. An aged branch has either found
its own way in or is waiting somewhere nobody looks, and you cannot read which from its
date.

That is why there is no shortcut around the third step. Whatever filter I tried, the only
thing that answered the question was looking at the file.

What the minus sign does not say

The method's most important blind spot was sitting inside my own data.

claude/beautiful-noether-e1c690 carries two commits. One +, one -:

$ git cherry -v origin/main claude/beautiful-noether-e1c690   # SHAs abbreviated
+ fd33e470  feat: yeni makale — ispmanager Lite'ın Altına Baktım: 23 Bulgu
- ea466a4b  fix(ispmanager): final denetim — yayın 25 Eyl, disclosure, kanıt düzeltmeleri

The equivalent of the minus-marked ea466a4b really does exist in main's history:
6003feb2. But there is one more line right after it:

fd17e0ff Revert "fix(ispmanager): final denetim — yayın 25 Eyl, ..."
         This reverts commit 6003feb2381fc9f5877c6ef687ed1f58dac3e77b.

So the minus in git cherry does not mean "this work is live"; it means "an equivalent of
this entered upstream at some point." If it entered and was later reverted, the mark does
not change. I caught it only because this branch also carries a + — had the branch held
that - commit alone, my four-step method would have called it "empty, safe to delete."
And the work is not live.

Catching your own method with your own data is a strange feeling. The thing you are
measuring turns out to be measuring your instrument too.

Two correct notes, two jobs never done

Two of the six are where I actually stopped — but not for the reason I expected.

For claude/dreamy-maxwell-17b977 I wrote this on 18 September: the social posting
pipeline hung for 40 minutes, the cause was a fetch without a timeout, the fix is this
module. The note's last line reads: "Awaiting verification: check this line in the first
social-post log after the merge." I asked gh:

$ gh pr view 13 --json number,state,mergedAt,headRefName
{"headRefName":"claude/dreamy-maxwell-17b977","mergedAt":null,"number":13,"state":"OPEN"}

Still open. Open for thirteen days. The note was not wrong — on the contrary, it said
plainly that it was waiting for a merge. That merge simply never happened, and nobody
went back to check. The code in main today still runs with the old timeouts; that
40-minute hang could happen again tomorrow.

The second one is starker. For the ispmanager review, my note records in three separate
places that the article never went live, including the verification of the live 404. The
note even contains the command to run on 25 September: revert the two revert commits, push,
trigger the deploy run. The command was written correctly. It was never run. Today the
article is absent from the 1300 records of the live feed and the URL still returns 404.

The story I expected was "my notes misled me." The real story is more uncomfortable:
neither git nor my notes were wrong. Both were correct, both were right there, and nobody
went back to either. I had nothing at all that distinguishes a decision written down from
a job carried out — and that gap was sitting inside the branch list the whole time,
invisible because a list of names cannot show it.

The feeling was familiar. Five days ago I wrote about forming the right hypothesis two
months early and losing it; the day
after that, about rolling back in ninety seconds and leaving things broken for five
hours. All three share
one thing: the record was there, and nobody read it.

How I count now

Four steps, in order. If you want to try it on your own setup:

Do not mistake a number for information. git branch | wc -l produces a feeling, not data. The only job of this step is to accept that counting is not enough.
Ask the genealogy (--merged / --no-merged). Which branches closed is proven. The ones that look open are suspects, not convicts.
Ask the diff (git cherry origin/main <branch>). This is where you catch work that landed via squash. For me, 15 of the 24 suspects dissolved at this step.
Read the survivors by hand. Rewritten work does not resemble the diff; no automation that skips this step can answer "was this work done."

Three traps caught me while applying the fourth step, and I am writing all of them down:

A - does not mean "live" (the section above). Work whose equivalent landed and was then reverted also shows as -. If in doubt, look at what came after the equivalent commit.
git branch -d refuses these branches. The documentation's condition is explicit: the branch must be fully merged in its upstream branch or in HEAD. Those 15 branches are --no-merged by definition, so you have to use -D. The evidence you gather does not complement the safety of -d; it replaces it — so do not type -D without gathering it.
A branch attached to a worktree cannot be deleted. In this repo four branches are currently held by a worktree, one of them from the "genuinely open" six. You have to remove the worktree first.

And one area the recipe does not cover at all: all of this is local. I have 56 remote
branches against 70 local ones, and deleting a local branch does not delete
origin/<branch>. A branch that exists only on the remote never appears in git branch
output at all — so this count is itself incomplete.

What I took away

The count did not show me my unfinished work. It showed me how I audit myself.

For four and a half months I looked at a seventy-line list and felt vaguely bad, because
staying vague was free. Reading nine things one by one was uncomfortable — and it was
precisely that discomfort that found two jobs never done. Vague guilt turns out to be a
defense mechanism: as long as I know nothing for certain, I never have to admit that
something clearly did not get done either.

A number is a feeling, a diff is a fact, and only reading tells you which one is right.
But I saved the real lesson for last: in this count, no record lied. Git was right, my
notes were right, the PR's status was in plain view. The only thing missing was someone
going back after writing. Black boxes spend their worst nights without telling anyone;
mine had recorded everything properly — nobody opened the lid.

DEV Community: Mustafa ERBAY

CPU Quota Does Not Look at Your Average

I swept the fleet and found 16 quota'd cgroups

The window is 100 ms, and there is no average in there

Four ways to misread cpu.stat

The fix has been in the kernel for four years: cpu.max.burst

The experiment: half a burst buys you nothing

The dial nobody turns: period

Where the chain breaks

What to ask about your own setup

What I did not prove

Official Sources

I Could Not Turn On Journal Sealing; Only the Debug Log Said Why

The lab

What "Operation not supported" does not say

Once sealing really engages

How wide does the tail get

What sealing looks like inside the file

What --verify checks independently of sealing

What sealing catches

Then I deleted the logs

The patch that seals empty epochs

Delete the key too, and verification goes quiet

How v262 changes this picture

What I do in practice

Conclusion

Official Sources

I Trust 158 Root Certificates. I Installed Three of Them

I asked the same question in four places and got four answers

That extra certificate in the store isn't a bug

Counting certificates is not counting trust

Of the 157 roots, zero carry a decision of mine

The three roots I added myself

I apparently promised an ad blocker until 2045

And then there are the expiry dates

If you want to read your own list

Trust is the sum of decisions you don't remember

Official Sources

Forty-Four Posts in a Row From One Category, a Reader Caught It

The queue ran out on 3 September

The fingerprint of a reflex

A control group: what came out while the queue was full

The volume indicator never turned red

The warning came from a reader

Back-testing the detector I did not have

Why the reflex kept going to the same place

Writing the rule into the system: compute, do not choose

Eleven days later — and the limits of this measurement

The reflex moved up a level

Applying it to your own work

Conclusion

Official Sources

Who Inherits the Queue of a Closing Socket?

The lab, and an honest disclaimer

First measurement: is the loss real?

The mechanism: when does the socket leave the group

Who picks the target: the docs say random, the code says hash

Migration does not respect the target's backlog

Which connections recover on their own

Who is allowed into the group

Before you turn it on

Where the counter goes quiet

Official Sources

I Enabled casefold, and the Directory Stayed Case-Sensitive in Turkish

The lab

First, the portability trap: mke2fs does not consult your kernel

Baseline behaviour: names are preserved, lookups flex

Turkish: one directory, two different rules

Why: the kernel drops the T rows

This is not an ext4 quirk

casefold does normalization too, not just letters

Three tools, zero exit codes, one file

Before migrating: scan, but scan correctly

strict: keeping invalid UTF-8 at the door

Cost: the first measurement was wrong

Your backup does not carry +F

Security: anyone who can choose a name can choose a collision

Backing out: two separate doors

A decision frame

Conclusion

What `--verify` checks independently of sealing

Your backup does not carry `+F`