DEV Community: ilya mozerov

Your default branch is an allowlist, and it votes healthy

ilya mozerov — Mon, 31 Aug 2026 00:05:11 +0000

We run a fleet of long-lived agent sessions that coordinate through a claim file: before touching a
shared resource, a session claims it, and other sessions stand down. A claim that is never released
would deadlock the fleet, so there is a sweep that decides whether a claim's owner is still tending
it or has gone away.

The sweep's core is a case over an exit code:

mesh-mind-state "$win" >/dev/null 2>&1
case "$?" in
  5)   echo STALE ;;          # DEAD pane
  8)   echo STALE ;;          # DEAD-SHELL: no engine at all
  9)   echo STALE ;;          # AUTH-DEAD: logged out
  7)   echo UNKNOWN ;;        # ABSENT: not a window at all
  4)   echo LIVE  ;;          # NEEDS-INPUT: blocked but alive
  *)   # 0 = WORKING / IDLE / UNKNOWN
       if ! mesh-mind-state "$win" 2>/dev/null | grep -qiw IDLE; then
         _strike_reset "$key"; echo LIVE; return
       fi
       ...
esac

Read the *) branch as what it actually is. It does not mean "the state is 0." It means every
exit code nobody wrote an arm for, and the only question it knows how to ask is whether the word
IDLE appears in some text. Anything that is not the string IDLE is treated as working.

That is a classifier whose unhandled input votes healthy.

The state that walked in

The tool being classified grew a state, on its own schedule, for its own reasons. A session that
hits an API quota wall prints a banner and stops taking turns; the state reporter exits 6 for it.

# Exit (single window): 0 WORKING/IDLE/UNKNOWN · 4 NEEDS-INPUT · 5 DEAD · 6 RATE-LIMITED
#                        7 ABSENT · 8 DEAD-SHELL · 9 AUTH-DEAD

There was no 6) arm. A quota-shed session fell to *), its banner did not contain the word
IDLE, and the sweep declared it LIVE — tending its claim.

Nothing crashed. No log line said anything was wrong. The claim just quietly belonged to a session
that could not execute a single instruction, and the tool responsible for noticing that was the
tool reporting everything was fine.

The bill: one claim sat there reading as merely expired for over five hours, while a nagging
reflex kept sending its owner reminders to renew or release it. I can still count fifteen of
those on the surviving log; the incident report that filed the bug counted 38 across the whole
window before the log rolled. The owner was behind the quota wall for every one of them. It could
not read a single one.

The part that makes this worth writing down

Look again at what the branch does before it returns:

_strike_reset "$key"; echo LIVE; return

It is not only a wrong verdict. It is a write.

The sweep does not convict an owner on a single reading — that would be far too trigger-happy, since
every session looks idle between turns. Conviction requires the owner to look idle across
MESH_CLAIM_IDLE_STRIKES (default: 2) separate, spaced sweeps, with the claimed artifact untouched.
A counter accumulates the evidence.

And the fall-through branch resets that counter. Every sweep.

So the misclassification was not a one-shot error that a later, better-informed sweep could correct.
It was self-sustaining: each sweep looked at a state it did not understand, guessed "healthy,"
and destroyed the record that would have let the next sweep guess less. Conviction could never
reach 2 because the counter never survived to 1.

That is the shape I want to name. A classifier that mislabels is a bug you find by reading the
label. A classifier that mislabels and erases the evidence of the mislabel is a bug you find only
when somebody counts 38 unanswered reminders by hand.

If your default branch has a side effect, the side effect is running on inputs you have never
thought about. Mine was clearing evidence. Yours might be caching, or acking, or advancing a cursor.

Three repairs, one branch, fifty-four days

Then I ran git log on the file, and the finding got worse in the way findings do.

The 6) arm was not the first repair to that fall-through. It was the third. The other two are
still in the source, each with a comment explaining itself:

8) DEAD-SHELL, landed 2026-07-07:

Without this the default branch below falls through (DEAD-SHELL never contains the word "IDLE")
to LIVE — a shed/killed owner's claim would never age out.

9) AUTH-DEAD, landed 2026-07-10:

Same default-branch trap as DEAD-SHELL (AUTH-DEAD text never contains "IDLE"), so without this
arm a logged-out owner's claim reads LIVE forever.

6) RATE-LIMITED, landed 2026-08-30 — the one I started with.

Three times, fifty-four days apart end to end, someone diagnosed the same mechanism, wrote a
lucid comment about the same mechanism, and then fixed one input to it. Not one of the three
repairs touched the *) branch — I diffed each commit, and the branch body comes out byte-identical
on both sides of all three. Every fix was a new name added to a list of states
that get correct treatment, which is another way of saying every fix left the unnamed state
getting the wrong treatment — and the set of unnamed states is, by construction, the ones nobody
has thought of yet.

The second comment is the tell. It names the trap, calls it "the same trap," and then does the same
thing about it.

We had already written the rule

Here is the part that stings, and the reason this is a confession rather than a lesson.

Our own engineering doctrine has an entry for exactly this shape, added months earlier off a
different tool:

When a guard has been widened by one more member three times, invert its POLARITY — an exclusion
allowlist's failure direction is SILENCE, so gate on what the thing CLAIMS to be and let the
unlisted case fail LOUD.

The 6) arm is the third widening. The rule came due in the same commit that widened it a third
time. We had the rule, in writing, in a file every session reads, and we added another name to the
list anyway — because from inside the fix, adding one arm is obviously correct. It is correct.
It is a good arm. It has a live test and a real incident behind it.

It is also the third instance of a pattern that the third instance is supposed to end.

Inverting the polarity here means the case stops being "these codes are bad, everything else is
fine" and becomes "these codes are understood, everything else is a state I have never seen and
I will say so loudly." An unhandled exit code should produce UNKNOWN and a complaint on the way
out, not a confident LIVE and a wiped counter. It is a smaller diff than the arm we wrote. The
reason it did not get written is not that it is hard; it is that a fall-through is invisible from
inside the ticket that made you open the file, and the ticket always names one state.

What to take

Three things, none of which are about bash:

A default:/else:/case _: arm is an allowlist of everything you happened to know about
when you wrote it. The question is not whether it is exhaustive today. It is which way it
fails when the enum on the other side of the boundary grows — and enums on the other side of a
boundary always grow. If the answer is "toward healthy," the failure will be silent, and silent
is how it stays for fifty-four days.
Audit your default branches for writes, not just for verdicts. A wrong verdict is
recoverable next tick. A wrong verdict that clears the state a later tick would have used is not
a wrong verdict, it is an amnesia loop, and it will look from the outside like the healthy case
holding steady.
Count the repairs before you write another one. git log -S on the branch you are about to
extend costs thirty seconds. If it is the third time, the diff you came to write is not the
diff. We wrote the rule for this down and still missed it — which is roughly the argument for
making it a mechanical check instead of a maxim.

The 6) arm shipped and is correct. The class it belongs to is still open. Exit code 10, whenever
somebody adds it, will fall straight through to LIVE.

Your fake binary on PATH does not survive sudo, and the test does the real thing instead of failing

ilya mozerov — Sun, 30 Aug 2026 17:45:20 +0000

The standard way to test a script that shells out to some dangerous command is to put a fake copy of
that command first on PATH, run the script, and assert on what the fake recorded. It is cheap, it
needs no container, and it works — right up until the code under test escalates.

Here is the whole thing, measured on the box I am writing this from:

$ cat "$stub/tailscale"
#!/bin/sh
echo "STUB REACHED: $*"
exit 7

$ PATH="$stub:$PATH" tailscale --version
STUB REACHED: --version

$ PATH="$stub:$PATH" sudo -n tailscale --version
1.102.2
  tailscale commit: 6cac918179d4d673bfebe2fc74f81183ddd73fea

Same shell. Same PATH. Same command. The second one reached the real binary.

Why

sudo does not inherit your environment. Two lines in /etc/sudoers on this machine, both stock
Debian/Ubuntu defaults:

Defaults    env_reset
Defaults    secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

env_reset throws away the environment you handed it. secure_path then replaces PATH with that
fixed list — which contains no /tmp, no ~/.local/bin, and nothing else a test harness could
write to without being root already. This is not a misconfiguration to fix; it is the entire point
of secure_path. It exists specifically so that a user who controls PATH cannot decide what runs
as root.

Which means: the security property that makes sudo safe is exactly the property that silently
uninstalls your test double.

The part that makes it worse than a broken test

If this merely broke the test, it would be a nuisance. It does not break the test.

The function I was looking at tries unprivileged first, then falls back:

ts_set(){
  local out
  if out="$(timeout 15 tailscale set "$@" 2>&1)"; then printf '%s' "$out"; return 0; fi
  local unpriv="$out"
  if out="$(timeout 15 sudo -n tailscale set "$@" 2>&1)"; then printf 'via-sudo %s' "$out"; return 0; fi
  ...
}

That is a good shape — a machine that has been configured not to need root should not be pushed
through one. But now trace a test that stubs tailscale to deny, in order to exercise the refusal
path:

The stub is reached. It denies. Correct so far — this is the branch under test.
The fallback fires: sudo -n tailscale set ...
secure_path discards the stub directory.
The real tailscale runs.

And this box is:

mesh-home ALL=(ALL) NOPASSWD:ALL

so step 4 needs no password and no prompt. The command in question changes the machine's network
egress. The "smoke test" would have reconfigured the live route out of the host — the same route the
operator reaches the host over — and then reported green, because as far as the test could tell the
command succeeded.

The failure mode of a defeated test double is not a red test. It is the real action, performed on
the real system, reported as a pass. A red test tells you something. This tells you nothing, and
does the thing.

I want to be precise about how this generalises, because there is a weaker and much better-known
version of it: a subject that sets its own PATH will also lose your stub. That one is your code's
fault and you can fix it in your code. This one is not. You cannot make sudo honour your PATH
from the calling side — that is what it is for — so no amount of care inside the script under test
removes the hole. The boundary itself is the problem.

The fix is to stub the thing that is still on your PATH

You cannot fake what runs behind sudo. You can fake sudo.

sudo is resolved from the caller's PATH, by the caller, before any privilege exists. So it is
stubbable by exactly the mechanism that failed for the binary behind it:

printf '#!/bin/sh\necho "sudo: a password is required" >&2\nexit 1\n' > "$stubdir/sudo"
chmod +x "$stubdir/sudo"

Now the escalation path is closed at the boundary rather than open past it, and — this is the useful
part — you have also made the "no sudo available" branch testable, which is a real branch that
otherwise only ever runs on someone else's machine.

Two things worth keeping around this:

Run the subject under a private HOME too. env -i HOME="$fake" PATH="$stubdir:/usr/bin:/bin",
with the fake home's bin being the stub dir. Otherwise a subject that resolves tools relative to
$HOME walks straight out of your sandbox by a second door.

If the library that holds the escalating call is missing, exit non-zero — do not fall back to a
raw call. The temptation is command -v mylib || do_it_directly. That silently reintroduces the
unstubbed path in exactly the environment where something is already wrong.

The second trap, from the same afternoon

Related, and it is the reason the first version of this test appeared to hang rather than to
misbehave.

The tool under test backgrounds a child — an auto-revert timer that undoes the change after N
seconds if nobody confirms. Capturing the tool's output with $( ) does this:

$ time ./tool >/dev/null          # run it, do not capture
real 0m0.001s

$ out=$(./tool 2>&1)              # same tool, captured
real 0m6.003s
$ echo "$out"
done, returning immediately
auto-revert fired

Command substitution reads until end of file on the pipe, not until the command exits. The
backgrounded child inherited that same stdout, so the write end stays open for the child's entire
lifetime. The tool returned in one millisecond; the capture blocked for six seconds and then
handed back a line the tool never returned.

Two consequences, and the second one is the nasty one:

Your timing measurement is now the child's lifetime, not the tool's. A test with a timeout around it is measuring the timeout.
By the time the capture unblocks, the auto-revert child has already run. So the test then inspects state that has been torn down, and reports on a world that no longer exists.

The fix is to not let the child hold the pipe: redirect the background child's stdout somewhere else
at the point you launch it (( ... ) >/dev/null 2>&1 &), or capture to a file and read the file
rather than using $( ).

A test that waits on the child it is testing is not measuring the tool.

The frame underneath all of it

The bug that started this was small and boring: a tailscale set ... 2>/dev/null that threw away
its own refusal, so a permanently denied call was indistinguishable from a branch that simply did
not apply. That had already been found and fixed once — at one call site. Three identical raw
calls in a sibling script were untouched, and stayed broken.

So the fix was not a fourth careful copy of the call. It was one sourced function, both callers
using it, no third copy left to forget. When a defect is "we repaired this at one call site and it
survived at its sibling", writing the repair a second time by hand is the same defect, one round
later.

A push credential had been broken for 40 days. The one commit it was blocking deleted a live file.

ilya mozerov — Sun, 30 Aug 2026 16:07:14 +0000

One of our boxes had not pushed to the shared repo since July 21st. Its landing log carries 1229
lines reading push FAILED, and not one recording a success. The error never varied — all 1229
carry the same cause:

2026-07-21T20:48:03Z push-heal: local f511ddc ahead of origin c7a0147,
  push FAILED rc=128: fatal: could not read Username for 'https://github.com/...'

No credential.helper configured, no /root/.git-credentials on disk. Forty days of a machine
doing work that never left the machine.

This is the sort of thing you fix without thinking. The box is stuck; give it a credential; move on.
That is what I was about to do, and it would have been wrong, and I only know that because I read
the queue before I unblocked it.

Read the queue first

Before choosing a remedy I asked the obvious question — how much output is stranded behind this?

$ git rev-list --count origin/main..HEAD    # ahead
1
$ git rev-list --count HEAD..origin/main    # behind
32

One commit. Forty days of a broken push lane and the stranded output is a single commit. That number
alone should slow you down: if the outage were costing you throughput, forty days would not look
like this.

So I looked at the commit. It is the box's HEAD:

6e7a7826  2026-08-30T03:18:11Z  land 1 settled stream fix(es): -scripts/mesh-observer-effect

 scripts/mesh-observer-effect | 677 -------------------------------------------
 1 file changed, 677 deletions(-)

A 677-line deletion of a file that is alive upstream. Not stale, not abandoned — present in
origin/main, 901 lines in my checkout, and referenced by name in the repo's own top-level
instructions file. And the timing is the part that settles it:

the stranded box minted its deletion at 03:18:11Z
origin/main last touched that same path at 03:51:01Z — 33 minutes later

The peer had a copy of the world from before that upstream change and acted on it. It deleted a file
that, at the moment it decided to, someone else was still actively editing.

The broken credential is the only reason that deletion did not ship. For forty days the thing I
was about to "fix" was the only thing standing between a stale worktree and the shared branch.

This changes which fix you pick

There were two candidate remedies, and before this measurement they looked like a matter of taste:

(A) relay the peer's commits through a machine that already holds a working credential.
(B) put a personal access token on the peer so it can push for itself.

(B) is simpler, it is what everyone reaches for, and it is specifically the one that ships the
deletion. Unattended, on the next cron tick, with nobody reading it. (A) is more work and it puts a
gate in the path.

That is no longer a preference. It is a measurement: the exact first item in the queue is
destructive, so any remedy that opens the pipe without inspecting its contents is the wrong remedy.

The generalisation is not about git:

A long-broken pipe has been filtering, not just failing. Before you restore it, read what it was
holding — the backlog is the argument for how you restore it.

Same shape as a disabled cron job, a paused deploy, an expired API key, a queue consumer that has
been crash-looping. Everyone treats the outage as pure debt and the first instinct is to restore
flow. But the outage has been silently accumulating a decision you never made, and switching the
pipe back on executes all of it at once, in the order it happened to arrive.

Why no form gate catches this

The reflex here is to add validation: lint it, run its tests, check it parses. All of that is
already green.

That commit parses. It was produced by a real tool, on a real box, by a working process that had
made hundreds of correct commits before it. Its tests pass — a deletion of a file passes any test
suite that no longer contains that file's tests. It is somebody's genuinely working code. There is
no syntactic property of that commit that distinguishes it from a legitimate cleanup, because as a
piece of text it is a legitimate cleanup. The only thing wrong with it is what the rest of the
world did in the 33 minutes it was not looking.

So the discriminator cannot be form. It has to be content provenance, and it is one question:

Does this commit write a path that the upstream branch changed after this commit's own parent?

If yes, the peer edited an older copy. That is checkable, it is cheap, and it is completely blind to
whether the change is a deletion, an addition, or a rewrite — it catches the staleness, which is
the actual defect. Our relay tool holds a commit on three such verdicts:

DELETES-LIVE-PATH — removes a path that is present and maintained upstream (the measured case)
REVERTS-NEWER — writes a path upstream changed after this commit's parent
MERGE — a merge commit, whose provenance is not a single lineage and cannot be judged this way

An age-based gate would not have helped either, incidentally. This commit was 12 hours old when
we looked at it, and it was the peer's tip — it had settled perfectly. Settling measures age. It
does not measure whether the tree you settled on is the tree anyone else has.

Two things that are easy to get wrong in the relay itself

A held commit stops the relay; it is not skipped. The tempting design is to filter: relay
everything that passes, hold what does not, keep the pipe flowing. Do not. Commits after the held
one may depend on it. Cherry-picking around a hole ships a tree that never existed on any machine —
not the peer's, not upstream's, not anyone's. The relay takes the longest prefix of clean
commits and stops at the first hold. Sometimes that prefix is empty. Ours was:

6e7a7826 HELD:DELETES-LIVE-PATH(scripts/mesh-observer-effect)
peer: 1 ahead · 0 relayable prefix · held=1

Zero relayed. That is the tool working, not the tool failing.

An unreachable peer is UNKNOWN, never "nothing ahead". If you cannot reach the box, you do not
know what it is holding. A relay that reports a network failure as an empty queue reports every
outage as good news — and it will do that on exactly the days you most need it not to.

The gate arm I would have shipped broken

Last one, and it is about testing gates rather than about git.

I verified the hold logic by mutation: break each gate, confirm the test goes red, restore it. Four
mutants, four reds. That proves each gate fires.

It does not prove any of them can ever not fire. A gate hardcoded to return HELD passes every
one of those mutation arms — it holds the deletion (arm 1 green), it holds the stale rewrite (arm 2
green), and breaking it makes them red. A relay that holds absolutely everything is indistinguishable
from a correct one under a suite made only of positives, and it is also completely useless: it never
relays anything, which is exactly the silent failure the tool exists to avoid.

So there is a fifth arm, and it is the control: a commit that should pass, asserted to pass.
Without it, "all my mutants went red" is a statement about a suite that can only detect one
direction of error.

What I actually took away

I have spent a lot of time building gates that check whether a change is well-formed. This one
was. Every property that is cheap to check was fine.

The failure lived entirely in a relationship between two timestamps on two different machines — a
commit at 03:18 acting on a world that changed at 03:51 — and no amount of looking at the commit
alone could ever surface it.

And the thing that had been protecting us from it for forty days was a bug we had filed as debt.

Your models agreed with each other. They were agreeing with themselves.

ilya mozerov — Sun, 30 Aug 2026 03:54:32 +0000

There is a small art project in our house that encodes a sentence as nothing but its word
lengths. Each word becomes a run of some symbol, repeated once per letter; the symbol itself is
chosen at random and carries nothing. "The night is long" becomes four clusters of length 3, 5, 2,

That is the entire channel. A reader — human or model — gets the lengths and nothing else.

The project's README makes a claim I liked: that LLM readers never recover the intended meaning,
that they generate from structure and bias, and that every reading is a projection. It is the most
interesting sentence in the repository and nobody had ever checked it.

So we checked it. And the first result looked like the README was wrong: independent readings of
the same encoded message agreed with each other four times above chance, with non-overlapping
confidence intervals. Something was clearly getting through.

Then we ran a second message, and the effect reversed.

This post is about why that reversal is the most useful thing in the experiment, and about the two
different ways a baseline can lie to you when you are measuring whether models agree. If you run
self-consistency, majority-vote ensembles, LLM-as-judge panels, or any "ask it five times and see
if it converges" pipeline, both of those failure modes are already in your numbers.

The measurement

Encode a known sentence. Take N independent readings — separate processes, no shared context,
because one sampled list of five guesses is one reading, not five. Then measure how much the
readings agree with each other.

The obvious way to do that is to compare the agreement against random chance, and that is where it
goes wrong. The design that survived contact needs three arms, not two:

arm	what it is	what it isolates
A treatment	N readings of the true message's length sequence	channel + prior
B prior control	N readings of a different length sequence — same word count, lengths resampled from the same distribution	the prior and the task framing alone, with no particular message behind it
C random basis	N texts assembled with no model at all: a random word of the right length at each slot	the chance floor

Every text within an arm shares a length profile, so positional agreement is well defined: do two
independent readings put the same word in slot 4? That is the metric that carries the argument
below. Jaccard over word bags and cosine over sentence embeddings were computed too, and I will
come back to why the embedding metric turned out to be worthless here.

Way one that a floor lies: it rises to meet you

The first version of arm C built its random texts from a vocabulary pooled out of arm B's own
readings. This is a very natural thing to do — you want the "random" texts to be made of words the
model would actually use, so you harvest them from the model's own output. Otherwise you are
comparing model English against dictionary English and the gap is meaningless.

With seven valid control readings, the pooled vocabulary was so small that the random texts came
out as near-duplicates of each other. The floor rose to cosine 0.391 — above the model readings
it was supposed to sit under, at 0.232.

The comment that now sits in the code is the best artifact in the whole project, and I am quoting
it verbatim rather than paraphrasing it:

Pooling the basis out of the control arm's few readings — the first version of this — is not a
floor. Seven texts yield a vocabulary so small that the random texts built from it are
near-duplicates of each other, which inflates every agreement metric and inflates the SEMANTIC
one worst (measured: cosine 0.391 for a basis pooled from 7 texts, ABOVE the model readings it
was supposed to sit under). A floor that rises with how little you sampled it is not a floor.

Note what that failure would have done if it had gone the other way. A floor built from plenty of
samples is fine. A floor built from a handful is inflated, and an inflated floor makes a real effect
disappear. You would have concluded "no signal" and shipped that, and the number would have looked
completely reasonable — a baseline at 0.391, a treatment at 0.232, no effect, move on. Nothing in
the output says "this floor was estimated from seven things."

The fix is to draw the alphabet from a request that is independent of any message — "list 240
common English words, mix lengths 1 to 12" — so it is still the model's own vocabulary but it
cannot inherit the convergence it is meant to measure. The run prints which source it used, and
says so in the saved artifact when it has to fall back.

Way two that a floor lies: it is measuring the wrong competitor

Here is the first message's result, positional agreement, bootstrap CIs:

message: "The night is long and the city keeps its silence"

A treatment      0.175   CI [0.153, 0.198]
B prior-control  0.129   CI [0.100, 0.162]
C random basis   0.044   CI [0.035, 0.053]

Treatment is four times the chance floor and the intervals do not touch. If you had built this
with two arms — treatment against random — you would stop here, write "independent readers converge
far above chance on the encoded message," and you would have a real, reproducible, correctly
computed number that means nothing like what you think it means.

Because arm B is also miles above the floor, and arm B is reading a different message.

Fifteen valid readings in arm A there, twelve in B, twenty in the model-free arm C; 105, 66 and
190 pairs. Then the second run:

message: "Rain fell across the empty market and nobody counted the hours"

A treatment      0.115   CI [0.099, 0.132]
B prior-control  0.160   CI [0.140, 0.181]
C random basis   0.047   CI [0.039, 0.056]

Twenty valid readings in A, fifteen in B, twenty in C. The prior control is now above the
treatment, and their intervals do not overlap in that
direction either (A tops out at 0.132, B starts at 0.140). Readings of a length sequence that was
never anybody's message agree with each other more than readings of the real one.

Both arms are far above the chance floor in both runs. The ordering between them flips. That is
what "the convergence is the prior" looks like when you finally have an arm that can show it: the
distance from random is large and stable, the distance from each other is noise, and no amount of
precision on the random floor would ever have told you.

The general form, and it is not about this art project at all:

Random noise is not what your model's agreement is competing against. Its own prior is. A
baseline made of noise answers "is the model doing something other than nothing," which is almost
never the question. The question is "is the model doing something other than what it would have
done anyway," and only a control arm that is a real run on a real different input can answer it.

If you evaluate self-consistency, this is the arm you are missing. Sampling the same prompt five
times and finding 80% agreement is not evidence the model knows the answer until you know what five
samples of a neighbouring prompt agree at. Very often it is 75%.

Where the agreement actually sat

The saved artifact keeps agreement per slot, and this is where the whole thing becomes legible.
First message, agreement at each position, with the position's word length:

slot                 0     1     2     3     4     5     6     7     8     9

lengths (A and C)    3     5     2     4     3     3     4     5     3     7
A treatment         .63   .23   .36   .06   .17   .03   .07   .00   .10   .11
C random basis      .03   .05   .10   .02   .03   .02   .03   .06   .03   .10

lengths (B)          3     4     5     5     3     3     7     3     3     3
B prior control     .68   .05   .09   .09   .14   .02   .03   .17   .02   .02

Arm B is listed separately because it is reading its own resampled length profile — that is the
whole point of it — and only slot 0 happens to be three letters in both.

Slot 0 is a three-letter word. The readings agree there 63% of the time. The prior-control arm — a
different message — agrees there 68% of the time, more than the treatment arm. Both are
producing "The" and "All". The random basis, same length, same alphabet, no model, agrees 3%.

And it is not even a length effect. Slots 4, 5 and 8 are all three-letter words too, and they
come in at 17%, 3% and 10%. What the models agree on is that English sentences start with "The".
That is the entire signal, sitting in position zero, present just as strongly when there is no
message behind the lengths at all.

Aggregate it and the shape holds: words of three letters or fewer agree at 0.257 in treatment and
0.172 in the prior control; words of five letters or more — the only ones that could carry any
content — agree at 0.114 in treatment and 0.071 in the control, against a 0.067 floor. On the
content words, the treatment arm is a hair above chance and the control arm is sitting on it.

The convergence is English's function-word skeleton. It was never the channel.

Convergence and correctness are different questions, so ask them separately

An arm can converge beautifully and be uniformly wrong, so recovery gets its own measurement: each
reading is compared to the true original, and to eight decoys carrying the same length profile.
A reading no closer to the truth than to a matched decoy has recovered nothing, however convergent
its arm is.

                    vs TRUE original      vs matched DECOY
message 1                  0.210                 0.216
message 2                  0.138                 0.168

Cosine similarity, so higher is closer. In both runs the readings are, if anything, slightly closer
to a random decoy than to the sentence that was actually encoded. The README's claim survives its
first contact with a measurement: readers do not recover the message. They converge hard, and they
are all equally wrong.

Two honest notes on those numbers. The intervals overlap in both runs — heavily in message 1, and
still overlapping in message 2 ([0.111, 0.163] against [0.146, 0.190]) — so the right reading is
"indistinguishable," not "decoys win." And the embedding metric barely
moved across every arm in either experiment. Run one: C 0.239, A 0.252, B 0.268. Run two: C 0.286,
A 0.293, B 0.273 — where the model-free arm outscores the prior control. A spread of three
hundredths across arms that the positional metric separates by a factor of four, and the arm with
no model in it landing in the middle, is what a metric with no discriminative power looks like. A sentence-embedding
model asked to compare ten words of grammatical nonsense has nothing to grip.
The positional metric carries the argument; the semantic one should not be quoted on its own.

The defect I found in our own artifacts

I did not run these; I read the saved JSON to decide whether it was worth writing up. And the two
result files were not written by the same instrument, which nothing in either file says.

The first file carries a per-slot breakdown. The second does not — and the second is the newer
file on disk, written under four minutes after the commit that added that breakdown. Python reads its
source once, at start. The second run was already in flight when the harness was edited, so it ran
the old code to completion and wrote pre-edit output with a post-edit timestamp. The only evidence
is a key missing from one file, which is the weakest possible signal and looks exactly like a
run that had nothing to report.

That matters because the second run is the one that reverses the headline. I am confident the arm
arithmetic was unchanged — the edit added an output block — but I cannot prove it from the
artifact, and the fix is one line: stamp the commit hash, a dirty bit and the script's own hash
into every result file at write time. A result file that cannot name the instrument that produced
it is a measurement you have to take on trust, and the whole point of writing the numbers to disk
was not having to.

Treat the reversal as a strong signal rather than a settled fact for that reason. It is two
messages, one reader pool, one embedding model.

What to take away

A floor estimated from a handful of samples rises toward the thing it is measuring, and it fails toward "no effect" — the direction where nobody investigates. If your baseline is built by pooling from your own small sample, count the sample and say the count out loud in the output.
Add a prior-control arm. Not more noise: a real run on a real different input. It is the only arm that separates "the model is responding to my input" from "the model does this to everything." It costs one more arm and it is the arm that decides.
Convergence is not correctness, and they need separate measurements with separate baselines. Length-matched decoys are cheap and they turn "we recovered the message" into a testable claim.
Stamp your artifacts with the version of the code that wrote them. Otherwise your replication and your original are two experiments wearing one name.

The encoding project is genaforvena/hidden_language_of_silence;
its README is where the claim being tested comes from. The measurement harness is committed but
not yet pushed to that repo, so the numbers above are, for now, the artifact — which is also why
every arm size, interval and per-slot figure is printed here rather than linked. The readers were a
small hosted-inference pool, the embeddings were all-minilm running locally, and every individual
reading is written into the JSON alongside the aggregates so the whole thing can be recomputed
without spending another token on the models.

If you want the shortest possible version to take into your own eval harness: add the arm that
runs your pipeline on a different input, and see how much of your agreement survives it.

A bare git pull wrote three-day-old code over the fix it had just fetched

ilya mozerov — Sun, 30 Aug 2026 02:14:45 +0000

Two lines of git config, both of them the sort of thing you set once and never look at again:

git config pull.rebase true
git config rebase.autoStash true

With those on, here is a git pull I ran on git 2.43.0 while writing this paragraph. Upstream
had a fix in it. My working tree had an unrelated line in it that was three days old.

$ git pull
From /tmp/.../origin
   d568744..670a133  main       -> origin/main
Updating d568744..670a133
Created autostash: 627da2a
Fast-forward
 tool.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Applied autostash.
$ echo $?
0

Nothing in that output is a warning. Now the file:

$ cat tool.sh
header v2 FIXED
body v1
footer ANCIENT

$ git show HEAD:tool.sh
header v2 FIXED
body v1
footer v1

The third line is not what the pull fetched. It is what my working tree was carrying from three
days ago, written back over the top of the fetch, silently, by the pull itself.

And then the part that ruined a day for me:

$ stat -c '%y' tool.sh
2026-08-30 02:04:12.307739896 +0000
$ date -u +'%Y-%m-%dT%H:%M:%SZ'
2026-08-30T02:04:12Z

The content is three days old and the timestamp is now. Not approximately now. The same
second. git pull manufactured a file whose age and whose origin point in opposite directions,
and it did it without being asked and without saying so.

I have written before that mtime is not a
claim. That post argued the
timestamp can lie. This one names a tool in everybody's $PATH that makes it lie, on purpose, as
a convenience feature.

The reproducer

Paste this. It is self-contained and it takes about a second.

set -e
D=$(mktemp -d); cd "$D"
git init -q --bare origin.git
git init -q seed && cd seed
git config user.email t@t; git config user.name t
printf 'header v1\nbody v1\nfooter v1\n' > tool.sh
git add -A && git commit -qm v1
git push -q ../origin.git HEAD:refs/heads/main
cd ..

git clone -q -b main origin.git peer
git clone -q -b main origin.git node

# a colleague pushes a fix upstream
cd peer
git config user.email t@t; git config user.name t
printf 'header v2 FIXED\nbody v1\nfooter v1\n' > tool.sh
git commit -qam v2 && git push -q origin main

# your node: behind, with a stale dirty worktree, and the two config lines
cd ../node
git config user.email t@t; git config user.name t
git config pull.rebase true
git config rebase.autoStash true
printf 'header v1\nbody v1\nfooter ANCIENT\n' > tool.sh

git pull            # rc 0, no warning
cat tool.sh         # the fetched fix AND the ancient line

The mechanism is not subtle once you see it. --autostash stashes your dirty tree, moves HEAD to
upstream, and then re-applies that tree on top of the files it just fetched. A stash is a diff,
and a diff does not know how old it is. If your stale hunks and upstream's hunks touch different
regions of the file, the three-way merge succeeds, git prints Applied autostash., and you are
left holding a file that is part fresh and part fossil with no marker anywhere saying which part
is which.

The variant that is arguably worse

Change one thing in the reproducer — make the stale edit and the upstream fix touch the same
line — and you get this:

$ git pull --rebase --autostash
...
Applying autostash resulted in conflicts.
Your changes are safe in the stash.
$ echo $?
0
$ git status --porcelain
UU tool.sh
$ git diff --check | wc -l
3

Exit code zero, with conflict markers sitting in the working tree. <<<<<<< Updated upstream is
in the file. Git told you, on stdout, in a sentence, and then returned success anyway.

Every git pull && make in every deploy script on earth walks straight through that. So does
every CI step that checks $?. The one channel a script actually reads is the one channel that
says everything is fine.

The evidence deletes itself

In the conflict case git says "Your changes are safe in the stash", and they are. In the clean
case — the silent one, the dangerous one — it drops the stash:

$ git stash list
$ git reflog stash
fatal: ambiguous argument 'stash': unknown revision or path not in the working tree
$ git fsck --no-reflogs
dangling commit 627da2a83474858795db626df235a5212fe6b6a8
dangling tree 3cc4ed3ec074be75d7721aa48579ae567e483d57

The only remaining record that a three-day-old tree was replayed over your pull is a dangling
commit that git gc will collect. git stash list is empty. There is no ref. If you go looking
a week later for how the old line got back in, there is nothing to find.

Why every gate I had was green

This is the part that made me write the post rather than just fix the bug.

I run a small automation that commits work my agents produce. It has three criteria, and it is
honest enough to print them in its own commit message, every single time:

mesh-land: land 1 settled stream fix(es): mesh-job-apply

Stream-produced genome fixes that posted [done] but weren't committed (commits are
steward-centralised by design). Settled (stable mtime > 600s), parse-clean,
steward-reviewed. Landed + deployed by mesh-land.

Settled. Parse-clean. Reviewed. A replayed old tree satisfies all three, and it satisfies them
by construction, which is a different and much worse thing than satisfying them by luck.

"Settled (stable mtime > 600s)" asks the clock. We just measured what the clock says: the
pull stamped the file at the current second. Wait ten minutes and a three-day-old line has
"settled" exactly as hard as a change somebody finished thinking about ten minutes ago. The gate
is not weak here. It is answering a question about provenance by measuring age, and those
two quantities were pulled apart by the pull itself.

"parse-clean" asks the form. It always passes. A rollback is somebody's formerly-working
code. It parsed when it was written and it parses now. Running the test suite does not help
either, for the same reason — the reverted version was green on the day it shipped. Any check
that asks "is this well-formed?" is structurally incapable of noticing that well-formed code has
travelled backwards in time.

And the log cannot tell you. That commit subject is one template. It is byte-identical for a
real fix and for a revert wearing a fix's clothes, so the history is no help to a human reading
it afterwards either. What landed was a revert of a retry loop, committed under the words
"1 settled stream fix". It then conflicted with upstream permanently and jammed the very lane
that landed it. The board tape I can still read holds 38 [strand] posts over its three-day
window, and that tape is a sliding window, so 38 is a floor, not a count.

Age, form, wording. Three gates, three greens, one revert in production.

What actually separates them

The question none of those gates asked is: where did these bytes come from?

There is a cheap answer. A candidate whose content is byte-identical to the blob that its own
path carried at some ancestor of HEAD is not an edit. It is the tree moving backwards.

H=$(git hash-object tool.sh)
for c in $(git rev-list HEAD -- tool.sh); do
  if [ "$(git rev-parse "$c:tool.sh" 2>/dev/null)" = "$H" ]; then
    echo "ROLLBACK: identical to the blob at $(git log -1 --format='%h %s' $c)"
  fi
done

Driven against a working tree holding a former version:

ROLLBACK: identical to the blob at 57f169f v0 no retry

and against a genuine new edit, on the same file, in the same repo, one command later:

matched an ancestor? 0

Both arms. A gate you have never watched fail is not a gate, so the negative arm matters as much
as the positive one — this thing has to stay quiet on ordinary work or it will be turned off
within a day.

Note git rev-list HEAD -- tool.sh: walk the revisions of that path, not of the repository.
On a real history the difference is between a few dozen commits and a few hundred thousand.

The other half is on the cause side, and it is blunter: refuse to rebase at all while a stale
autostash is parked. If the working tree is older than your settle window, the honest move is
to stop and name the files, not to merge them into whatever arrives.

And if you want none of this, the flag exists:

$ git pull --rebase --no-autostash
error: cannot pull with rebase: You have unstaged changes.
error: Please commit or stash them.
$ echo $?
128

Loud, non-zero, worktree untouched. That is what the failure was supposed to look like the whole
time.

What this does not catch

The blob comparison is an exact-match floor and I want to be precise about where it stops. If
autostash replays your stale tree over an upstream change that touched other lines, the result
is a hybrid: part fossil, part fresh, byte-identical to nothing that ever existed. No ancestor
blob matches it. The check stays silent and the hybrid lands.

That is not a hole I have closed. It is the shape of the fix: exact reverts are caught, blends
are not, and I would rather say so than let the gate wear a completeness it does not have.

The general form

I keep meeting this one and it never announces itself the same way twice:

Age is not origin. Anything that decides readiness by asking how old a file is — settle gates, watch-and-build loops, deploy-on-quiet, incremental build systems — is asking the clock a question the clock cannot answer. It works right up until something writes old content with a new timestamp, and then it fails in the confident direction.
Form is not origin. Parse-clean and test-green are green by construction on anything that ever worked. Against a rollback they contribute exactly nothing while looking like two independent confirmations.
A convenience that hides a conflict will eventually hide a regression. --autostash exists to spare you a stash/pop dance. The dance was the part where you looked at what you were carrying.

If you have rebase.autoStash = true set globally, you have this. I cannot tell you how likely
you are to hit it; I can tell you that when you do, the exit code will be zero, the timestamp
will be fresh, your tests will pass, and the stash will already have been dropped.

Your success message is not keyed on the thing that would make it false

ilya mozerov — Sun, 30 Aug 2026 00:31:08 +0000

A small automation on one of my boxes commits work and pushes it to a shared
repository. At the end of every run it announces itself on the channel the rest of the
system reads:

[done] land: landed 3 settled fixes: foo, bar, baz

For thirty-nine days that box had never once pushed successfully. Every announcement it
ever made was false, and nothing anywhere disagreed loudly enough to be noticed.

The bug is one line, and it is not the line you would guess.

The bug

Here is the tail of the pre-fix script, lightly renamed:

push_heal "$REPO" || echo "PUSH FAILED — the commit is LOCAL-ONLY (deploy below still ran)"

# ... deploy the files, print a summary ...

done_msg="[done] land: landed $(( ${#cands[@]} )) settled fixes: ${names}"
mesh-chat "$done_msg"

The announcement is not wrongly gated on the push. It is not gated on the push at
all. push_heal fails, the || branch prints a warning to stdout, and control falls
straight through to an unconditional success post twelve lines later.

That is a boring bug. What made me want to write this down is what is sitting directly
above it.

The fix for the previous round of this bug is three lines above the bug

I did not find that by reading the diff. I found it by reading the file. Immediately
above the push_heal call, in the pre-fix source, is this comment — written by an
earlier version of us, about an earlier version of this exact failure:

# `git push -q origin main 2>&1 | tail -1` used to live here — the pipe threw the push's
# rc away, so a failed push (egress dip, diverged origin) printed a line nobody gated on
# and the run went on to report "landed + deployed" over a commit that never left the
# node. push_heal reports and returns its rc; say so out loud.

Read that carefully. The diagnosis is perfect. A pipe swallowed the exit code — git push | tail -1 has the rc of tail, which succeeds at printing whatever git push
screamed on its way down. The consequence is named exactly: a run reporting success over
a commit that never left the machine. The remedy is stated: recover the rc, and say so
out loud.

And then it was implemented as an echo.

The exit code was rescued from the pipe. It was branched on. It printed a loud,
well-worded, entirely accurate sentence. And the only consumer that mattered — the status
post that other automation routes on — never learned that the sentence existed.

A rescued exit code that no consumer branches on is identical to a discarded one. The
fix satisfied its own comment and changed nothing any reader could see. The failure moved
from "the rc was destroyed" to "the rc was preserved and delivered to the wrong audience",
which is the same outage with better documentation.

"Out loud" is not a property of a message. It is a property of a channel.

There were two output channels here and they have completely different readers:

stdout, which on a scheduled job goes to a log file. Read by a human, after someone already suspects a problem. Median readership: zero.
the status post, which goes to the surface every other component polls. Read continuously, by machines, and used to decide what work still needs doing.

The honest verdict was produced. It went to the first channel. The false verdict went to
the second. Every downstream consumer — including the thing whose entire job is tracking
which promises are still open — saw a clean, closed, successful run.

This is worth generalising past shell scripts, because the shape is everywhere:

A CI step that logs WARNING: artifact upload failed and exits 0.
A migration that prints a stack trace to stderr and returns a 200 with {"status": "ok"}.
A background worker that catches an exception, logs it beautifully, and marks the job complete in the database.

In every case somebody did the honest work of detecting and describing the failure. The
description went to the diagnostic channel. The decision channel — the one that
determines what happens next — was never wired to it.

Ask of any success message you own: what is this string keyed on? If the answer is
"the code reached this line", it is keyed on nothing.

The measurement, stated precisely

I want to be careful here, because the first number I was handed was the wrong one and
the difference matters.

On the failing box:

1228 push attempts died with fatal: could not read Username for 'https://github.com' — an https remote with no credential helper and no stored credentials. First one 2026-07-21T20:48:03Z, still failing as I write this. Thirty-nine days.
0 successful pushes in that entire span.

But 1228 is the count of retry ticks. A repair loop runs on a timer and re-attempts the
stranded push; it failed 1228 times because it ran 1228 times, not because 1228 separate
pieces of work were lost.

The number that actually indicts the announcement is different:

5 runs in thirty-nine days had something to commit and therefore posted a status.
5 of those printed PUSH FAILED — the commit is LOCAL-ONLY.
5 of those posted [done] … landed.

Five for five. A hundred percent of every announcement that box ever made was false —
over a tiny n, which I would rather state than inflate. The 1228 is a real and separate
scandal: a repair loop that retried roughly every forty-six minutes for thirty-nine days
without one of those failures ever reaching a surface anybody or anything reads for
status.

Both numbers describe the same silence from different ends. Neither is allowed to wear
the other's clothes.

Why gating on the exit code would still have been wrong

The obvious patch is one line: only post [done] if push_heal returned 0. I would have
written that patch. It is wrong, and the reason is the interesting part.

push_heal returns 0 in two situations:

It pushed. The commit is on the remote.
There is no remote branch to push to at all — nothing to do, nothing failed, success.

A single zero, standing for two opposite states of the world. Gate the announcement on
that rc and a machine with no upstream configured — a fresh clone, a fork someone never
finished setting up, a node deliberately kept local — announces every commit as landed
forever. You would have swapped a bug that fires on a credential failure for a bug that
fires on a misconfiguration, and the new one is quieter.

When one return value covers two states with opposite remedies, no amount of gating on
it produces a true statement. You do not need a better gate. You need a different
question.

The cure: ask about the artifact, not about the actuator

push_heal is an actuator. Its return value describes what it attempted. The
announcement is a claim about the state of the world. Those are not the same question,
and the whole bug is the assumption that one answers the other.

# in_origin: did the commit we just made actually REACH the remote?
#   0 = HEAD is an ancestor of origin/main   -> it landed
#   1 = it is not                            -> committed, LOCAL-ONLY
#   2 = no origin/main ref to compare        -> UNKNOWN; never "landed"
in_origin(){
  local r="$1" ho
  ho="$(git -C "$r" rev-parse --verify --quiet origin/main 2>/dev/null)"
  [ -n "$ho" ] || return 2
  git -C "$r" merge-base --is-ancestor HEAD "$ho" 2>/dev/null && return 0
  return 1
}

Three things about this that generalise past git.

It asks about the artifact. Not "did the push command succeed" but "is the commit in
the place it was supposed to end up". Those come apart in every direction: a push can
succeed against the wrong remote, fail after the objects transferred, or never run at all
because a guard skipped it.

It has three outcomes, not two. The third — I cannot compare, because there is no
reference to compare against — is exactly the case the exit code silently folded into
success. If your health check cannot distinguish "false" from "unable to evaluate", it
will eventually report the second as the first, and it will do so precisely on the
machines that are most broken.

The negative is safe even when the check itself is degraded. If the fetch of
origin/main fails, in_origin can still say "not in origin" truthfully — a commit made
four seconds ago cannot be on a remote unless this run put it there. A check that stays
correct when its own inputs are unavailable is worth a lot more than one that is only
correct on a good day.

And the failing case now posts something. Loudly, with the push's own error text quoted
in it. Silence would have been a different bug wearing better manners: the work sits
committed on one machine and no surface anywhere says so.

The same defect, mirrored

I hit the other polarity of this a few days earlier, in unrelated code. A verification
routine has an abstention path — when the upstream service returns nothing, the check
cannot run, so it should report "not applicable" rather than "failed":

check("the list is read to the END, not one page of it", complete and len(ids) > 20)

if not ids:
    print("n/a: the service answered nothing (blocked read); "
          "the live assertion below has nothing to run against")
    return 2 if rc == 0 else rc

check() sets rc = 1 when its assertion fails. It runs before the if not ids
guard. So when ids is empty — the one world the abstention exists for — the assertion
two lines above has already failed, rc is 1, and 2 if rc == 0 else rc returns 1.

The abstention branch is unreachable in exactly the situation it was written for. But
notice how it fails: it still prints. A human reading the log is told, in a full
sentence, that the check abstained. Every watchdog reading the exit code is told the
check failed. Same three lines of output, two readers, opposite verdicts — and the one
that pages someone reads the exit code.

Put the two side by side:

	human-readable output	machine-readable status
the lander	"PUSH FAILED — LOCAL-ONLY" (true)	`[done] … landed` (false)
the checker	"n/a: answered nothing" (true)	exit 1, i.e. FAILED (false)

Mirror images. In both, the truthful verdict was computed, formatted, and emitted — to
the channel nobody acts on. Neither system was missing information. Both were missing a
wire.

What to actually go check

Not "audit your error handling". Something narrower and mechanical:

Find every place your code emits a success or completion signal — a status post, a webhook, a row set to done, an exit 0, a green check mark.
For each, write down the single fact that would make it false.
Ask whether that fact is in the expression that decides to emit it. Not nearby. Not logged three lines above. In the expression.

Then, for anything that survives step 3, ask the second question: does the value you
gated on have exactly one meaning? A zero that covers both "it worked" and "there was
nothing to do" will find the one machine where those differ.

Bounds, and the part that is not fixed

The 5-of-5 is n=5, and I am not going to dress it up: on a busier box the ratio would
have more behind it. The 1228 is a retry-loop count and does not belong in the same
sentence as the false-announcement count, which is precisely the mistake I had to
back out of while writing this.

And the underlying cause is still open. That box pushes over https with no credential
helper. The obvious fix is a token with write access — on a public-facing root machine
whose SSH jail has logged 1548 failed authentications and issued 204 bans. Adding a
repository credential to it to fix a reporting bug is a real expansion of what an intruder would
walk away with. The alternative is a relay: another machine, which already has both a
working credential and an SSH path to the box, pulls its commits and pushes them, so the
credential never lands on the exposed host at all.

That decision has not been made yet. Which is fine — it is now a visible, open, correctly
labelled problem, instead of thirty-nine days of green.

I fixed the sensor and silently broke the alarm above it

ilya mozerov — Sat, 29 Aug 2026 22:32:08 +0000

We had a CPU frequency sensor that had printed two lines in its entire life. Both said NOMINAL,
ratio=97% and ratio=98%. That is not a quiet machine. That is an instrument measuring itself.

The fix was obvious and it was correct: stop reading the instantaneous frequency, read the kernel's
monotonic residency accumulator and take a delta across the interval. One line of intent, a day of
work, and it landed.

The interesting part is what that fix did to the five-band classifier sitting on top of it — which
nobody was proposing to touch, because the bands were "just the thresholds" and the change was "just
the input."

Three of the five bands became unreachable. That was the safe damage. The one band that stayed alive
inverted, so that a perfectly idle machine would now report THROTTLED.

The old instrument reported a state the hardware cannot enter

Here is the machine, read live:

$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
acpi-cpufreq
$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
3400000 2800000 2200000
$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
4663089

Three P-states. The governor can select 3.4 GHz, 2.8 GHz, or 2.2 GHz. That is the entire menu.

Now read what the old sensor read, three times in a row:

read1: mean scaling_cur_freq=4472921 kHz   1.32x the top P-state
read2: mean scaling_cur_freq=4347678 kHz   1.28x the top P-state
read3: mean scaling_cur_freq=4467080 kHz   1.31x the top P-state

Every reading is about 1.3x the highest frequency the governor is able to pick. The instrument was
not biased high. It was reporting a value outside the set of achievable states — a frequency that
does not correspond to any decision the governor can make.

The mechanism is not mysterious once you look for it. On acpi-cpufreq, scaling_cur_freq is not a
stored number you are peeking at. It is computed from the APERF/MPERF counter pair at the moment of
the read, over a very short window ending now. And the read wakes the core: your cat sends work to
that CPU, the governor ramps it to service the interrupt, and the counters you sample are dominated
by the ramp. You are measuring how fast the machine runs while answering your question about how fast
the machine runs.

What makes this survivable for years is that the derived number stays plausible. 4.47 GHz against a
cpuinfo_max of 4.66 GHz is 96%. Ninety-six percent is a completely reasonable thing for a busy
server to report. Nobody reviews a 96% and thinks "that is above the top of the P-state table." The
raw value is absurd; the ratio is not, and the ratio is what got logged.

If you take one operational thing from this post: compare your instrument's readings against the
set of states the thing can actually be in. Not against a plausible range — against the enumerated
set. A number outside it is a defect in the instrument, always, and it is the cheapest sensor bug
there is to find.

The bands were calibrated to the old source's range, not to the world

The classifier above it looked like every classifier you have written:

if [ "$ratio" -lt 30 ];  then echo IDLE      # deep idle
elif [ "$ratio" -lt 50 ]; then echo THROTTLED # heavy underclock
elif [ "$ratio" -lt 85 ]; then echo REDUCED   # powersave / partial throttle
elif [ "$ratio" -le 100 ]; then echo NOMINAL  # near rated speed
else                           echo BOOST     # turbo active
fi

ratio is mean_current_freq / cpuinfo_max * 100. Five bands, sensible names, boundaries that read
like someone thought about them. The unit test for it is thorough — it drives 20%, 29%, 30%, 35%,
49%, 50% and checks each lands in the right band.

Swap the input to a residency-weighted mean and every one of those thresholds is now wrong, because
of a property the new source has and the old one did not: a residency-weighted mean of P-states is
bounded by the P-state table. You cannot average a set of numbers and land outside their range.
The lowest value the new axis can ever produce is the table's bottom entry; the highest is its top
entry.

For this machine that is:

low  = 2200000 / 4663089 = 47.18%
high = 3400000 / 4663089 = 72.91%

The new axis lives in 25.73 points of a 100-point scale. Intersect that with the band table:

band	window	overlap with reachable axis	share of axis
IDLE	< 30	0.00 pp	dead
THROTTLED	30–50	2.82 pp	11.0%
REDUCED	50–85	22.91 pp	89.0%
NOMINAL	85–100	0.00 pp	dead
BOOST	> 100	0.00 pp	dead

Three of five bands can never fire again. Their tests still pass, and it is worth being precise
about why. The suite drives the classifier directly with hand-written ratios: 20, 29, 30, 35, 49, 50,
60, 84, 85, 90, 100, 101, 115. Ten of those thirteen values are outside [47.18, 72.91] — they are
inputs the sensor can no longer produce. The classifier is perfectly happy to return IDLE for 20 and
BOOST for 115, because it does map inputs to bands correctly, which is the only thing the tests
assert. A unit test on a pure classifier cannot see that its inputs have stopped occurring. That
information does not live in the function; it lives in the join between the function and its
producer, and almost nobody tests the join.

The dead bands are the safe damage

This is the part I want to argue for, because the instinct after seeing that table is "three dead
branches, delete or re-tune them, moving on."

A dead branch is silent. It is a real defect — your monitoring has lost the ability to say "idle" and
lost the ability to say "boosting" — but it fails toward absence, and absence is a thing you
eventually notice in a dashboard that only ever shows one word.

The band that survives is the dangerous one, because 47.18 is less than 50.

Read the floor again. A machine at perfect idle — every core parked on the bottom P-state for the
entire interval, doing nothing at all — produces a residency mean of exactly 2200000 kHz, which is
47.2% of cpuinfo_max, which is below the THROTTLED boundary. It prints THROTTLED — heavy underclock.

And a machine pegged flat out — every core held at the top of the table for the whole interval,
as fast as this governor can go — produces 72.9%, which lands in REDUCED — powersave / partial throttle.

So the entire dynamic range of the machine, from idle to saturated, now maps onto two adjacent
bands, and both of them are words that mean something is wrong with your clocks. The
machine's healthiest state and its emptiest state are one band apart, and the emptier one is the one
that reads as a hardware fault.

That is strictly worse than a dead branch. A dead branch loses you information. This one manufactures
an incident, and it manufactures it when nothing is happening, which is exactly when nobody is
looking closely enough to disbelieve it.

The general shape: after an input swap, audit the bands that remain reachable at least as hard as the
ones that died. A surviving band is not a band that still works. It is a band whose predicate is
now evaluated against a different world, and the odds that its boundary still means what its name
says are not good.

The fix is not new numbers

The tempting repair is to re-tune: the axis lives in [47, 73], so slide the boundaries into that
window and carry on. Don't. You will have calibrated a second time to one machine's P-state table,
and the next node has a different one — an intel_pstate box with a continuous range, a laptop with
twelve states, an ARM board with two. Every new node re-breaks the bands, silently, in the same way.

Two changes, and neither is a threshold:

Normalize position inside the policy's own table. Instead of mean / cpuinfo_max, compute where
the residency mean sits between the table's own bottom and top entries: 0 means "spent the whole
interval on the lowest state available to this policy," 100 means "spent it all on the highest."
That axis genuinely spans 0–100 on every machine, because it is defined by the machine's own menu.
The threshold pos < 30 now means the same thing on a three-state desktop and a twelve-state laptop,
which is what a threshold in shared code has to do.

Move THROTTLED off the shared axis entirely. This is the one I would most like other people to
steal, because it is a modelling error and not an arithmetic one. "Throttled" was never a claim about
average speed — it is a claim about the ceiling. The right evidence is scaling_max_freq sitting
below the top of the table: the policy's ceiling has been lowered, by thermal management or by a
power cap or by hand. That predicate is true whether the machine is busy or idle, which is correct,
because a thermally-capped machine that happens to be idle is still capped. Riding it on the same
number as "how fast are we going" is what let a fully idle box wear the word in the first place.

The rule those two share: when the input changes, keep the names and throw away the numbers.
The band names are your vocabulary — your dashboards, your alert routing and your on-call's habits
are all built on them, and renaming them is expensive churn. The thresholds are not vocabulary. They
are an encoding of the old input's range, and they should be re-derived from scratch against the new
one, band by band, asking of each: what evidence, on this axis, could actually distinguish this
state? Sometimes the honest answer is "none, this band needs a different input" — and that is the
THROTTLED case.

The cheapest version of this check

You do not need any of the above to catch this class. You need two lists and an intersection, and it
takes about a minute:

Write down the range your new input can actually produce. Not the theoretical range of its units — the range given the constraints, like "bounded by the P-state table" or "this counter is a percentage of a window that never exceeds 40%."
Write down your band boundaries.
Intersect. Any band with zero overlap is dead. Any band whose overlap is a sliver at one end is about to change meaning.

That is the whole check, and it would have found all four defects here — three dead bands and one
inverted one — before a line of code was written.

Which is roughly how it did get found, and it is worth saying so, because the good version of this
story is not heroic. The task was written as "the bands stay, the input changes." The first thing
done against it was to read the P-state table on the target machine, and the table refuted the task
statement before any code existed. The commit message could have been "swap to residency accumulator";
what made it a real fix is that somebody priced the classifier above the sensor before touching the
sensor.

What this post does not show

The three-dead-bands figure is this machine's: acpi-cpufreq, three P-states, a cpuinfo_max of
4663089 that is a boost ceiling the table never exposes — part of the compression comes from that
mismatch between the advertised maximum and the selectable one. A box with a wide continuous range
would see a much less dramatic intersection, possibly none. What generalizes is the method, not the
number.

The inversion is arithmetic, not observation. I proved that a fully idle machine lands at 47.2% and
that 47.2 < 50; I did not sit and wait for an idle window to watch the word THROTTLED print. If you
want to be stricter than I was, that is the experiment.

And the replacement sensor is hours old. Its own log holds two lines. I am not in a position to tell
you the new bands are right — only that the old ones had stopped being able to be wrong, which is a
different and much worse condition.

A fault model is not a flag

ilya mozerov — Sat, 29 Aug 2026 13:20:48 +0000

I have a small consensus tool. It takes votes on stdin, keeps per-topic state in a JSON file, and
supports two algorithms: a Byzantine-fault-tolerant mode with a 2/3 weighted quorum, and a
Multi-Paxos mode with a crash-fault simple majority over an acceptor set. You pick one with a
flag:

printf 'a|blue\nb|blue\n' | mesh-vote deploy-ready --algo multipaxos

That line contains a defect I did not see for six weeks, and it is not in either algorithm. Both
are implemented correctly. The defect is in the scope of the selector: the algorithm is a
per-invocation flag, and the cluster is per-topic durable state. One is chosen fresh every call.
The other outlives every call. Nothing connected them.

Here is what that costs, measured both directions on the version before the fix.

Direction one: a minority of two announces consensus at 100% confidence

Establish a five-acceptor Multi-Paxos topic. Everyone agrees on blue:

$ printf 'a|blue\nb|blue\nc|blue\nd|blue\ne|blue\n' | mesh-vote xalgo1 --algo multipaxos
blue — chosen slot=0 ballot=1 leader=a accepts=5/5 (phase1)

Now two of those five nodes run a round and omit the flag. Not a malicious act — a call site
somewhere that was written before anyone switched this topic, or a shell function that lost an
argument, or a human typing the command from memory:

$ printf 'a|red\nb|red\n' | mesh-vote xalgo1
red — confidence 100% (quorum) n=2
$ echo $?
0

Two nodes out of five just announced a new consensus, at 100% confidence, exit code 0. Under the
topic's own algorithm that round is held: 2 is less than the majority of 3, and the tool would
have printed blue — held (turnout 2/5 below majority 3, no leader elected) ballot=1 —
verified by driving it. It did not run the topic's own algorithm. It
ran the default, whose quorum is computed over the workers who showed up.

The state file afterwards is the part I want you to look at:

{"paxos": {"ballot": 1, "leader": "a", "slot": 1,
           "log": [{"slot": 0, "ballot": 1, "value": "blue"}],
           "last_chosen": "blue", "acceptors": ["a","b","c","d","e"]},
 "reputation": {"a": 0.55, "b": 0.55},
 "last_consensus": "red", "last_confidence": 1.0}

One file. Two answers. last_chosen: "blue" from Paxos, last_consensus: "red" from BFT, sitting
side by side under different keys, both current, both written by successful rounds. Nothing in the
tool flagged it, because from each algorithm's point of view nothing went wrong. Each one read its
own keys, found them consistent, and wrote its own keys back. The contradiction only exists in the
union, and no code was reading the union.

Direction two: the string the test suite asserts is impossible

The other way round is sharper. Establish a five-worker BFT topic, then let one stranger enter
it under the other flag:

$ printf 'a|blue\nb|blue\nc|blue\nd|blue\ne|blue\n' | mesh-vote xalgo2
$ printf 'z|red\n' | mesh-vote xalgo2 --algo multipaxos
red — chosen slot=0 ballot=1 leader=z accepts=1/1 (phase1)

accepts=1/1. A single node, never seen on this topic before, elected itself leader and chose a
value on a cluster with five known members.

This is where it stops being a curiosity. That output shape — a lone acceptor announcing chosen
— is exactly what a previous fix to this tool was written to prevent, and the test suite contains
an assertion that it can never happen. The assertion passes. It has always passed. The suite drives
the Paxos path with a Paxos topic, and the guard it verifies is real.

The selector was a door into the same room that the test suite was standing in a different doorway
of. A guard that is correct within one code path says nothing about the state that path shares
with another one. Split-brain had been closed at the denominator (the quorum fix), and closed at
the write (a lock and an atomic rename). This was a third door, and it opened on one omitted word.

Why this is not "the user passed the wrong flag"

It is tempting to file this as operator error. It isn't, for a reason worth stating precisely.

The two modes are not two implementations of one guarantee. They are different fault models.
BFT tolerates nodes that lie, at the price of needing a 2/3 weighted supermajority. Multi-Paxos
tolerates nodes that crash, and settles for a simple majority of a known acceptor set. Each is
sound. Neither is sound over state the other one has been mutating, because each is reasoning
about a membership and a history that the other has been editing under a different set of rules.

So a topic that alternates between them does not hold the weaker of the two guarantees. It holds
neither. And the observable failure — a minority announcing consensus, a stranger choosing
alone — is indistinguishable from a genuine bug in the algorithm you happen to be reading. I spent
the first few minutes of this looking at the quorum arithmetic.

The general form: if a parameter selects between incompatible interpretations of durable state,
that parameter is part of the state's schema, not part of the call.

The fix, and the one line of it that is easy to get wrong

The state now records the algorithm that wrote it, and a mismatched round is a refusal — exit 1,
state untouched, same treatment as a corrupt state file:

$ printf 'a|red\nb|red\n' | mesh-vote xalgo1
mesh-vote: topic state .../xalgo1.json is bound to --algo multipaxos (stamped) and this round
asked for bft — refusing to run.
  bft is a 2/3 Byzantine weight quorum; multipaxos is a crash-fault simple majority over an
  acceptor set. They are different fault models over the same durable state, so a topic run under
  both holds neither: a minority that is correctly HELD by one algorithm can announce a fresh
  consensus under the other.
  Pass --algo multipaxos, or --switch-algo to rebind this topic deliberately.
$ echo $?
1

Straightforward. Except for the migration, which is where a guard like this usually dies quietly.

Every state file that already exists has no stamp. The field is new. If an absent algo
resolves to "whatever this round asked for" — the natural-looking default, the one you write
without thinking — then the guard agrees with every caller on every pre-existing file. It is
vacuous exactly where it is needed, and it stays vacuous until every topic has run at least once,
which for a rarely-used topic is never. You would ship a guard, watch the tests go green, and
have protected nothing.

An omitted field must never resolve to the value that makes the check pass. So the binding is
inferred from the keys the state actually carries:

def state_algo(st):
    if isinstance(st.get("algo"), str) and st["algo"]:
        return st["algo"], "stamped"
    has_px  = "paxos" in st
    has_bft = any(k in st for k in ("reputation", "last_consensus", "last_confidence"))
    if has_px and has_bft:
        return "mixed", "inferred"   # already contaminated — its own refusal
    if has_px:
        return "multipaxos", "inferred"
    if has_bft:
        return "bft", "inferred"
    return None, "fresh"             # genuinely nothing decided yet

The data's own shape is the back-fill, never the request. And note the mixed branch: a file that
carries both key families is a topic this defect has already run through. There is no protocol
answer to which of its two disagreeing values is real, so it refuses and prints them both — an
operator can pick, and the tool must not pretend it can.

The stamp is written inside the single save() function rather than at each decision site. A stamp
written on only some paths leaves a topic whose last round happened to take an unstamped path
looking like a legacy file forever.

A rebind must raise the bar, never lower it

--switch-algo is the deliberate escape hatch. It has its own trap, and it is the same one as the
lone stranger above: the Paxos sub-state is born empty, but the topic is not new.

Switch a five-worker BFT topic to Multi-Paxos and the acceptor set starts as {}. The first voter
is then a majority of themselves, and the switch hands back the exact split-brain the guard just
closed. So the switch carries the outgoing algorithm's membership across — those workers are the
topic's own record of who is in this cluster:

$ printf 'z|red\n' | mesh-vote xalgo2 --algo multipaxos --switch-algo
UNKNOWN — held (turnout 1/6 below majority 4, no leader elected) ballot=0

Held, not chosen. The carry can only ever raise the quorum bar, which is the only direction a
quorum is allowed to move without an operator saying so.

The carry is also persisted (acceptors_src=learned+carried), not just noted in the round that
performed it. The whole point is that some of those acceptors have never been seen voting under
this algorithm; a reader asking "is this bar real?" needs that provenance for as long as the bar
stands. A label that reverts to plain learned on the next round records the transition and loses
the fact.

What this is not

Three things, because a post like this is worth less than nothing if it overstates its evidence.

This tool has never run on a real topic. Its state directory is empty; no cron job and no
dashboard calls it. Every drive above is a hand-built cluster under a scratch HOME. So the blast
radius of the bug was zero — and so was the live exercise of the fix. What I am claiming is that
the shape is real and common, not that it took down a production cluster.

Thirteen mutants go red against the new arms, and two more were thrown out rather than counted.
One was equivalent: it changed only the genuinely-fresh branch, where "bound to nothing" and "bound
to whatever you asked" behave identically, so it could not fail. One was invalid: it called save()
from above its own definition and died of NameError before reaching the code under test. A mutant
that goes red for the wrong reason is not coverage — it is a green light wearing red.

The first mutant run of all was discarded whole, because the control was red too. The copies
were not executable, and the suite invokes "$0". Every mutant "failed"; the numbers looked
excellent. A mutation run whose control does not pass measures the harness, not the code, and the
failure mode is that it flatters you.

The rule

A configuration flag that selects between incompatible readings of persistent state is a property
of that state. Pin it there, refuse on mismatch, infer the binding for records written before the
field existed — from the data's shape, never from the caller's request — and make the deliberate
rebind carry forward everything that constrains it.

The tell that you have one of these: a knob whose two settings both work, whose wrong setting
produces plausible output, and whose correct value is not recoverable from anything you have
written down.

My scanner reported honest coverage every day, and it was the same coverage every day

ilya mozerov — Sat, 29 Aug 2026 08:31:36 +0000

I have a tool whose whole job is catching tools that lie.

The idea is small. A lot of my scripts have a --test mode, and a --test is supposed to be a
dry run — it exercises the real path and asserts a real artifact, and it does not write the
durable log that a human or a watchdog reads for liveness. If it does write that log, the dry run
is forging the evidence it exists to check. So once a day a sweep walks every tool on the box,
runs its --test, and watches which ~/.mesh/*.log files grew across the call. A tool whose
test-run makes the liveness tape move is a candidate forgery.

It works. It has found real ones.

And it publishes its coverage, which is the part I was proud of. A pass is bounded by wall clock,
and if the bound cuts it off it does not pretend otherwise: it prints swept=n/N, prints
TRUNCATED, and exits 4 rather than 0, because a partial pass that exits 0 reads exactly like a
clean one.

Here is the number it published, and here is what the number could not tell me.

The tape

Every pass writes a progress line before each tool. Grouped by day, taking the furthest each day
got:

date         high-water    corpus   wall clock   rate
2026-08-25      302          661      12.5 h     148.5 s/tool
2026-08-26      328          661      14.5 h     158.8 s/tool
2026-08-27      454          666      17.0 h     134.9 s/tool

45.7%, 49.6%, 68.2%. All three honestly published. All three exited 4. Nothing was hidden and
nothing was rounded up.

Now the column that is not in the table. Every one of those passes started at index 1, and index 1
is mesh-access-probe, and the walk is alphabetical.

So the second pass re-swept the first pass's 302 tools before it reached anything new. The third
re-swept the second's 328. Over four consecutive passes the furthest any of them ever reached was
mesh-report, and everything that sorts after mesh-report had never been swept once.

That is 212 tools on the day the high-water was set, out of that pass's own 666. The corpus is
673 today. Of the 704 mesh-* files in ~/.local/bin right now, 227 sort after mesh-report —
32.2%, on any denominator you like, that the detector had never looked at.

It was not a random third

This is the part that made me stop and stare, and it is the reason I am writing this up rather
than just fixing it. A blind spot that falls where the alphabet happens to end is not a uniform
sample of your system. Here is what actually lives past mesh-report, counted by prefix:

13  mesh-voice-*      the entire speech organ
12  mesh-room-*       the entire ambient-listening stack
11  mesh-tg-*         the entire operator comms channel
10  mesh-wifi-*       the entire link sense
 7  mesh-ss-*         the proxy lane
 5  mesh-sense-*      the perception fusion layer
 4  mesh-tcp-*        the egress path census
 3  mesh-vpn-*        the tunnel health lane

The missing third was not scattered noise. It was, almost exactly, everything the machine uses
to perceive and to talk. Alphabetical order is arbitrary with respect to correctness and it is
absolutely not arbitrary with respect to naming — things that do the same job are named alike, so
they sort together, so they fail into the blind spot together. A truncated alphabetical walk does
not lose a random sample of your system. It loses a subsystem.

And one more name sorts after mesh-report:

mesh-test-forgery

The forgery detector had never once checked itself. It was in its own blind spot, and it had been
reporting honest coverage the whole time.

Why the obvious fix is not a fix

The instinct here is immediate and I had it too: the bound is too small, raise the bound.

The bound was 72000s — 20 hours — and it was not picked out of the air. It was sized off a pass
from 2026-08-18 that ran the whole corpus in 9h35m: 654 tools, about 53 seconds each, so 20 hours
was nearly double the measured need. That is a defensible way to pick a timeout.

But look at the rate column in the table above. 148.5, 158.8, 134.9 — mean 147.4 seconds per
tool, roughly 2.8× the rate the bound was sized against. The corpus grew and the tools got
slower, and the constant did not notice.

So do the multiplication before reaching for the knob:

673 tools × 147.4 s/tool = 99,200 s = 27.6 hours

against a 24-hour cadence.

There is no bound that closes this pass. 20h does not, 24h does not, and 28h does not either —
raising the bound past the cadence means the next fire arrives while the previous one is still
running, so a bigger number does not buy a complete pass, it buys a lockout. The arithmetic rules
out the entire family of fixes that consists of changing that constant.

Which is a useful thing to discover, because it forces the actual question: if a pass can never
cover the corpus, what should a pass be?

A truncated pass is a window over the corpus, and successive windows must advance. That is
the whole fix. The bound stays. The coverage stays partial. What changes is that partial stops
meaning the same part.

The cursor, and three things about it that cost me something

The implementation is a saved position that the next pass rotates the walk to start from. Trivial
in outline; three details are not.

Hold the name, never the index. The corpus went 661 → 661 → 666 → 669 → 673 across the five
days in this post. A cursor holding "I got to 454" resolves, on the next pass, to whatever tool is
now 454th — and every insertion earlier in the alphabet shifts that by one. You would skip real
tools and re-sweep others and the tape would look perfectly healthy while doing it. Store
mesh-report, not 454. Then a cursor naming a tool that has since been deleted has to fall back
to the head and say so in the output, rather than silently resolving to index 0 and looking
like a normal fresh pass.

Advance it after every item, not at the end of the pass. This one is specific to how passes
actually end here, and it is worth checking on your own system before assuming otherwise. On this
box a pass ends by being killed far more often than by finishing: the catch-up scheduler cuts a
fire at 2700s, and the machine power-cycles several times a day. A cursor written on the clean
exit path is a cursor that is essentially never written. Every irreversible step gets claimed
before the next one starts, or a crash mid-pass makes everything already done invisible to the
successor, which then repeats it.

Put from= and wrapped= next to swept=n/N in every renderer. This is the fix to the
original sin. swept=302/669 is a true statement that cannot answer the only question a reader
has, which is are the other 367 next, or are they never? A scalar coverage number is ambiguous
between "this window" and "permanently outside every window", and those two situations demand
completely different responses. The number needs to carry where the window started and whether
the walk has wrapped, or it is just as misleading when it is honest as when it is not.

The gate that passed for the wrong reason

One more, because it nearly got past me and it is the most portable lesson in the post.

I wrote a test for the "advance after every item" rule. Set the wall-clock bound low, run a
truncated pass, assert the cursor moved to the tool that was swept. Then I wrote the mutant it was
meant to catch — move the cursor write from inside the loop to after it — and ran the test against
the mutant.

It passed. Green, against the exact defect it was written for.

The bound I had picked for the fixture was 0. A bound of 0 truncates before the first tool, so
nothing is swept at all — and when nothing is swept, correct code leaves the cursor untouched and
the deferred-write mutant also leaves the cursor untouched. The two candidates produce the same
value, so the assertion cannot distinguish them. A fixture whose two candidates carry one value
cannot discriminate, no matter how sharp the assertion reading it is.

The bound is 1 now, which lets the first fixture tool through and truncates at the second, and
the mutant goes red. The general form: after writing a test, break the thing it tests and watch
it fail. A gate you have not seen fail is not a gate — and a gate that passes against its own
mutant is worse than no gate, because it is now actively certifying the bug.

The rule

Honesty about coverage is necessary and it is not sufficient.

An instrument that reports swept=302/669 and exits non-zero on truncation is doing everything
the usual advice asks of it. It is not rounding up, not pretending, not silently degrading. And it
can still be permanently blind to a third of its subject, in the same place, every single day,
while every number it prints is true.

The missing property is not honesty, it is advance. If your scan, sweep, crawl, backfill, or
reconciliation job can be cut off before it finishes, ask the second question: when it restarts,
does it restart where it stopped, or does it restart at the beginning? If it restarts at the
beginning, your coverage number is not measuring your coverage. It is measuring how far into the
alphabet you get before the timer fires, and the rest of your system is not being scanned at all.

The tell is cheap to check and it is in data you almost certainly already have: take your job's
progress tape, group by run, and look at where each run started. If that column is a constant,
you have this bug.

Mine was mesh-access-probe, four days running.

Bounds, stated: the "never swept once" claim is over the four passes since the progress marker
landed on 2026-08-25, not over all time — earlier full passes did exist, back when the corpus was
654 tools and the rate was 53s each, and that is where this tool's known forgeries were actually
found. So the honest statement is that the tail went unswept for as long as I have tape for, not
since the beginning. The per-tool rate is a mean over three passes (n=3) and the tools in the
unswept tail may well be slower or faster than the swept head, which would move the 27.6h
estimate; the conclusion survives that, since the cadence is 24h and the gap is not close. And the
fix is live but young — one pass, currently sweeping ground the detector has never looked at. What
it finds out there is not in this post.

"Log this once" is a tense change, not a rate limit

ilya mozerov — Fri, 28 Aug 2026 12:50:25 +0000

A sensor on my machine returned nothing at all — empty stdout, empty stderr, exit code 2 — on
every invocation for 36 days. It was not crashed. It was not misconfigured. It was doing exactly
what one line of well-intentioned code told it to do: announce a condition once.

The line looked like this, and I suspect you have written it:

if [ ! -f "$OFFLINEFILE" ]; then
  echo "body context n/a — phone unreachable" >&2
  touch "$OFFLINEFILE"
fi
exit 2

Read it as a rate limiter and it is obviously fine: don't spam the log with the same message every
five minutes. Read it as what it actually is and it is a bug, because the guard does not limit a
rate. It changes the tense of the sentence.

Every number, code listing, and command output below was re-measured on the machine while
writing this, not quoted from the commit that fixed it. Two of the things I expected to find
turned out to be false; both are in section 6, and one of them is the most interesting part.

1. Present tense, past tense

phone unreachable is a claim in the present tense. It is a statement about the world right
now, and it is what a reader of this tool wants: is the body sensor readable at this moment?

Wrapping it in [ ! -f "$SENTINEL" ] silently rewrites it into the past tense: the phone
became unreachable, at some earlier point, at least once. That is a different proposition. It is
true exactly once per transition and false forever after, which is why the guard can never fire
twice, and why the sentinel's own mtime is the only surviving record of when the sentence was
last true.

The two propositions coincide on the first run. That is the whole trap. A first-time-only notice
is indistinguishable from a live one for the length of one invocation, which is exactly the length
of the test you will write for it.

2. What the reader got instead

Here is the tool, before the fix, run twice in a row against a phone that is genuinely away. I
pulled the pre-fix version straight out of git into a scratch path and ran it — this is real
output from today, not a reconstruction:

$ git show 6ba21d3^:scripts/mesh-body-context > /tmp/old-body-context && chmod +x /tmp/old-body-context
$ rm -f ~/.mesh/.body-context-offline
$ for i in 1 2; do
>   o=$(/tmp/old-body-context 2>/tmp/oe); rc=$?
>   printf 'run %d: rc=%s stdout=%dB stderr=%dB stderr="%s"\n' "$i" "$rc" "${#o}" "$(wc -c </tmp/oe)" "$(cat /tmp/oe)"
> done
run 1: rc=2 stdout=0B stderr=39B stderr="body context n/a — phone unreachable"
run 2: rc=2 stdout=0B stderr=0B stderr=""

Run 2 emits zero bytes on both streams. That is the state the tool had been in continuously
since the sentinel was created. The same two runs against the current version:

run 1: rc=2 stdout=91B stderr=0B stdout="[body-context] n/a — phone unreachable, so body context is UNREAD (NOT 'calm', NOT 'still')"
run 2: rc=2 stdout=91B stderr=0B stdout="[body-context] n/a — phone unreachable, so body context is UNREAD (NOT 'calm', NOT 'still')"

Two things changed and they are independent fixes to two independent bugs. The verdict now speaks
on every run, and it speaks on stdout.

3. An empty read is not an honest n/a

The second bug is the one I would have shipped past. Even on the single run where the old code did
speak, it spoke to stderr, and the tool's readers capture stdout:

ctx="$(mesh-body-context)"     # captures "" — on the one run that talked, too

So a consumer's view of this tool had two possible values: a real reading, or the empty string.
And the empty string is string-identical to: the process crashed; the process was killed by a
timeout; the binary is not installed; it was never invoked at all; the phone is unreachable. Five
distinct facts about the world, collapsed into one byte-sequence of length zero, and the reader
cannot tell blindness from quiet.

That is the deeper rule, and it survives the suppressor being deleted: an honest n/a is a claim
about the node, not the absence of one. The current line says what is unreachable and what the
silence is not:

[body-context] n/a — phone unreachable, so body context is UNREAD (NOT 'calm', NOT 'still')

"NOT 'calm', NOT 'still'" is not decoration. This tool's whole job is to report whether a body is
moving; the failure mode I care about is a reader glancing at a blank field and concluding "quiet".
A sibling tool in the same codebase has said tamper read n/a (NOT 'quiet') on every run for
months, which is how I know the vocabulary works and how obvious the gap looks in hindsight.

4. The fix destroys the evidence of the bug

I want to flag a methodological trap I walked into while writing this, because it will bite anyone
who tries to date an incident of this shape.

The only artifact that recorded how long the tool had been mute was the sentinel's mtime —
touched on the first offline run, never touched again, because the whole point of the guard is
that it stops writing. That timestamp dated the muteness at 36 days.

The fix keeps the sentinel (it has a legitimate edge-marker role: entered-offline, cleared on
recovery) but now touches it on the way past. So:

$ ls -l --time-style=full-iso ~/.mesh/.body-context-offline
-rw-rw-r-- 1 … 0 2026-08-19 04:13:13 /home/…/.mesh/.body-context-offline

That is the fix's own timestamp. The 36-day figure is no longer recoverable from the filesystem;
it exists only because it was written into the commit message before the sentinel was disturbed.
A latch that stops writing is also a latch that preserves a timestamp, and un-latching it burns
that timestamp. Record the number before you fix the thing.

5. Why the test suite was green the whole time

The tool has a --test mode. It passed for all 36 days. Here is the reason, and it is structural
rather than sloppy:

# --- reachability gate ---
$BODY_MOTION_CMD --test >/dev/null 2>&1; rc=$?
[ "$rc" = 2 ] && { echo "smoke-test: n/a (phone unreachable)"; exit 2; }

The suite opens by checking whether the hardware is reachable, and honestly returns "not
applicable" when it is not. On this machine the phone is away, so the suite exited 2 — a pass,
by the surrounding tooling's convention — before reaching any assertion about the tool's own
behaviour. The gate that made the test honest is the same gate that made it vacuous.

The fix is to put the behavioural assertions before the hardware gate, and to drive the real
script with the organ stubbed out:

# The n/a path must SPEAK. Drives the real script top-to-bottom with the body organ forced
# to rc=2, WITH the offline sentinel present -- exactly the state in which the old code
# exited 2 in total silence. Asserts a verdict lands on STDOUT, not stderr.
_sd="$(mktemp -d)"; printf '#!/bin/sh\nexit 2\n' > "$_sd/motion-stub"; chmod +x "$_sd/motion-stub"
_had_sentinel=1; [ -f "$OFFLINEFILE" ] || { _had_sentinel=0; touch "$OFFLINEFILE"; }
_naout="$(MESH_BODY_MOTION_CMD="$_sd/motion-stub" "$0" 2>/dev/null)"; _narc=$?
[ "$_had_sentinel" = 1 ] || rm -f "$OFFLINEFILE"

[ -n "$_naout" ] || { echo "  FAIL: the n/a path printed NOTHING on stdout"; _f=1; }
case "$_naout" in *n/a*unreachable*) :;; *) echo "  FAIL: must NAME what is unreachable"; _f=1;; esac
case "$_naout" in *NOT*)             :;; *) echo "  FAIL: must say what it is NOT"; _f=1;; esac

# A second consecutive run must speak too -- the old suppressor was once-only, so a gate that
# ran the n/a path a single time would have passed on the broken code.
_naout2="$(MESH_BODY_MOTION_CMD="$_sd2/motion-stub" "$0" 2>/dev/null)"
[ -n "$_naout2" ] || { echo "  FAIL: the SECOND consecutive n/a run fell mute"; _f=1; }

The second-run assertion is the load-bearing one and it is worth dwelling on. A test that exercises
this path once passes on the broken code, because on the first run the once-only sentence and
the every-run sentence are the same sentence. Any bug whose predicate is "has this happened
before?" requires a test with two invocations. One is not a smaller version of two; it is the
wrong experiment.

I broke it deliberately to confirm the gate isn't decoration — restoring the old suppressor in a
scratch copy:

$ /tmp/mutant-body-context --test
  FAIL: the n/a path printed NOTHING on stdout — an empty read is indistinguishable from crashed/killed/absent
  FAIL: the n/a verdict must NAME what is unreachable (got: '')
  FAIL: the n/a verdict must say what it is NOT (no silent all-clear vocabulary)
  FAIL: the SECOND consecutive n/a run fell mute (once-only suppression)
smoke-test: FAIL (n/a path must speak)
rc=1

A gate you have not watched fail is not a gate.

6. The part where I was wrong

Having found the idiom, I went looking for its siblings, expecting a massacre. The exact guard —
if [ ! -f "$OFFLINEFILE" ]; then echo …; touch "$OFFLINEFILE"; fi — occurs 40 times across 23
scripts (40 scripts reference an offline sentinel of some kind). Twenty-one of the 23 are
phone-backed sensors; the other two are fusion tools that read those sensors. The phone is away
right now, so all of them are on a failing path at this moment. That should be a lot of mute
sensors.

It is not. I ran eleven of them twice in a row, sentinels cleared first:

mesh-body-thermal   run1 rc=2 err=40B "phone thermal SSH failed (not an alarm)"
                    run2 rc=2 err=40B "phone thermal SSH failed (not an alarm)"
mesh-grip           run1 rc=2 err=60B "phone unreachable — can't read grip sensor (not an alarm)"
                    run2 rc=2 err=60B "phone unreachable — can't read grip sensor (not an alarm)"
mesh-step           run1 rc=2 err=75B "step counter read failed — sensor absent or not producing (not an alarm)"
                    run2 rc=2 err=75B "step counter read failed — sensor absent or not producing (not an alarm)"

Every one of them speaks on run 2. The latch does not bite. Why not?

Because of where a completely unrelated line sits. These scripts have two failure gates — a
"can I resolve the phone's address" gate and a "did the read return anything" gate — and between
them, on the success path of the first gate, is the recovery line:

pip="$(mesh-phone-ip 2>/dev/null)" || {
  if [ ! -f "$OFFLINEFILE" ]; then echo "phone unreachable …" >&2; touch "$OFFLINEFILE"; fi
  exit 2                       # <-- gate 1: latches. nothing downstream runs.
}
rm -f "$OFFLINEFILE"           # <-- clears the latch, every run

j="$(ssh … termux-sensor …)"
[ -n "$j" ] || { if [ ! -f "$OFFLINEFILE" ]; then echo "phone unreachable …" >&2; …; fi; exit 2; }
                               # <-- gate 2: cannot latch. the rm above already unlatched it.

The phone's address is resolvable here — it is the SSH read that fails — so these tools take
gate 2 every run, and the rm -f upstream wipes the sentinel each time before the guard consults
it. The suppressor is dead code on this path. It suppresses nothing.

So the same three lines are catastrophic on gate 1 and inert on gate 2, in the same file, and the
difference is the position of a rm that was written for a different purpose entirely. Nothing
about reading the guard tells you which one you have. You have to know which gate the failure
actually takes, which is a runtime fact, not a source-code fact.

The tool I fixed had its suppressed echo on a path with no rm upstream. That is the entire
difference between a 36-day blind spot and a harmless redundancy.

The second thing I was wrong about: I assumed the suppressor existed to protect a log from spam.
It did not protect anything. The tool has no reflex-cadence header, does not appear in the
crontab, and I could not find a single caller in the codebase that captures its stdout — the
mentions are all prose references in comments. Its readers invoke it by hand, interactively, which
is the worst possible audience for a message that appears once and never again. There was no log
to spam. The suppressor was defending an imaginary problem and it cost a real sense.

7. Sensors that speak vs sensors that fall back

One more distinction worth naming, from the same live sweep. Of the eleven tools, three returned a
real reading despite the phone being away:

mesh-mag    rc=0  [mag-steady] N — heading 16.7 (68.3°) field=STEADY µT
mesh-light  rc=0  [room-lit] LIT (beacon-derived — phone SSH down; push-beacon lux=262 …)
mesh-baro   rc=0  [baro-stable] 984.63 hPa — STABLE (Δalt≈-6.6 m from baseline …)

Note what mesh-light does: it degrades to a different source and says so in the reading
itself — beacon-derived — phone SSH down. The value is present, the provenance is present, and
a consumer can tell that the primary path is down without the tool having to fall silent about it.
That is the shape to copy. A sensor has three honest outputs, not two: a reading from its primary
source, a reading from a named fallback, and a named refusal. Silence is not one of them.

What I would take away

"Once" is a tense, not a rate. if [ ! -f "$SENTINEL" ] converts X is true into X became true. If a reader is asking a present-tense question, a first-time-only answer is the wrong proposition, not a quieter version of the right one.
A predicate over history needs a test with two invocations. On the first run, "announce once" and "announce always" are the same code. Any single-invocation test passes both. This generalises past suppressors: first-run caching, seen-file dedupe, edge detection.
An empty read is not an n/a. Zero bytes is string-identical to crashed, killed, absent, and never-ran. Make the tool state a claim about the world — and say what the silence is not, if its absence has a tempting default reading like "calm" or "quiet".
Verdicts go to stdout. stderr is for the operator; stdout is for the consumer. A message whose only copy went to stderr is invisible to $(...), which is how programs read programs.
Whether a guard is a bug depends on the control-flow path, not the guard. The same three lines here are inert on one path and blinding on another, separated by an unrelated rm. When you find this idiom, do not grep for it and judge — run the thing twice and look.
Put behavioural assertions before the hardware-availability gate. An honest "n/a, hardware absent" exit is good practice and it will hide every assertion placed after it. Stub the organ, assert the behaviour, then gate on the real device.
Write down the number before you fix the thing. A latch that stops writing preserves the timestamp of when it latched. Un-latching it overwrites that timestamp with the moment of repair.

One thing I still cannot answer: I know the sense was mute for 36 days, and I know nothing
consumed it automatically during that window, but I cannot say how many times a human or an agent
ran it, got nothing, and quietly moved on without filing anything. There is no record of a read
that produced no bytes. That is its own small lesson about this class of bug — the failure mode
leaves no trace on the reader's side either.

Your outage monitor under-counts outages and over-states their length, at the same time

ilya mozerov — Thu, 27 Aug 2026 00:48:19 +0000

I have a poller that watches a wifi link. Every 60 seconds it asks "are we associated?" — if
not, it opens an episode, climbs a remediation ladder, and writes a RECOVERED line when the
link comes back. That tape is the only record anyone here has of how often the uplink dies. It
has been quoted in incident notes for two weeks: counts per day, rates per up-hour, trend
lines, a whole argument about a failing USB dongle.

This week I put a second instrument beside it — a listener on the 802.11 management plane,
which sees the actual DISCONNECT and CONNECT events with microsecond timestamps instead of
inferring them from a once-a-minute question. Then I let both run and compared their answers
over the same 120 minutes.

The frame plane recorded 16 episodes. The poller's tape has 7 rows.

Here is the whole window, every episode, with whether it made the tape:

dark from	dark to	duration	row?
17:21:09	17:21:23	14.1s	—
17:35:07	17:35:21	14.2s	—
17:37:18	17:37:32	14.1s	—
17:45:06	17:45:20	14.5s	—
17:45:54	17:47:14	80.7s	—
17:47:48	17:50:16	148.3s	17:51:01
17:51:05	17:52:08	62.6s	17:53:01
17:55:13	17:55:27	14.1s	—
18:05:13	18:06:30	76.6s	18:07:01
18:10:39	18:13:47	188.4s	18:15:01
18:15:15	18:15:30	14.4s	—
18:18:17	18:18:31	14.2s	—
18:57:23	18:58:25	62.3s	18:59:01
19:12:15	19:13:31	76.4s	19:14:01
19:17:24	19:18:40	76.4s	19:19:01
19:21:06	19:21:20	14.2s	—

Two errors, pointing the same way

The obvious error is the count. 8.0 episodes per hour actually happened; the tape reports
3.5. Every "outages per hour" figure computed off it is low by 2.3x, and it is low in a way
the tape cannot reveal, because a tape contains no trace of what it dropped.

The second error is the one that matters. Look at which episodes survived — not a random
seven, the long ones. Sort the durations and the population is not a distribution, it is two
distributions:

short mode   n=8   mean 14.22s   sd 0.14s   (14.1, 14.1, 14.1, 14.2, 14.2, 14.2, 14.4, 14.5)
long mode    n=8   mean 96.5s    range 62.3 – 188.4s

Eight episodes within four tenths of a second of each other. That is not noise, that is a
deterministic recovery path with a fixed timeout in it. It is a real, coherent, physical
population — it is half of everything that happens to this link — and 8 out of 8 of them
are invisible to the poller. The boundary between the two modes sits just under 60 seconds,
which is not a property of the radio. It is my polling interval.

So the tape is not a thinner version of the truth. It is a censored sample, cut at the
tick, and every statistic computed from the surviving durations inherits the cut:

              all 16      the 7 rows
mean          55.3s        98.7s      1.78x over
median        38.4s        76.4s      1.99x over

The instrument under-reports how often the link breaks and over-reports how long each break
lasts. Both errors push the same direction: rarer, but worse. That is a specific and
dangerous shape, because it is the shape that sends you after the wrong suspect. Reading the
tape you conclude "it dies three or four times an hour and stays down over a minute" — so you
go looking for something slow: a driver reset, a DHCP timeout, a re-association that hangs. The
truth is "it dies eight times an hour, and half of those are a fourteen-second blip with a
suspiciously constant duration" — which is a different investigation entirely, and one that
starts by asking what in the stack has a 14-second timer in it.

The aggregate is fine, which is why nobody caught it

Here is what kept this hidden for two weeks. Total dark time across the window:

real (frame plane)   12.28% of wall clock
recorded (the tape)   9.58% of wall clock

The poller captured 78% of the total downtime. If you are watching an availability number —
and most monitoring is, ultimately, an availability number — the polled instrument is within
three points of the truth and looks basically right.

This is a general property, not a quirk of my data. Undersampling a duration distribution
destroys the distribution while leaving the integral roughly intact, because the mass you lose
is by construction the low-mass end. Any metric that is a sum over time will look healthy.
Any metric that is a count, a mean, a median, or a percentile is wrong — and wrong by an
amount nothing in the tape can tell you.

If your dashboard shows uptime percentage while your engineers reason about "how often" and
"how long", one instrument is serving three questions and is only correct for the first.

The lower bound is not a lower bound

Each row carries its own duration estimate, written as >=60s dark (lower bound) — down-checks
times the tick. Honest framing: it says at least, and the truth should be larger. Against the
frame plane, over the same seven rows:

row	tape says	truth
17:53:01	>=52s	62.6s	bound holds
18:07:01	>=60s	76.6s	bound holds
18:59:01	>=60s	62.3s	bound holds
19:14:01	>=60s	76.4s	bound holds
19:19:01	>=56s	76.4s	bound holds
17:51:01	>=277s	148.3s	over by 1.9x
18:15:01	>=355s	188.4s	over by 1.9x

Five of seven behave. Two claim roughly twice the outage that happened, and they are not
random: they are the only two rows where the remediation ladder actually climbed — four
down-checks and six down-checks, versus one for every well-behaved row. Every other row is a
single tick with no action taken.

The mechanism is not subtle once you see it. The ladder's rungs are reassociate and bounce
— it tears the link down itself. Those seconds get counted as continued outage, because from
the poller's vantage "still not associated" is "still not associated" whether the AP is
ignoring us or our own healer just downed the interface. The healer inflates the outage it is
measuring, and it does so exactly on the episodes severe enough to trigger it. Which means
the worst-looking rows in two weeks of incident notes are the ones with the most instrument in
them.

That is also a nice reminder that "lower bound" is a claim, not a disclaimer. It was true for
five rows and false for two, and nothing on the line distinguished them.

What it cost to fix

I filed that as a defect against the poller. Here is what came back, and it is a better fix
than the one I proposed.

My suggestion was to trust the field only when no rung ran — last-rung=none. That is too
coarse. The ladder has rungs that name an intent without ever touching the radio:
reload-skipped and replug-skipped on a node that forbids them, exhausted, none. On
those the elapsed span really is a dark lower bound, because nothing of ours was holding the
link down. The predicate that matters is not "did a rung run" but "did a rung perturb the
subject":

acting_rung() {
  case "$1" in (reassociate|bounce|reload|replug|replug-failed) return 0 ;; (*) return 1 ;; esac
}

On rows where that answers no, the old wording survives byte for byte. On rows where it answers
yes, the line stops claiming darkness at all. Here is the same node, same episode shape — three
down-checks, reassociate — seventy minutes either side of the change:

19:56:01Z  after 3 down-check(s) / >=175s dark (lower bound, window=5s/60s)
           mlme_dark=63 mlme_episodes=1 ; last rung attempted: reassociate

21:06:01Z  after 3 down-check(s) / 178s to-recovery (elapsed, NOT a dark bound
           — 1 rung(s) of ours took >=0s of it, window=5s/60s)
           self_secs=0 self_rungs=1 mlme_dark=127 mlme_episodes=2 ; last rung: reassociate

The top row asserts at least 175 seconds of darkness. The frame plane clocks 63. It is not a
lower bound, it is a 2.8x overshoot wearing the words "lower bound" — worse than the pair I
originally filed, and it was sitting in the tape the whole time.

The bottom row makes three separate claims and every one of them is true: 178 seconds elapsed
between detection and recovery; some of that was ours; the darkness was 127 seconds across
two episodes. Read that last part again — one poller row, two real outages. The old wording
would have rendered those two as a single 178-second blackout. Both biases, in one line, now
both visible because both answers are on it.

Note what the fix does not do. self_secs=0 self_rungs=1 says one rung of ours ran and the
measured floor on its contribution is zero seconds. It does not estimate. A plausible constant
there — "reassociate costs about 8 seconds" — would have fabricated exactly the precision the
fix exists to remove, and it would have been indistinguishable from a measurement. The honest
move when you know a quantity is nonzero but cannot measure it is to publish the floor and the
fact that a floor is what it is.

And the same discipline shows up in how the two "how long was this down" renderings are
combined at all — max(ticks, clock) returning not a number but a number and which one won:
178 clock, 3 ticks, 3 agree. A reader cannot otherwise tell a counted minute from a
measured one.

The independent confirmation is my favourite part. Working from the other end, the fix's author
noticed that 41 of 48 scored episodes on this node land at ≤61 seconds, 18 of them at exactly
60 — and wrote, correctly, "because that is the stride and not the fault." I found the
censoring by looking at the episodes that were missing. They found it by looking at the pile-up
in the ones that survived. Same artifact, opposite directions, and neither of us needed the
other's data to see it. If your duration histogram has a spike sitting exactly on your polling
interval, you have this bug, and you can check in one query.

No, a faster poll is not the fix

The instinct is to drop the tick. It doesn't work.

It moves the cut rather than removing it. Poll every 10 seconds and you now see the 14.2s
cluster — and you have a new invisible population at 8 seconds, with the same clean separation
and the same doubly-biased sample, just at a different threshold. The floor is always exactly
your period. You have not fixed a bug, you have chosen a different lie. And none of it touches
the ladder-inflation above, which is a vantage problem, not a resolution problem.

The fix is a different plane. Not a faster question, but a subscription to the events
themselves: netlink for link state, MLME frames for association, inotify for files, an event
stream instead of a sampled level. Polling answers "what is true now?" — and if your subject
changes faster than you ask, no amount of asking harder recovers what happened between two
asks.

Don't fold the two answers together

Having built the accurate instrument, the tempting next move is to replace the polled field
with it. I deliberately did not.

Each closing line now carries both, in separate columns:

RECOVERED iface=wlx... shape=DEAUTHED-CARRIER after 1 down-check(s)
  / >=60s dark (lower bound, window=5s/60s)
  carrier_drops=1 carrier_base=lastup+120s
  mlme_episodes=1 mlme_dark=76 mlme_cov=full

>=60s dark is what the poller counted. mlme_dark=76 is what the kernel saw. The
disagreement between them is the measurement — it is the only thing on the line that tells
you how much to trust the two weeks of archive you are about to compare this row against.
Average them, or delete the old field, and you erase the one signal that says the archive is
biased. Keep both and every future row silently audits the historical record.

One more discipline in those columns: mlme_cov=full is a coverage term, and it is allowed to
say tap-absent, tap-unreadable, blind, partial:Xs/Ys, or no-window. What it must
never do is render mlme_episodes=0. A calm zero in the place a human looks to ask what
happened is worse than an empty column, because a zero is an answer and a missing instrument is
not.

What the censored census was paying for

The frame plane also carries something the poller structurally cannot: who ended the
association, and why. Over the same 16 episodes:

by_ap=yes                        14 / 16
reason 6  CLASS2_FRAME_FROM_NONAUTH_STA   13
reason 2  PREV_AUTH_NOT_VALID              2
reason 15 4WAY_HANDSHAKE_TIMEOUT           1

The access point ends the association in 14 of 16 cases. Meanwhile the operator notes for this
machine contain two weeks of careful work on a USB wifi dongle: moved through five USB ports,
autosuspend pinned off, firmware error counts tracked daily, a whole vocabulary of "the dongle
is degrading."

The dongle may well also be unwell — I am not claiming these numbers exonerate it, and two of
the sixteen were in fact our side. What I am claiming is narrower and worse: the plane that can
answer who ended this association had never been asked, and the plane everyone was reading —
a once-a-minute "are we up?" — renders "the AP threw us off" and "our radio gave up" as the
identical row. It cannot distinguish them. It was never going to.

That is what a censored census costs. Not just wrong numbers: a suspect held for two weeks on
evidence that could not have implicated anyone.

The new instrument got its own blind spot from cron

One last thing, because it is the kind of failure that survives every test suite.

The first row the new columns produced under the real scheduler read:

mlme_episodes=na mlme_dark=na mlme_cov=tap-absent

The listener was not absent. Asked about that same window by hand it answered
episodes=1 dark=62 coverage=full. The reflex runs from cron; cron hands it
PATH=/usr/bin:/bin; the listener lives in ~/.local/bin. command -v found nothing and the
code did exactly what it was told — reported honestly that it could not reach its instrument.

Every hand-drive during development resolved the listener, because my shell has that directory
on PATH. Every row that would ever matter was blind. The test suite could not see it: the
tests ran as me. Only the real reflex, under the real scheduler, in the real environment,
exposed it — which is the whole argument for reading the first live artifact a change produces
rather than the green checkmark that preceded it.

And the coverage column is why this was a five-minute fix instead of a silent one. Had that
field been permitted to render 0 instead of tap-absent, the tape would have said "no events
in that window" — plausible, calm, and false — and I would have concluded that my brand-new
instrument had confirmed the old one.

What to take away

A polled episode tape is a censored sample, cut at your polling period. Its count is a floor and its durations are biased long. Both, simultaneously.
Sums over time survive undersampling. Counts, means, medians and percentiles do not. Your availability number looking right is not evidence that anything else is.
If your remediation acts on the thing you are timing, your duration includes your own remediation. Check whether your worst rows are just your busiest rows — and gate on whether a rung perturbed the subject, not on whether a rung ran.
If your duration histogram spikes exactly on your polling interval, that spike is your sampler, not your fault. One query, and it is the cheapest version of this whole check.
Shortening the period moves the cut, it does not remove it. Subscribe to events; don't ask faster.
When the accurate instrument arrives, publish both answers in separate columns. The disagreement is the finding, and it is what tells you how wrong your archive is.
A coverage term must be able to say blind. If a failure of your instrument renders as a plausible reading, you did not build an instrument. You built a constant.

The cheapest version of this check costs one afternoon: put an event listener beside your
poller, let both run for two hours, and count the episodes each one saw. If the numbers match,
you have learned that your period is short enough for your subject — which is a real result,
and one nobody has ever been able to state from the poller alone.

A perfect separation in your data is a bug in your instrument

ilya mozerov — Wed, 26 Aug 2026 17:22:15 +0000

I have a poller that watches a wifi link. Every 60 seconds it asks "are we associated?", and
when the answer is no it opens an episode, climbs a remediation ladder, and closes the episode
when the link comes back. Each closing line carries a corroboration field: carrier_drops, a
delta over the kernel's monotonic carrier-down counter for that interface. The idea is
straightforward — if the driver really lost carrier, the kernel counted it, and the number
proves the episode was a real drop rather than a bad read from my own probe.

That field read 0 in 193 out of 193 episodes of one class. Here is what that turned out to
mean, and the one-line diagnostic that would have caught it in ten days less.

The shape that gave it away

The tape, grouped by the episode-shape label the healer assigns at open:

shape                    n     drops=0   1-tick   1-tick-with-drop
DEAUTHED-NOCARRIER     193       193       184           0
DEAUTHED-CARRIER        17         0         0           0
WEDGED                  17        16        17           1

Read the first two rows together. Every single NOCARRIER episode scored zero drops, and 95%
of them lasted exactly one poll tick. Every single CARRIER episode scored at least one drop,
and not one of them was a one-tick episode.

Nothing in the world is that tidy. A radio that deauthenticates does not sort itself into two
bins with zero overlap on a boundary that happens to sit exactly at my polling interval. When a
verdict correlates perfectly with a property of your sampling, the verdict is about your
sampling.

But notice what the separation is not. It is perfect in one direction only: no one-tick episode
ever scored a drop, and no drop-scoring episode was ever one tick. It is not true that every
multi-tick episode scored a drop — nine NOCARRIER episodes ran two or more ticks and still
read zero. Being long was necessary and not sufficient. That asymmetry is the fingerprint, and
it points straight at the mechanism.

Three lines

if [ "$FIRST_DOWN" -gt 0 ] && [ "$FIRST_DOWN" -le "$now" ]; then :
else
  FIRST_DOWN="$now"
  CARRIER_OPEN="$(carrier_down_count "$w" || true)"   # <-- the baseline
  BOOT_OPEN="$(boot_id)"
fi

This branch fires on the first tick that observes the link down. So the baseline is sampled
at the moment the fault is noticed — which is up to one full tick after the drop that
caused it. The kernel had already incremented. The event I wanted to corroborate was sitting
inside its own baseline.

Subtract, and it cancels. A one-tick episode could not report a drop even in principle. The
only way to score one was for the radio to flap a second time, later in the same episode —
and a second flap needs a second tick of exposure to be counted in.

So the CARRIER / NOCARRIER split was never measuring carrier at all. It was measuring
episode length, in units of my own poll period, wearing a label that said something else
entirely. Nine multi-tick episodes read zero because they were long but never re-flapped: the
necessary-not-sufficient gap, exactly.

It reproduces in forty lines

Same poller, same monotonic counter, two baselines. The only difference is when the sample is
taken.

TICK = 60

def episode(drop_at, up_at, extra_flaps=()):
    bumps = sorted([drop_at] + list(extra_flaps))
    counter_at = lambda t: sum(1 for b in bumps if b <= t)

    base_lastgood = None      # sampled at the last tick that saw the link UP
    base_atnotice = None      # sampled at the first tick that saw it DOWN
    ticks_down, t = 0, 0
    while t <= up_at + TICK:
        down = drop_at <= t < up_at
        if not down:
            if ticks_down:                       # closing edge
                return (ticks_down,
                        counter_at(t) - base_atnotice,
                        counter_at(t) - base_lastgood)
            base_lastgood = counter_at(t)        # refresh while healthy
        else:
            ticks_down += 1
            if base_atnotice is None:
                base_atnotice = counter_at(t)    # the event is already in here
        t += TICK
    return (ticks_down, 0, 0)

Real output:

episode                            ticks  at-notice  at-last-good
-----------------------------------------------------------------
1 tick down, single drop               1          0             1
1 tick down, single drop               1          0             1
2 ticks down, single drop              2          0             1
3 ticks down, re-flap at t=135         3          1             2

The at-notice column is the production tape: zeros everywhere until a re-flap appears, at
which point it finally reports one — and reports it one short. The at-last-good column is
right in every row. Note the third case: two ticks down, still zero at-notice. Length alone
does not rescue it. You need a second event, which is why the real separation was
one-directional.

The part that cost me ten days

There was a test. It was green the entire time. Here it is:

case "$(cat "$tmpd/s-carrval" 2>/dev/null)" in
  *" 77 "*) echo "  ok   the episode-open baseline is the counter's live value" ;;
  *) echo "  FAIL the open path did not stamp the live counter: ..." ;;
esac

Read the passing message. The condition it asserts is the defect, stated in plain English and
checked on every run. Somebody — me — looked at the open path, decided the baseline should be
the counter's live value at open, and wrote a gate to keep it that way. The gate did its job
perfectly for ten days: it pinned the bug in place and reported success.

This is worth more attention than the bug. A test encodes an assumption, and if the assumption
is the error, the test converts it from a mistake into a maintained invariant. Everything
around it can be right — and here it was; the reboot and counter-reset guards on this axis were
careful, correct and well-commented. They guarded the arithmetic. Nobody asked where the
baseline sat relative to the event.

And the tape had been saying so, in the clear, the whole time. Zero drops in 184 of 184
one-tick episodes, and zero one-tick episodes among all 17 that scored a drop, is not a
subtle statistical signal. It just needed someone to notice that a perfect split is a claim about the instrument.

What the fix has to do

Moving the sample is one line. Making the reading honest is four things:

Take the baseline at the last observation of the good state. Not at the notice, not at the
open — at the last tick that saw the link up. That moment provably precedes the drop.

Carry it in state, and publish its age with the reading. Every row now ends
carrier_base=lastup+121s. A delta is only as good as the moment its baseline was taken, and a
reader who cannot see that moment cannot judge the delta. If the last-good sample is 121 seconds
stale, say so in the row rather than making the reader assume.

When there is no baseline, render na — never 0. This is the trap that created the bug in
the first place, one ring out. An absent baseline with a 0 printed next to it is
indistinguishable from a real measurement of no drops, and the whole failure here was a
plausible constant standing in for a missing fact. Sampling now as a fallback would restore
exactly that.

Count the two eras apart. There are 343 historical rows and not one carries a trustworthy
baseline. The drift report now opens with BASELINE COVERAGE: 3 of 346 scored row(s) rather
than averaging a couple of hundred structural zeros into a claim about the radio. Rows written before a field exists
are not rows where the field was zero.

Red, then green

The proof is not a fixture. Within four minutes of deploy, unprovoked:

16:49:02Z RECOVERED shape=WEDGED           after 1 down-check(s) carrier_drops=1 carrier_base=lastup+121s
16:53:01Z RECOVERED shape=DEAUTHED-CARRIER after 1 down-check(s) carrier_drops=1 carrier_base=lastup+120s
17:17:03Z RECOVERED shape=DEAUTHED-CARRIER after 1 down-check(s) carrier_drops=1 carrier_base=lastup+121s

Three one-tick episodes, each scoring a drop. That combination had appeared zero times in the
210 preceding rows, and could not have. The two gate legs on the open path are now inverted so
they fail loudly if the old behaviour returns.

The generalisation

If you own a delta over a monotonic counter — SNMP interface errors, a rate() around an
incident window, retry counts, GC pauses bracketed by a health check, anything where you
subtract a stored reading from a live one to characterise an event — the question is not
whether the subtraction is right. It is where the baseline sits relative to the event.

Sample it when the fault is noticed and you have baked the fault into it. The arm you built to
say uncorroborated is then quietly measuring how long the episode lasted, and it will keep
doing that, consistently and green, for as long as you let it.

The diagnostic is cheap: look for a verdict that separates your data perfectly. A clean
split with zero exceptions is almost never a fact about the world. It is your instrument,
telling you which of its own properties you accidentally published.

Bounds, stated: ten of the NOCARRIER episodes have no corroborating kernel deauth line inside
their window — window-edge misses or a genuinely different fault, undecided at n=10. And this
fixes the corroboration label only. A separate defect is still open on the same tool: a 26-minute
window at 2-second sampling held five distinct down episodes where the 60-second poller logged
two, so every episode count here is a lower bound. That one needs an event plane, not a faster
poll.