Originally published on hexisteme notes.
I run a small fleet of AI agents on my own machine, and one of them has a nightly 03:00 cron job that re-parses a local corpus from scratch. It normally takes about 40 seconds. One night it took 1,725 seconds.
The hypothesis that survived by elimination
I tested the obvious suspects first: pathological input, O(n²) behavior in the parser, database fragmentation. I reproduced each one directly and ruled all three out. What was left wasn't a hypothesis I had positive evidence for — it was the one hypothesis nothing had killed yet: something else runs at 03:00, and it's competing for the machine.
That's a weak position to fix anything from. So instead of changing code, I built two pieces of infrastructure to test it properly.
An instrument. A temporary sampler for the 02:50–03:45 window, polling every 10 seconds: load average, vm_stat, disk activity, and the top CPU-consuming processes. It had a built-in expiry so it wouldn't linger as permanent infrastructure for a temporary question.
A falsifier. Around the same time, I'd wired a new integrity guard into the nightly chain. It ran for about 150 seconds and made the window heavier. So I pre-registered a condition: if WARN frequency rises, split the guard off into its own time slot.
An instrument to gather evidence, and a falsifier to say in advance what would make me revert a specific change. Both are things I'd tell anyone to do before touching a flaky nightly job. I did them.
The first real night looked perfect
Everything ran exactly as designed. The guard reported PASS 10 out of 10 times. The logs had their tokens. The early-warning condition fired — WARN frequency was up 2.52x. On paper this was a clean, complete recovery: a hypothesis under test, an instrument gathering data on it, and a safety net that had already caught something.
Both pieces of the setup were wrong.
Problem 1: the instrument was measuring the passage of time
The summary report had a line like this:
- Max cumulative Swapouts: 24,592,563
Laid out over five nights, the number climbed hard:
| Night | Reported value |
|---|---|
| Night 1 | 135,932 |
| Night 2 | 1,927,263 |
| Night 3 | 2,387,142 |
| Night 4 | 11,642,338 |
| Night 5 | 24,592,563 |
That's a 180x increase over five nights. Wall time climbed over the same five nights too, from 38.4s to 91.7s. The correlation looked airtight — memory pressure apparently getting rapidly worse, tracking almost perfectly with the slowdown.
I went back to the raw samples. The within-window increase was zero on all five nights.
vm_stat's Swapouts field is a counter that accumulates since boot. Taking max() of a cumulative-since-boot counter over any window just returns whatever the last sample happened to be, and that number keeps growing as long as the machine stays up, whether or not anything happens inside the window you're sampling. The field carried zero bits of information about the window it was supposedly reporting on.
What makes this worse than an ordinary bug is that this instrument existed for exactly one purpose: to adjudicate the "something else runs at 03:00" hypothesis. It produced a convincing, escalating, tightly-correlated fake signal on precisely the axis it was built to judge.
The real signal was sitting in the next column the entire time: within-window load1 average went from 1.78 to 8.54, and the CPU sum of processes cohabiting the window went from 0% to 611%. On a 10-core machine, something else was using six cores that weren't mine.
What I took from problem 1: a metric that describes an interval has to be defined as a delta — last sample minus first — never as a cumulative absolute value dropped straight into an interval report. If you have fewer than two samples, the honest output is n/a, not a fabricated number. A negative delta means a counter reset, not "recovery." And the fix isn't only the arithmetic — if you correct the value but leave the field labeled "max cumulative," the next person who reads it makes the exact same misreading I did.
Problem 2: the falsifier pointed at something that could not be the cause
WARN had genuinely fired, so the remedy I'd pre-written applied: split the guard into its own time slot. Read that conditional sentence on its own, and the action looks obvious — go execute it.
The timestamps stopped me:
Ingest 18:00:15Z -> 18:01:47Z (91.7s)
Guard 18:02:19Z start
The guard runs after ingest finishes. Something that starts after a job is already done cannot be the reason that job ran slow. If I had executed the remedy — moving the guard to a separate time slot — the WARN condition would have kept firing, completely unchanged, because the guard was never in the causal path in the first place.
Here's the structural problem with the sentence I'd written. A falsifier needs three parts: an observation, a causal claim you're asserting to be true when that observation holds, and a remedy. Mine had the observation ("if WARN frequency rises") and the remedy ("split the guard off"), but it never wrote down the causal claim — "the guard's 150 seconds makes the window heavy, and that's what slows ingest down." That claim existed, but only in the surrounding prose. Whoever comes back later to execute the recovery reads the conditional sentence, not the essay around it.
This can be worse than having no falsifier at all. With no falsifier, a person re-derives the cause from first principles. With one that looks complete, the situation gets filed as already anticipated, and the thinking stops right there — one step before the check that would have caught the error.
What I took from problem 2: write all three parts, every time — what's observed, the causal claim you're asserting is true at that moment, and what to do about it. And when you're the one recovering: check the causal claim before you execute the remedy. If the claim is false, hold the remedy and go re-derive the cause instead of running the pre-written fix.
What was actually running at 03:00
The real cause was outside the project entirely. A CLI from a different vendor was spawning 54 MCP server processes on every invocation and never reaping them on exit. Four days of that had piled up: 210 processes, 3.1GB of RSS, a CPU sum of 245%, and 119 of those processes didn't belong to any live session. After cleanup: 122 processes, 1.46GB, 0% CPU.
Even this diagnosis isn't confirmed yet. With n=5, all three series I have — wall time, load, and cohabiting CPU — are monotonically increasing by date, which means I can't distinguish "contention is the cause" from "some other date-monotonic factor" using this data alone. So I left a falsifier open for it: if the first night after cleanup comes back to the ~40-second baseline, that confirms it; if it stays around 90 seconds, the diagnosis is wrong. This time I wrote the causal claim down too.
The claim
Installing an instrument and installing a kill-condition are both good practice. Neither act is the same thing as validating that what you installed actually works. Each one needed its own acceptance test before the night it was supposed to matter, and I'd skipped that step for both. On the one night that mattered, the dashboard was green, the early warning fired, and the logs had their tokens — and both pieces of infrastructure were producing the wrong answer while doing it. "The pipeline runs" is proven by an exit code. "The pipeline is telling the truth" is not — that has to be checked by going back to the raw numbers and asking what they actually measure, and there's no green checkmark that does that step for you.
This is the sibling case to I wrote a rollback rule and no way to read the number — there, the criterion had no instrument behind it at all and the number was simply unreadable; here, the instrument and the falsifier both existed, both ran, and both were wrong anyway. Missing and invalid are different failures — this time the number lied instead of going unread.
More notes at hexisteme.github.io/notes.
Top comments (2)
The cumulative counter problem hits AI agent pipelines in the same way. I've seen token-usage dashboards report monotonically rising "cost" for a window because they read the billing counter at sample time instead of computing a delta. The chart correlates with wall time, the team panics about runaway spend, and the real cost inside the window hasn't moved.
Your three-part falsifier rule (observation, causal claim, remedy) is the part I'd steal first. Most of the runbooks I've inherited skip the causal claim entirely. They jump from "if this metric crosses threshold" straight to "restart the service," and nobody goes back to check whether the restart was actually connected to the metric. The fix gets filed as successful because the alert eventually clears, usually because something else changed.
The n=5 caveat at the end is the honest part that gets cut from most post-mortems. Five monotonically increasing data points can't distinguish your hypothesis from any other date-correlated variable, and most teams would have shipped the "confirmed" label anyway.
The billing-counter example is the same bug, one layer up: read the accumulator instead of computing the delta, and the chart tracks wall time instead of the thing you actually care about. It's a good sign the failure mode isn't specific to
vm_stat— it's specific to treating any monotonic counter as if sampling it were the same as measuring the window.Agreed on the causal claim being the part that goes missing first. It's the one clause that isn't needed to make the sentence sound actionable, which is exactly why it's the one that gets dropped — "if X, do Y" reads complete without it, right up until Y turns out to be aimed at the wrong process.
And no argument on n=5 — that section stops short of calling it confirmed for the reason you're describing, not despite it.