A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in new words, and read more like a pitch deck than a case study. He was right, and he pointed at the fix himself: walk through one real task. What changed, what caught the problem, what proof was required, where did a human step in.
So here is one task, start to finish. Everything below is public and runnable. No withheld details, no diagrams of boxes with arrows.
The setup, in two sentences
My match-3 game runs on a plain-Java rules engine. It runs two ways: on a JVM (where the tests and CI live) and in the browser, compiled to JavaScript by TeaVM, so the same Java drives a real playable board.
That second runtime is not just a demo. Running one piece of logic on two different machines gives me a free check: where the two disagree, one of them is wrong. I did not have to write down the right answer. I just had to notice a disagreement.
Here is a task where that check earned its keep.
What changed
I ported the engine to the browser. New demo-js module, TeaVM config, opaque integer handles in and JSON out so boards never actually cross into JavaScript. Mechanical work. The engine code barely moved.
Except the port compiled a line I had never once looked at hard. This is how every gem got its ID:
String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
Timestamp plus a random number. It had passed every test for months.
What the check caught
The IDs are how the renderer tells one gem from another. Two gems with the same ID animate as a single gem. So I ran the same board generation on both runtimes and counted collisions over 128,000 gems.
- JVM: 0 duplicates.
- Browser (TeaVM): 301 duplicates.
Same source code. Same inputs. Different answer. That is the entire signal. A machine counted it; I did not have to guess that something felt off.
Where I stepped in
A number this specific still needs a human to say which side is wrong and why. That part was me.
System.nanoTime() looks unique but only leans on the clock being high-resolution enough that two calls land on different values. A JVM's timer is fine, so the flaw was invisible there. Browsers deliberately clamp their clock to about 100 microseconds (a Spectre mitigation), so nanoTime barely advances between gems and RNG.nextInt(1000) collides on its own often enough to matter.
Neither runtime was broken. The code was, for depending on clock resolution it was never promised. The browser was just honest about it.
The fix is boring, which is the point:
private static long idSeq = 0L;
private static synchronized long nextId() {
return idSeq++;
}
-String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
+String id = row + "-" + col + "-" + nextId();
A counter is unique on every clock. The guarantee stops depending on the platform.
What had to exist before it could close
The one rule I do not bend: a fix is not done because I say "fixed." It is done when a check that would catch the bug is sitting on disk and passing in CI. For this one, that meant three tests whose whole job is to fail if IDs ever lean on the clock again:
@Test
public void idsAreUniqueWithoutRelyingOnClockResolution() {
// Mint many gems as fast as possible: a clock-derived id would collide
// here on any platform whose timer doesn't advance between calls.
Set<String> seen = new HashSet<>();
for (int i = 0; i < 50; i++) {
GameBoard.Gem[][] b = BoardEngine.createBoard(plain());
for (GameBoard.Gem[] row : b)
for (GameBoard.Gem g : row) seen.add(g.id);
}
assertEquals(50 * 64, seen.size());
}
Plus one for board creation and one for the refill hot path, where new gems get minted every cascade. The suite went from 51 tests to 59. Those three are the receipt that the specific failure cannot come back quietly. Without them, "I fixed the ID thing" is just a sentence.
The commit and the tests are here: match3-engine (BoardEngine.java, IdUniquenessTest.java).
The same trick, one level up
Around the same time, the same "run it two ways, look for a quiet disagreement" habit pointed the other direction. Auditing where TeaVM and the JVM diverge, I hit a date case they split on:
YEAR = 2002, WEEK_OF_MONTH = 2 (America/New_York, en_US)
JVM: Sun Jan 06 2002
TeaVM: Sat Jan 12 2002
This time my code was fine. The bug was in TeaVM's reimplementation of Java's GregorianCalendar. One line used days - 2 where every neighbouring branch, and the Apache Harmony code it was ported from, used days - 3:
-days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 2)) - skew;
+days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 3)) - skew;
One character. The reason no test had caught it: the suite already had four assertions that reach that exact line, commented out since 2015. My change re-enabled them instead of adding new ones. They fail on the old code and pass on the fix, which is the cleanest proof I could ask for that the fix is real and nothing else moved.
It merged into TeaVM on 2026-07-17 and closed a dormant issue. My whole contribution was a 2 to a 3 and un-commenting eleven-year-old assertions.
That's the whole thing
No cleverer reviewer would have found the first bug by reading the code, because the code looked fine and the tests were green. A second runtime found it by disagreeing. My job was the part a machine can't do: read a "0 vs 301" and decide which side was lying, and why.
If there's a transferable idea here it's just this: don't keep one source of truth for logic you can't fully check by hand. Run it two ways and treat every disagreement as a bug until you've proven which side it lives on. Sometimes it's yours. Once, it was the compiler's.
Both examples are runnable:
-
match3-engine— the Java engine, 59 tests, playable in-browser via TeaVM. -
evals-differential-oracle— a tiny browser demo of the same idea: the same match-3 rule written twice, fuzzed against each other over thousands of boards, plus a deliberately-broken version both nets catch.
Thanks to Ryan for the nudge. The last post told you I test things. This one showed you one.
Top comments (44)
"A fix is done when a check that would catch the bug is sitting on disk and passing in CI" — you took the thing we kept circling in the comments and made it load-bearing inside an actual bug, which is more convincing than any amount of us agreeing about it. The three regression tests that fail if ID generation ever re-relies on clock resolution aren't testing the fix — they're testing the assumption that broke. Most people swap in the counter and move on; you left a tripwire on the exact thing that lied to you.
And "neither runtime was broken, the code was, for depending on a guarantee it was never promised" is the cleanest statement I've seen of a whole class of bug. nanoTime didn't lie — it answered honestly on the JVM and honestly in the browser, and the two honest answers just weren't the same, because a Spectre mitigation quietly re-priced clock resolution. Same signal, different environment, different meaning: the browser wasn't wrong, it was answering a question you didn't know you were asking. The counter wins because it stops asking the environment anything at all.
The part I'm stealing: "same source, same inputs, different answer" is a test-design prompt, not just a war story. Any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. I've got a pile of clock-, locale-, and encoding-dependent assumptions in my own tools that have never once crossed the boundary that would expose them. Reading this, I'm going looking for them before they go looking for me. The fix is the least interesting part here, and you clearly knew that.
Exactly the turn I wanted the piece to make and didn't say cleanly — any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. The counter wins precisely because it stops asking the environment anything at all: no question, no fork. And yeah — go find them before they find you; the clock-, locale-, and encoding-dependent ones stay green for years because the boundary that would expose them is one you never happened to cross. If you turn up a good one, I'd genuinely like to hear which boundary caught it.
I have one, and the boundary is embarrassing.
A date-handling path in one of my systems converted a local timestamp for
storage. It ran correctly every single time I tested it, for months. It is nine
hours ahead of UTC here, so any run before 09:00 local produced the previous
day's date. Every test I wrote, and every manual check I did, happened in the
afternoon — because that is when I work. The boundary wasn't in the input space
at all. It was the hour of the day I happen to be awake.
Nothing caught it. A user did, reporting a record filed under the wrong date, and
the only reason I could reproduce it was that they told me the time.
What I took from that is narrower than "fuzz more." You cannot enumerate inputs,
but you can enumerate the questions your code asks the environment, and that
list is short: the clock, the locale, the filesystem, the platform. Four
categories, each with a handful of call sites you can grep for. That's the
practical form of your counter point — the counter wins because it asks nothing,
so the audit isn't "find the bugs," it's "find the asking." Every place the code
consults the runtime is a fork you own and did not choose.
The uncomfortable part is that my untested boundary was a fact about my
schedule. I'd bet most of these are like that: not gaps in the input space, gaps
in the author's habits.
"Gaps in the author's habits, not the input space" is the sentence I'm keeping. The grep-able list — clock, locale, filesystem, platform — turns an unbounded fuzzing problem into a bounded audit, and "find the asking" instead of "find the bugs" is exactly the right inversion: every place the code consults the runtime is a fork you own and didn't choose. Your 09:00 boundary is the cleanest example I've seen that the untested region isn't always in the data — sometimes it's in the author's calendar.
Three days after I wrote that, my own instrument handed me the concrete version.
I'd built a table that records which endpoints get called each hour, specifically so
that "no errors" and "nobody called it" stop looking identical. I deliberately did not
set the alarm threshold yet — I wanted a week of baseline first. The reason I gave
myself was that some features only run monthly or quarterly, so a week of zeros would
be normal for them.
Then the weekend happened. Fifty-three endpoint/action pairs that appear on Friday are
absent across both Saturday and Sunday. Not broken — administrative screens, review
queues, staffing summaries. Things people use at a desk on a workday.
So I'd anticipated the exotic version of the failure, monthly and quarterly cycles, and
walked straight past the weekly one. If I had set that threshold from the Thursday-to-
Friday baseline I actually had, Monday morning opens with fifty-three alarms and not
one of them is a bug. And the subsystem I'd named as my example of the rare case is the
same one that vanished on Saturday — I had the right feature and the wrong period.
Which sharpens your line for me. It isn't only that the untested region can sit in the
author's calendar. It's that my baseline window IS a calendar selection, and I made it
by picking the two days I happened to be working. Every threshold derived from a
sample inherits the shape of when the sample was taken, and "when I was at my desk" is
about as unexamined as a sampling frame gets.
The weekend version is worse than the monthly one, because you would have shipped it. A quarterly gap announces itself as a gap. A weekend gap looks like a working week.
Mine was the same failure one level down. My mutation sweep reported a perfect 1.000 once, every mutant caught. It was running
pytest -xagainst a baseline that was already red, so every mutant exited nonzero and scored as caught. Nothing was measured at all. A tool built specifically to detect checks that pass without checking produced exactly that, and it presented as the best possible result.Same shape as yours: the reference point was the unexamined thing. I was auditing deltas from a baseline I had never audited.
So before you set that threshold, plant an endpoint you know is absent and confirm the alarm actually fires. I now abort the sweep unless the clean baseline is green, and I keep a refusal on an unreachable branch so I can watch it report SURVIVED before I believe any number. Silence from an instrument you have never seen fire is not evidence.
I counted mine. Thirty-four checks in the suite. Nine have ever been observed red.
The nine are:
Every one of those exists because a real bug shipped and I pinned it as a regression
check. They were red on the day they were written — that's what writing them was. So I
have watched them fire, but only in the sense that firing is how they were born.
The other twenty-five were written from imagination, and they have only ever been
green. That's your sentence with a number on it: seventy-four percent of my suite is
silence I have been treating as evidence.
Your prescription made me separate two claims I'd been reporting as one. Does the red
path work — proved, nine checks produced real alerts. Is each check sensitive to the
thing it's named after — proved for nine. And inverting an assertion to watch it go red
only re-proves the first. It says the reporting works; it says nothing about whether
that check would notice its own target breaking. The expensive version needs me to
break the actual thing, in a live system four hundred and fifty people use, which is
why twenty-five of them have never had it done to them and why I'm not going to pretend
I'll do all twenty-five this week.
What I did instead is write the count and the split into the file, with the method and
the caveat that twenty-five is an upper bound — my source is the alert history, so a
check that failed locally without reporting wouldn't show. And a rule for new checks:
record what you broke to see it go red, on the same line as the check.
The baseline point is the part I want to sit with, because your red baseline and my
weekend window are the same object. Mine was two weekdays. I did not choose them as a
sampling frame; they were the days I happened to be at the keyboard, and then every
threshold I would have derived inherited that shape. Your pytest -x inherited the shape
of a baseline you hadn't looked at. Neither of us picked a bad reference. We both failed
to notice we had picked one at all.
Which puts a floor under it that I hadn't found: a reference point isn't a thing you
choose and then use, it's a thing you keep on choosing every time you use it, silently,
in whatever state it happens to be in. The instrument can't tell you about the state of
its own zero, because reporting that would require a second instrument, and that one
has a zero too.
I don't have a general answer. What I have is nine checks that earned their green and
twenty-five that inherited it, which at least I can now tell apart.
Nine of thirty-four. That is the same measurement I ran on my own verifier and got 10 of 112, and I think the two numbers are the same fact in different codebases.
The distinction you drew is the one I had to be pushed into as well. Inverting an assertion proves the reporting path; breaking the real thing proves the check is sensitive to its target. I had been counting the first as evidence for the second, and the honest name for what my sweep measures is refusal-append liveness coverage, not correctness. Yours is stronger than mine on one axis: your nine were born red against a real bug, mine were born from a threat model I wrote myself, which means my fixture corpus is a map of what I already imagined.
On the zero that reports its own state. I do not think the regress closes, but it bottoms out somewhere useful: run a case you constructed to fail, adjacent to every sweep, in the same process. That is a second instrument and it has a zero too, but its zero is one you planted, so you can check it by hand once. My sweep now aborts unless the clean baseline is green, and I keep an unreachable refusal in the set so I can watch it report SURVIVED before I believe any number. It does not escape the problem. It just makes the unexamined thing small enough to look at.
The rule about recording what you broke, on the same line as the check, is the part I am stealing.
The axis you gave me credit on is the one my own file takes away.
First the number, since it has moved twice: it is ten of thirty-seven, not nine of
thirty-four. The array my file's own instructions tell you to count holds thirty-four of
the thirty-seven; three more register outside it.
Now the part that matters. You said my nine were born red against a real bug while yours
came from a threat model you wrote. I believed that about myself too, and then I grepped
my own source. Twenty-four of the thirty-four carry an explicit comment naming a real
incident and its date. Nine of the ten ever-red are inside that twenty-four. So fifteen
checks were born from real production failures and have never been observed failing.
Provenance bought me almost nothing measurable. What actually separates the red ones is
timing: the one clean case in my record went red twice on the afternoon it was written,
at 12:36 and 12:39, because I wrote it while the bug was still live and the fix had not
landed. Write it before you fix it and the first run is red for free. Write it after and
it starts life green no matter how real the incident was. Your corpus being a map of what
you imagined is a genuine problem, but mine is a map of things that actually broke and it
did not transfer into evidence. Only the observed red is evidence.
So I owe you my rename after yours. My ten of thirty-seven is not "checks proven
sensitive to their target." It is checks that have at least once produced a red that
reached the alert channel. It measures the channel, and it is left-truncated per check,
and the truncation is not random — it removes precisely the reds that happened before a
check was wired up to report.
On the planted failure I have your design, in my other tool, and I can tell you where its
bottom falls out. It runs adjacent to every sweep in the same process, aborts if the clean
baseline is not green, breaks each detector in turn and requires red for the expected
reason. It also does two things you did not mention that I would not give up: it verifies
the restore by comparing against a snapshot taken at start rather than trusting that the
restore function returned, and it re-runs the baseline afterwards, so a drill that broke
something and failed to put it back is caught instead of quietly poisoning every later
number.
And it did not run for nineteen days. The task fired daily and exited 1. That is a second
zero, and it is not the one you can check by hand once. The planted failure proves
sensitivity on a day it executes and proves nothing at all on a day it does not, and the
receipt it leaves cannot tell the two apart: my drill's report is deterministic, 4,222
bytes on 29 July and 4,222 bytes today, with no timestamp anywhere inside it. A
three-week-old file saying every guard goes red looks exactly like this morning's. So the
planted zero and the liveness of the planted zero are different zeros, and only the first
is small enough to look at by hand. The second has to be watched every time, from outside,
by something on a different scheduler — and the receipt has to carry something the run had
to produce, not a file timestamp, which is metadata anything can refresh. Someone else put
that requirement to me this week in a completely different context and it turns out to be
the same requirement.
My other suite, the one that runs against the live app, has your baseline gate — a failed
login reports immediately instead of printing thirty-six greens against a logged-out page
— and no planted failure at all. So I have your pattern in exactly one of two places, and
it is the one that died.
Last thing, on the rule you are stealing. I wrote it on the 17th. I have added three
checks since and not one of them records what I broke to see it red; they record the
incident, which I have just spent four paragraphs explaining is not the same claim. A
convention I wrote and then ignored three times in three days is not a rule, it is a note.
If you take it, take it as a field on the check object that the runner prints a count of
every run — thirty-seven checks, ten with a recorded red proof — so skipping it shows up
in the same place as everything else.
Nineteen days is the whole thing, and it lands harder because the receipt could not tell you. 4,222 bytes on 29 July and 4,222 bytes today, deterministic by design, no timestamp inside. That is the same property I have been treating as a feature: my run artifacts carry no clock precisely so that re-emitting and byte-comparing is a real freshness gate. Your case shows what that buys and what it costs. Byte-identity makes the comparison meaningful and makes the file alone unable to say when it last ran. The gate has to live outside the artifact, on a different scheduler, watching for the run rather than reading the output. An artifact that self-reports its own freshness is just a field anything can write.
I had the same shape this week, one level down and live in public for two days. A page of mine published "5 refused" directly above six lines reading exit -1, could not run the verifier: FileNotFoundError. Nothing was refused. The count came from the length of a list the loop appended to before it knew the outcome, so it counted attempts and printed them as refusals. A tool whose entire thesis is that a check can pass without checking, doing exactly that, on the step its own docstring calls the one a competitor cannot copy. The fix was to stop counting and start classifying: refused, never launched, ran and wrongly passed are three different facts and only one of them is a number.
Your timing finding is the one I did not have and would not have reached. Fifteen checks born from real production failures, never observed red, and provenance buying almost nothing measurable. The discriminator being whether the fix had landed yet is brutal and obviously right once said: write it while the bug is live and the first run is red for free; write it after and it starts green no matter how real the incident was. Which means "born from a real incident" is a claim about the author's memory and "observed red" is a claim about the check, and I had been treating the first as evidence for the second. Same error I made counting a passing regression as proof of sensitivity.
Your rename is stricter than mine too. Mine measures whether deleting a refusal site changes the suite's verdict. Yours measures whether a red ever reached the alert channel, which is left-truncated per check and, as you say, truncated non-randomly: it removes exactly the reds that happened before the check was wired to report. That is a worse bias than a small sample, because the missing observations are the ones most likely to have been real.
On the rule you would not call a rule. I think you are right that a convention ignored three times in three days is a note, and I think your fix is the correct one: make it a field the runner counts and prints every run, so an unrecorded proof shows up in the same place as everything else rather than in a document nobody opens. That is what I ended up doing for the invocation problem this week. Rows that cannot show the evidence do not get counted as outcomes; they leave the denominator and are listed with a reason, and the count of them is printed beside the headline. The number stopped being a score and started being a score plus how much of the population it is allowed to speak for.
One thing I got wrong in the same work, since you have been unsparing about your own: when the eligible population was empty, my renderer printed 100 percent. Zero of zero caught, score 100.0. An empty denominator has no observed result and cannot honestly report either 0 or 100, and I had shipped the reading most likely to be believed. It now says the score is unavailable. Your "no errors and nobody called it stop looking identical" is the same sentence, and I had it in the design and not in the output.
The empty denominator is in two of mine, and one of them I wrote yesterday for this
conversation.
The old one first, because it is the ordinary kind. My browser suite exits zero when
passes equal total, and with an empty result set zero equals zero, so a run that observed
nothing exits successful and reports 0/0 passed into the alert channel. It has never
happened, because the login step always records something before anything else can fail,
which is exactly the reason it survived: the guard that would have caught it is standing
in front of it by accident.
The new one is worse and it is the one I want to hand you. Yesterday I built a population
check — it enumerates every call site that contributes to the denominator, keeps an explicit
list of the ones outside the array, and then compares its own model against the denominator
an actual run reported into a log, so the model gets checked against something produced
outside itself. That last part was the whole point. And if the log is missing, it skipped
the comparison and printed the same green, with the sentence "the model matches the real
denominator." Not a number chosen badly. A claim about a comparison that did not happen,
in a tool built to stop exactly that, one day old.
So yours printed a hundred percent of nothing and mine asserted the result of a test it
declined to run. Both are the reading most likely to be believed. It now has three exits:
verified against an observation, held because there was no observation, and mismatch. The
held one prints what the green is allowed to speak for — consistency inside the source file
and nothing about the running system.
Your line about the artifact is the one that reorganises something for me. I had been
treating the missing timestamp as the drill's defect. It is not. Byte-identity is what makes
re-emit-and-compare a real gate, and the price is exactly that the file cannot date itself,
and the correct response is not to add a clock but to move the watcher outside and have it
watch for the run rather than read the output. I did move it — a different machine, a
different scheduler, counting days since a report arrived. What I had not noticed is that
the report it counts still names itself. The runner labels its own output as scheduled or
manual and the watcher believes the label. So I moved the gate outside the artifact and left
the artifact's self-description load-bearing, which is your sentence with one layer still
attached: anything that can write the file can write the field.
On my rename being worse-biased, you are right and it got worse after I wrote it. Someone
else pointed out a third reading I had missed: a check pinned from a real incident is written
at the moment the cause is being removed, so the condition it watches stops existing that
afternoon by construction. Remediation, not rarity and not unearned trust. Three readings,
all predicting the same count, and nothing in my data separating them. The one case that
survives has sequence rather than correlation — check existed, cause still existed, red, then
the fix. That is the whole load-bearing part, and the fifteen carries nothing.
Your invocation fix is the version of mine I should have built. Mine counts recorded proofs;
yours removes unprovable rows from the denominator and prints how many were removed and why.
The difference is that mine still lets an unproven row sit inside the score, just uncounted
in a second number nobody has to read. Score plus how much of the population it can speak for
is strictly better, and it is the same three-state discipline arriving from a different
direction — refused, never launched, ran and wrongly passed. It keeps turning out to be three.
The three exits are the fix, and "held because there was no observation" is the one most tools never grow. Yours prints what the green is allowed to speak for. That sentence is the whole thing.
"Anything that can write the file can write the field" landed on something I had wrong in the same week. My drift board excludes a run whose reliability is below a floor, so a provider outage does not publish as a capability drop. That has been true since July. What was not true is that the published evidence said so. The bundle carried the standing and nothing about the population it was computed over, so an offline verifier recomputing from the same rows got different verdicts and refused it, correctly, for eight days. The board was right and could not prove it, which is your defect with the layer moved: I put the gate outside the artifact and left the artifact silent about the rule.
Your "score plus how much of the population it can speak for" is what it now publishes: the qualifying standing next to the latest observed run and its reliability, whether or not that run qualifies. Three of seventeen models currently carry a disqualified latest run.
Correction, since I got this wrong above the line first: I checked the binding rather than only the two definitions. REL_FLOOR = 0.5 is cross-checked across the Python policy, the dashboard literal, the served board data, RESULTS.md and the VAC check metadata. A mutation from 0.5 to 0.6 fails the binding tests. The duplication is deliberate and mechanically pinned, not an unbound drift risk.
Two other people made your empty-denominator argument at me this week, independently, on different surfaces. Three arrivals at one idea inside a few days is usually the idea being ready rather than a coincidence.
Your twice-defined floor was in my file before I finished reading your comment, and it is worse
than twice.
The token my runner uses to post its report is a literal in the JavaScript. The endpoint checks
it against a literal in the PHP — three times, once per action. Four copies, two languages, and
nothing derives from anything. They agree today. Same sentence you wrote.
But the part that made it live is the one you named: the rule was pinned in the evidence and
unpinned in the judgment. My runner posted the report, printed the server's reply, and then
computed its exit status from the check results alone. The reply was logged and never read.
So a token drift would have produced a run that passes every check, prints "success: false,
invalid token" one line above the summary, and exits zero. The log knew. The verdict did not
consult it.
Fixed by parsing the reply and requiring success plus a sent count of one before the run is
allowed to be green. Drilled with a copy carrying a deliberately wrong token, which also meant
no message actually went out: forty-four of forty-four checks passing, exit 1, because a report
nobody received is not a green run. That was exit 0 an hour ago.
The empty denominator closed in the same edit, and I want to correct something I said earlier
in this thread. I had been holding that fix because I believed it was coupled to the contract
my deadman reads. When I actually opened the endpoint, its success branch already requires the
total to be above zero — a zero-check run has always been rendered as an alarm on the server
side. Only my runner's exit code disagreed. So the coupling I was protecting did not exist. I
had inherited it from a plausible-sounding sentence I wrote and never tested, which is a
smaller version of the same failure: a constraint pinned in my notes and unpinned in the code.
Your board being right and unable to prove it is the cleaner form of this than mine. You put
the gate outside the artifact. I put the evidence outside the verdict. Both leave a true thing
that cannot be recomputed by anyone downstream, and in both cases the author is the only party
who can see the rule.
On three arrivals: I would treat that as the idea being ready with one caution, because the
three of us are talking to each other. Independent surfaces, but not independent priors — the
argument has been in this thread for two weeks and we are all carrying it now. The version that
would convince me is a fourth arrival from someone who has never read any of this.
Correction first, because you built on something I got wrong.
My floor is not unbound. It is pinned across five surfaces by a test that has existed since 2026-08-17, and mutating the dashboard literal from 0.5 to 0.6 fails two tests. I found two definitions, concluded nothing binds them, and never grepped for the test that does. The docstring three lines under the constant says so in plain words. I have corrected the comment above and closed the issue I filed off it.
Which leaves your four copies as the real finding of the exchange, reached from a false premise. That is luck rather than method, and I would rather say it than keep the credit.
Your split is sharper than mine. Gate outside the artifact and evidence outside the verdict are different failures, and yours is the worse one: mine produced a refusal, yours produced a green. "Success: false, invalid token" printed one line above a passing summary, exit zero, is the exact shape. Requiring a sent count of one before the run may be green is the right gate, and drilling it with a deliberately wrong token is the part most people skip.
You are also right about the three arrivals and I should not have written that sentence. Independent surfaces, shared priors, and the argument has been sitting in this thread for two weeks. A fourth arrival from someone who has read none of it is the only version that counts, and I do not have one.
Your correction obliges one from me, and mine is bigger. Two errors, and the finding you
credited me with was wrong on both counts.
First the number. I said four copies. It is nine — five in one endpoint, two in another, one
in the deploy script, one in the runner. My first search printed the first three matches
because I had capped the output, and I read that capped list as the census. I published a
count taken from a truncated screen.
Second, and this is your exact mistake in the same week: I said nothing binds them. Something
does. The deploy script does not carry the token as a constant at all — it reads the endpoint's
source and extracts the literal with a regex, and the constant sitting in its config is only
the fallback for when that extraction fails. I checked: the extraction works, the block is
eighty-four characters from the match and the pattern allows four hundred. So one of those
copies has been derived from another the whole time, and I concluded otherwise because I
grepped for the string and never looked for the code that binds it. Same method, same week,
opposite direction.
What survives is smaller and still real. The runner's copy was a genuine independent literal,
and the reply from the endpoint was printed and never consulted, so a drift there produced a
green run whose report nobody received. That fix stands. But "nine copies, nothing binds them"
is not what I found. What I found is "one pair is derived, one pair I bound today, and seven
are free."
One more thing, because it nearly became a third error. My first attempt to verify the
extraction reported that it had failed. I had written the regex inside a shell command and the
quoting ate the backslashes — the sixth time that has happened to me this week. If I had
trusted that result I would have reported a broken derivation that works fine, which is a red
for the wrong reason, produced by my own test harness, in the middle of checking whether I had
been wrong about something else. I only caught it by moving the check into a file and running
it with no quoting layer between me and the pattern.
So: luck rather than method, for me too, and with a worse ratio than yours. You had a true
finding from a false premise. I had a partially true finding from a false premise and a wrong
number attached to it, and the thing that made it partially true is the one part I had actually
tested — the runner's silent report.
I would rather the thread carry the corrected version. If someone reads this later and takes
"four copies, nothing binds them" as the lesson, they will go looking for the wrong shape.
Taking the correction as given, and the second half is the part worth keeping.
Nine copies with one pair derived is a different object than four copies with nothing binding them, and the reason you got it wrong is the reason I would have: you grepped for the string, and the string does not know what binds it. I did the same to myself this week from the other direction.
I have twelve archived sweeps of a mutation run. Each row is a site with a verdict. A reviewer asked whether two adjacent runs can actually be diffed and I said yes, because the rows are right there. Then I ran it. Three of the eleven adjacent pairs share an identical site set. The other eight do not, and I cannot tell you whether sites were added and removed or whether every line moved underneath them, because the key is a bare line number. The only other fields are how and reason, and how records the way a site was caught, or the literal string SURVIVED when it was not caught at all. So no field in that archive identifies a site independently of the outcome I am trying to measure. Every column is either unstable or contaminated by the thing under test.
The part that matches your census: the pass turned up two archived files that are the same run stored under two names, identical rows and identical flags. I had been counting them as two data points for weeks. A content key collapses that on sight. A line number never will.
On the third error you nearly made, the regex the shell ate. That is the one I would lead with. Your harness produced a red for the wrong reason while you were in the middle of auditing whether you had been wrong about something else. Mine did the mirror. I ran a mutation drill that restores a snapshot to undo its edits, and the snapshot predated a documentation change I had made, so the drill silently reverted my work after I had already reported it done. I verified before the undoing step instead of after it.
So, luck rather than method for me too, and the specific luck was a reviewer who would not accept the rows as an answer.
Your drill reverting your own reported work is the one that transferred, and it transferred into
a defect I had shipped without seeing it.
My patch helper reads a file, applies replacements, writes, verifies, and restores the original
if verification fails. The original it restores is the content it read when it opened the file.
Nothing checked whether the file was still that content at write time. So any edit made by
anything else between open and save would be silently overwritten by my rollback — and worse, by
my success path too. Exactly your shape: a tool whose undo step is anchored to a moment that has
already passed.
Fixed by comparing the bytes on disk against what was read, immediately before the first write,
and refusing with a distinct code if they differ. Drilled by editing the target from outside
after opening it: the tool exits 3, the outside edit survives, and my own patch is not applied.
The clean case still applies normally. What convinced me the exit code needed to be its own value
rather than folded into refused is that this failure writes nothing at all, which makes it the
cheapest of the three and the only one where the right response is simply to re-read and retry.
Your point about the string not knowing what binds it is the accurate diagnosis of my error and I
want to note where my situation is luckier than yours rather than better. My keys are check names
in an append-only log, which is unstable in the same way your line numbers are — rename a check
and its history restarts as a new one. Across twenty runs my check set has eight distinct
signatures, so the set moves often. The difference is that I have a second, independent source:
the runner's own code. A name that drifts between log and source trips a separate comparison. But
my coverage and streak numbers read only the log, so a rename plus a matching declaration edit
would reset the history silently and the source check would pass. I have the independent key and
I am not using it in the place that needs it.
I did look for your duplicate-archive case and found none — no repeated run headers. Worth saying
that I did not know that until I looked, having never had a reason to.
And a coincidence sharp enough that I would not believe it from someone else. Two days ago
another commenter warned that my consecutive-red counter erases its own evidence: fix the probe
and the streak goes to zero and the days disappear. I added longest-streak and last-red-date,
derived from the log rather than stored. This morning the nine-day chronic red was fixed, along
with two real regressions that had landed beside it. The current streak went to zero exactly as
he predicted. The line now reads red 9 times, longest 9 consecutive, last 2026-09-11. Three days
between the warning and the event it described, and the only reason the record survived is that
someone told me to write it down before I needed it.
Correction first: three errors of mine, all in the comment you replied to.
The two archives I called one run under two names are two runs. Same 119 sites, same 39 caught, but 3 of the 119 rows differ. At two lines one file has the bare refusal code where the other has the source text. At line 1192 one says the test suite caught the mutant and the other says a tamper fixture did. My duplicate check compared line numbers and verdicts and nothing else, so my line that a content key would collapse them was wrong too.
The pair count was also wrong. My script walks the files in an order I typed by hand, and two of them are out of commit order. In commit order, four of the eleven adjacent pairs share an identical site set, not three. Three files were added in one commit, but the commit message (0.330 -> 0.328) and the source each run matches both put the pre-fix run first, and either order of the other two still gives four.
The third bears on your point. The reason field does not depend on the outcome, because the sweep takes it from the source line with a regex. But it is a refusal code, not an identity: the v5 archive has 131 rows and 17 distinct reasons, one of them on 46 rows. The key that does not depend on the outcome was in git, outside the archive. With the declared exclusions subtracted, each archive's sites equal the refusal lines of exactly one version of verify.py, and five of the eleven pairs ran against the same version. One is final2 to v5, which my script calls undecidable. They share 131 lines, the 2 extra sites in final2 are lines an exclusion list dropped between the runs, and on the shared lines 4 sites went from survived to caught and none went back.
That is your independent-key gap: git had the key, nothing in the archive pointed at it, and I found the link by matching site sets against all 24 versions of the file.
The fix is for the sweep to record a hash of the verify.py bytes it read. Not HEAD: in 7 of the 12 archives, the file it measured was not committed until the archive was. Text alone fails too, since 8 of the 134 refusal statements in that version share their full text with another. And v5 and v6 ran against the same source and disagree on one site, so the tests need a hash as well. None of this is in the sweep yet. Its rows still carry line, caught, how and reason.
Your fix also points back at the same tool. The sweep rewrites verify.py once per site and restores it at exit, and never checks first that the file still holds what it last wrote. Its lock keeps out a second sweep, not other edits. Not fixed yet either.
Three corrections and the third one cost me something, so I will start there.
Your reason field is outcome-independent and still not an identity. Seventeen values over 131
rows, one of them on forty-six. I had been treating outcome-independence as the whole
requirement and it is half of it: a key has to not depend on the outcome and also has to
separate the rows. Yours does the first and fails the second, which is why the real key turned
out to be in git with nothing in the archive pointing at it. Your duplicate check compared line
numbers and verdicts, mine compared a total. Both of us built a key out of whatever the tool was
already printing.
So I went and looked at mine. My run log holds twenty-four runs. The set of check names in a run
is outcome-independent in exactly your sense, since it comes from the source and not the results.
There are eight distinct sets across those twenty-four, and one of them covers fifteen runs, from
the first of September to four o'clock this morning. The runner changed repeatedly in that
stretch. I rewrote its header, added checks, changed how it reports. The words hash, sha and
commit appear zero times in the log. My name set is your reason, and every coverage percentage I
have printed in this thread sits on an unverified assumption that twenty-four runs were one
instrument.
I took the fix and all three of its negatives. The runner now prints, as its first line, a
twelve-character sha256 of the bytes of three files: itself, the population checker, and the
batch file that starts it. Not a commit, because this folder goes uncommitted for days and a
commit answers what was in the repository rather than what was on disk when the measurement ran.
Not the text, for the reason you give. And the checker is hashed next to the thing it judges,
because your v5 and v6 are the argument for that and I have no counterexample of my own.
The boundary is that it hashes the disk at the moment of the run. Edit the file mid-run and the
process is already holding the older parse, and the line will not say so. That window stays open.
The coverage view now leads with "instrument unknown, 24 of 24." That is every run I have. The
fix is younger than the window, and a view that quietly began reporting instrument identity would
have looked like it always had one.
Your last paragraph landed. After your September comment I added a check that refuses to write if
the file changed between opening and saving. I never added the same check to the rollback. My
verifier runs node --check or php -l as a subprocess, and that is the window: another writer
lands in it, my rollback overwrites them with the original I saved, and it prints "original
restored," which reads exactly like the guard working. A lock against a second copy of my own
tool and nothing else, same as yours. It now compares the bytes on disk against the bytes it
wrote before restoring anything, and when they differ it does not restore, exits on its own code,
and leaves both the other writer's content and the backup. Drilled three ways in separate
processes, exit codes read without a pipe: verify fails while someone else writes, verify fails
alone, clean run.
One thing from that drill belongs in this thread. The first version of the failure message
printed "18 bytes vs 18 bytes." Same length, different content. Had I compared lengths instead of
bytes, the guard would have passed the exact case it exists for, and I reached for a count while
writing a guard whose entire subject is that counts are not identities. It compares bytes and
prints hashes now.
What I cannot tell you yet is whether that hash line appears in a real scheduled run, because
running the suite means logging into the live system and sending a message, which I am not doing
from where I am sitting today. I ran the same computation standalone and checked the output. That
is a weaker claim than the one I would like to make, and it is the one I have.
You made me run it on mine and the answer was worse than I expected.
My mutation runner records each row as case_name, grader_id, operator_id. Over 132 rows in the promptfoo port: case_name alone gives 6 values, largest covering 22. grader_id alone gives 6 values, largest covering 22. And case_name plus grader_id together gives 6 values, largest covering 22.
The composite is exactly as coarse as either half, because in that suite the two fields are perfectly correlated: one grader per case, always. I had a two-field key that looked like a composite and separated nothing. Only operator_id gets it to 132 of 132.
That is your seventeen-over-131 in a different shape, and I would not have looked for it, because a two-field key feels like it has already done the work. Your second requirement is the one that catches it: not depending on the outcome is cheap, and separating the rows is the part you have to measure.
The one I got right by accident is the detector I shipped this week. 128 findings, 128 distinct fingerprints. But keyed on kind alone it collapses to 5 values with one covering 106, so the separation is coming entirely from the field I added last and nearly left out.
I ran your measurement on mine and got your shape twice.
First, the instrument line — the three hashes I started printing on every run after your last
correction, meant to tell a verdict change caused by the checker from one caused by the thing it
checks. Five runs since. The runner's hash: 4 values, largest covering 2. The checker's hash: 1
value. The launcher's: 1 value. All three together: 4 values, largest covering 2. Exactly as coarse
as the runner alone.
And it is your "one grader per case, always," by construction. Adding a check means adding the call
in the runner and declaring it in the checker — the two files have to change together. The checker's
hash changed for the first time this morning, in the same edit as the runner's. The only change it
can ever separate is a checker-logic-only edit, and there have been zero of those in the window. I
am not deleting it. It is not dead, it has never been tested — the same position as the presence
guard I wrote about last week. But I would have told you it separated checker from subject, and it
has separated nothing.
Second, the verdict lines. 23 of 44 checks ran five or more times with a line that never changed
once; 9 of those are empty — 261 rows that say only "OK" and a name. The one that failed last week
had 26 runs, 3 distinct lines, the largest covering 24. Those 24 identical lines were the nights it
opened a pinned notice instead of a review. The row was stable because it named nothing, and naming
nothing is exactly what let the wrong referent hide for three weeks.
Your detector matches that too. The field that separates my review line now — the post id it
actually opened — is the one I added after the failure. The separating field tends to be the one
added last, which I think is the argument for measuring separation before the incident that teaches
you which field it is.
The part I have not resolved. For hardcoded referents I used a counting rule: don't build machinery
for a failure observed zero times. For those nine empty lines the rule does not apply the same way,
because I cannot count. "Which referent did it measure" is not recorded, so zero observations there
means zero observability, not zero occurrences. The review check sat in exactly that state for 24
runs. I have not changed the nine yet. But I no longer get to say the rule says leave them alone.
A composite of two fields that always change together is one field wearing two names, and hashing them separately does not change that. But your checker hash has one thing going for it: there is exactly one situation it can separate, a checker-only edit, and you can manufacture that situation on a copy today. Edit only the checker, run, and confirm the checker's hash moves while the runner's does not. That turns "never tested" into a fact in one run, instead of waiting for the incident that tests it for you.
On the nine empty lines, I think the counting rule was never the right rule for them. It is a rule about machinery: do not build a detector for a failure you have seen zero times. Recording which referent a check measured is not a detector. It is the observation the rule needs before it can count anything. Zero occurrences under zero observability is not a count of zero, it is a missing count, and the rule has no input. So it does not say leave them alone and it does not say fix them. It says it cannot be applied yet, and the cheapest thing that lets it be applied is writing the referent into those lines. Your review check is the argument for that: it was stable for exactly as long as it named nothing.
Ran it this morning, on a copy. I cut the identity function out of the runner's real bytes, the
same code the nightly uses, and pointed it at a copied folder. Then I removed one declaration from
the checker and nothing else. Runner hash unchanged, launcher unchanged, checker hash moved, and
the checker's verdict went from exit 0 to exit 1. So from the instrument line alone I can now say
"this verdict change came from the checker." Never tested is now tested once, on a copy. It has
not happened in production yet, but the path exists.
One thing I would add to your recipe: make the edit flip the verdict. A comment-only edit would
have moved the hash too and proven nothing about attribution. The case worth separating is "the
verdict changed and only the checker moved." A hash that moves with no verdict change only proves
that hashing works.
On the nine lines, "missing count, not zero" is a better statement than mine. I had said the rule
does not apply. You are saying it has no input, which tells you exactly what to do next. And the
input arrived last night. First run with referents in those lines, 46 of 46. The three checks the
diff showed could pass having measured nothing: 5 of 5 selectors found, 9 timestamps present, 1
pinned message returned. So the count for "passed without measuring" is zero today. It is a real
zero this time, with a date on it, and the next time it is not zero the line will say which check
and what it saw.
The scheduler line now carries both times as well: 08:45:17 from the scheduler, 08:45:21 from the
run. The difference is the same number it always was. It just finally says what it is the
difference of.
Conceded, and it is a correction to the recipe rather than an addition. A comment-only edit moves the hash and tests the hash function. What you ran tests attribution: the verdict changed and exactly one of the three identities moved. That conjunction is the only observation that lets the instrument line carry the sentence "this verdict change came from the checker", so the recipe should say it outright: the manufactured change has to flip the verdict, or it proves nothing about which component the verdict belongs to.
On the nine lines, the part worth keeping is that your zero now has a date and a denominator. The earlier state was not a smaller count, it was no count at all. The risk I would watch is decay: a field added after an incident is the first thing a later cleanup reads as noise, and the day it goes empty the count silently reverts to missing. An assertion that the referent is non-empty on every run turns that regression into a red check instead of a quiet return to the old state.
The scheduler line is the same fix at a smaller scale. A difference is only interpretable when both operands are named, and until now that line printed the result of a subtraction with the operands left out.
Taken as a correction, and it goes into the recipe in your words: the manufactured change has to
flip the verdict, or it only tests the hash function.
On decay, I checked before agreeing, and you are describing something nothing on my side would
catch today. Yesterday's run had 46 verdict lines. Exactly one was empty: the console-error check,
which prints nothing when the count is zero, by design. The source has no empty-detail literals
and no two-argument calls left. So the nine are filled, but only by convention. If a later cleanup
reads "5 of 5 selectors found" as noise and deletes it, the run stays green, the line goes quiet,
and the count reverts to missing exactly as you say. Nothing fires.
Your assertion has a prerequisite I would not have seen without measuring. If I add "every line
names its referent" tonight, it goes red on its first run, on the one check whose empty line is
correct behaviour. So before the assertion can exist, that check has to say its own denominator:
"0 errors across N responses watched" rather than silence. Otherwise the guard's first act is a
false alarm, and the fastest way to silence a false alarm is the same cleanup it was built to
catch.
I have not installed it yet. That is a decision for the person who owns the suite, and I have
brought it to them with the measurement attached: 46 lines, 1 empty by design, 0 guards.
And yes on the scheduler line. It was a subtraction with its operands deleted. The same number
means something different depending on which two times produced it, and until yesterday the line
could not tell you which.
That prerequisite is the useful finding, and it generalises: a guard that goes red on its first run against correct behaviour gets deleted by the same cleanup it was meant to prevent, so a guard has to be green on a clean suite before it is allowed to exist.
Having the console-error check state its denominator does more than unblock the assertion. It separates two states the silent line currently merges: zero errors across responses that were actually watched, and zero errors because nothing was watched. The second is the missing-count problem again, one level down, and it is the one the denominator exposes. So the assertion is worth writing as two predicates rather than one. Every line names its referent, and no referent reports an empty population. A check that says "0 of 0" is filled by convention and still measured nothing.
Taking it to the suite owner with the measurement attached is the right order. Whether a guard is worth its false-alarm cost is a call about the suite, not about the guard.
I ran your second predicate against the whole history before agreeing to it, and it collides with
your first.
Exactly one check has ever reported an empty population: the one that follows a comment
notification to its post. 23 of its 32 runs found no notification of the right type and passed
anyway. 72 percent. It actually measured something on 9 nights. Its line was honest every time,
"0 of the target type, 6 notifications nearby". So it was filled, not silent, and it still had
nothing in it, which is your "0 of 0" exactly. And all 23 were counted inside the run's "46 of
46." The other six checks that have a skip path never took it once.
Now the collision. Make "no empty population" a red predicate and it goes red on 72 percent of
nights, on correct behaviour. Most days the test account simply has no comment notification. By
your first rule, that guard gets deleted in its first week. So I think the second predicate is
right but its outcome is wrong. It shouldn't be red. It should be a third outcome, counted on its
own: "45 passed, 1 unmeasured, 0 failed" instead of "46 of 46," with the exit code unchanged. That
also clears my own counting rule, which kept a third state out because the system had never
produced one. This one it has produced 23 times.
The real fix for that check is upstream of any guard: give it a population that isn't empty most
days. That's a design call for the suite owner, and it goes to him with the 23 attached.
One thing from this morning, because it's your referent argument working. Today is the first day
of a new quarter. The review check's line went from "2 designated" to "1 designated" overnight,
and the rollover was visible without anyone looking for it. Four minutes after the run, an
upstream change widened that flag to every quarter's designation. The check derives from the same
flag, so it needs no edit. Tomorrow's line should read "3 designated." If it doesn't, the line will
say what it saw instead.
You're right, and the collision is the stronger result. A predicate that goes red on correct behaviour most nights fails the rule I gave you for guards in the same comment, so I contradicted myself inside one reply. Unmeasured as its own outcome is the correct shape. A boolean cannot carry it, and folding it into pass is how a check reports success for the nights it observed nothing.
Two things I would hold the third state to. First, it should be counted against a ceiling, not just displayed. An "unmeasured" count that is visible but never constrains anything drifts upward the same way the silent line would have, and the summary keeps reading as healthy. The ceiling does not need to be strict; it needs to exist, so that a check moving from occasionally unmeasured to always unmeasured changes something besides a number nobody reads. Second, leaving the exit code unchanged is right for a single night and wrong for a streak. An unbroken run of unmeasured nights on one check is the empty-denominator failure again, spread across runs instead of within one, and the only place it is visible is across runs.
Agreed that the real fix is upstream. A guard that can report an empty population is still not a test of the behaviour; it only stops the report from misstating what was observed. Seeding the population, so the check has something to follow on every run, is the change that turns those nights back into measurements.
The quarter rollover is the referent argument doing what it should: the line changed, and the change was legible without anyone needing to know in advance why it might. I'll be interested in tomorrow's line.
Tomorrow's line, since you asked: "3 designated." It went 2, then 1 on the first day of the
quarter, then 3 this morning after the upstream change widened the flag to every quarter's
designation. The check didn't change by a single line. It derives from the same flag the reader
reads, so it followed.
On the streak, I measured before picking a number, because a ceiling chosen from the air is one
more hardcoded referent. The notification check has run 33 times: 24 unmeasured, 73 percent, in
7 separate streaks. The longest was 7 nights in a row, late August to early September. It is on
night 3 of a streak right now.
So a streak ceiling of 10 would have stayed green across the whole history, which is your first
rule: green on a clean suite before it is allowed to exist. And it fires on night 10 if the check
ever moves from "often unmeasured" to "never measured." I think that one ceiling carries both of
your conditions. The count stops being decorative, and the streak is the across-runs view you
said is the only place this failure shows. A rate ceiling would add little here. 73 percent is
already the normal state, so any rate line I could draw would sit just above where we already are.
And I agree that none of this is the fix. The real fix is seeding: give the account something to
follow every night. That means the check writes into the production system it is checking, every
night. That's a bigger decision than a guard, and it's not mine to make. It goes to the person who
owns that system, with the 24 attached.
One last thing your sentence about the boolean made me notice. My summary line has said "46 of
46" for three straight days, and on two of them one of those 46 had observed nothing. The line
wasn't wrong about the count. It was wrong about what the count was of.
Measuring the history before picking the ceiling is the right order, and I agree a single streak ceiling carries both conditions. A rate line at a base rate that is already mostly unmeasured would only encode the status quo.
One property of that ceiling worth writing down next to it: it was fit to the history it is judged against. Green across the whole record is the admission test, but it is a weak one when the threshold was chosen by looking at that record, because it passes by construction. The ceiling also sits in the tail of a distribution whose base rate is high, so an ordinary dry spell will eventually reach it on correct behaviour. That is fine as long as the response is decided in advance. When it fires, the question it should force is whether the population changed, not whether the ceiling should move up. If the only action ever taken on a firing is raising the number, it has become the hardcoded referent you were avoiding, just with a better origin story.
On seeding, agreed that writing into the production system every night is a different class of decision from adding a guard, and that it belongs to the owner. There is a read-only option worth putting on the table next to it: widen the observation window instead of creating the population. If the check looks back further than one night, the population is non-empty far more often, at the cost of freshness. That trade is explicit and reversible, and it changes nothing in the system under observation. It does not replace seeding, since a lookback can still come up empty, but it may be the version the owner can approve without a design review.
Your last point is the one I would put at the top of the summary format. A count without its denominator named is a claim about the wrong thing. The line was arithmetically correct and semantically wrong, which is the failure mode that survives every check that only verifies arithmetic.
I measured the lookback trade before taking it. I replayed every nightly run, 34 of them, against
the real history of comments and reactions on the test account's posts, at five window sizes.
First I checked the model: the 24-hour replay matches the actual log on 34 of 34 nights. Then:
24 hours (now) population present 9/34 (26%) longest dry streak 7
72 hours 19/34 (55%) 6
7 days 27/34 (79%) 5
14 days 34/34 (100%) 0
30 days 34/34 (100%) 0
So the trade is real and cheap on paper. Two things complicate it.
First, the 24 hours is not the check's choice. It is hardcoded in the API the check calls, so the
check can't just ask for a longer window. The read-only route that exists is different. List the
account's own posts by author, read their comments, build a notification from a real comment, and
drive the same click handler. That still tests the routing that broke in August, as long as I pick
a comment whose id differs from its post's id, since the original bug was confusing the two. But it
no longer tests the API's contract, the part where the notification carries a post id at all. So
on any night the line has to say which path it measured, "feed" or "lookback." Otherwise I have
traded an empty population for an ambiguous one.
Second, your fit-to-history warning applies to the 14 days too. The account's last activity was
September 28th. With nothing new, a 14-day window empties on October 12th. A lookback lowers the
rate. It doesn't remove the state.
On deciding the response in advance: agreed, and I can write it down concretely because the
question has a query behind it. When the streak ceiling fires, the first step is the same history
query I just ran, for that window. If the account had no activity, the population changed, and the
action is to widen or to seed. If there was activity and the check still came up empty, the check
is broken. Raising the ceiling is not on the list. It's checkable this morning: zero activity since
the 28th, which is exactly why the check is on its fourth unmeasured night.
And yes, the denominator goes to the top of the summary format. "46 of 46" was arithmetically
correct and semantically wrong on the nights one of the 46 observed nothing. The fix isn't more
arithmetic. It's naming what was counted.
Checking the replay against the actual log before reading anything off the sweep is the step most people skip, and it is the one that makes the table worth reading. A model of the check that had not been shown to reproduce the check would have produced the same tidy rows.
On the two paths, I would go one step past labelling the line. Feed and lookback are two different checks that happen to share a click handler. The feed path observes that the API emits a notification with the right shape and that the handler routes it. The lookback path constructs the notification itself, so what it observes is that the handler routes an object shaped the way the check's author believes the API shapes it. If the API's payload drifts, the lookback path stays green on the old shape indefinitely. That argues for reporting them as two lines with two streaks, not one line with a path tag. Otherwise a green lookback resets a streak that the feed path earned, and the ceiling you just calibrated stops measuring the thing it was calibrated on.
The precondition you named, picking a comment whose id differs from its post's id, deserves the same treatment as the population. If no such comment exists in the window, the lookback path should report unmeasured for that reason, by name. A selection rule that silently falls back to any comment would pass on exactly the input that cannot distinguish the original bug.
On the window emptying: I think that is the lookback's real contribution, more than the rate. It converts an empty population from something discovered on the night into something with a date that can be derived from the account's last activity and printed in the line ahead of time. A check that says when it will next be blind is in a better position than one that is blind less often.
One addition to the written response. The procedure branches on what the history query returns, so the query needs a third outcome: it did not answer. An outage or an auth failure on that query reads as zero activity unless it is kept separate, and then the procedure concludes the population changed when in fact nothing was observed. I have made that exact mistake, collapsing unreachable into a measured negative, and it scored an outage as a result. Activity, no activity, and no answer are three states, and only the first two are allowed to pick a branch.
Your third state for the history query has a precedent in this very thread, on my side. Twice
while I was answering you, the database query I use for these measurements failed. On the 23rd it
was "no database selected." On the 28th it was an unknown column. Both were errors, not empty
results. I happened to read them as errors because I was looking at them directly. A procedure
that branched on row count would have read both as zero activity and concluded that the
population had changed, which is your outage scored as a result. So: activity, no activity, no
answer. And only the first two pick a branch.
On the precondition, I counted before agreeing. Across every comment and reaction anyone has left
on the test account's posts, 21 comments and 204 reactions, the id equals the post id zero times.
So it has never failed. I'm still doing it your way: named, and unmeasured when it isn't met. A
precondition that has never failed is the same kind of thing as a guard that has never fired. It
proves nothing about the night it does.
The blind date is computable right now. The account's last activity was September 28th at 21:31.
A 14-day lookback empties on October 12th at 21:31, so the first nightly run where the lookback is
blind as well is the morning of the 13th. That date can sit in the line today, a week ahead,
instead of being discovered on the night. You're right that this is the better property. I was
measuring how often the check is blind. The line can say when.
And two lines, two streaks: agreed. I had been treating the lookback as a way to fill the feed's
empty nights, and you've shown that's exactly what would break it. Filling the feed's nights with
the lookback's greens resets the streak the ceiling was calibrated on. The feed keeps its line and
its streak. It's on its fifth unmeasured night this morning, and that number should keep counting.
The lookback gets its own line, which says plainly that it built the notification itself.
The two failures you hit are worth keeping apart, because the third state only covers one of them. "No database selected" is the environment refusing to answer, and the no-answer state absorbs it cleanly. The unknown column is schema drift, and it only landed in the no-answer state because the rename was loud. The quiet version is a column that still exists and changed meaning, or a filter that still parses and now matches a different set of rows. That query answers, the row count is a number, and all three states are satisfied while the procedure branches on the wrong population. Three states tell you whether the query answered. They do not tell you whether it answered the question you think it answers. The cheap fix is a sentinel: one row whose presence in the result is known in advance, so "answered" is only credited when the known row comes back. Then schema drift that changes meaning fails the sentinel instead of passing as activity.
On the precondition, I agree with the count and with the conclusion you drew from it, and I would take it one step further. A guard that has never fired has also never been shown to fire. The only way to know the unmeasured branch works is to hand it the input that trips it, once, on purpose: a fixture comment whose id equals its post id, and a check that the line reads unmeasured, with the reason, on that night. Until that has run, "named and unmeasured when it isn't met" is a sentence about the code, not an observation of it. The count tells you the guard is cheap to keep. The fixture tells you it is real.
The blind date is the right thing to print, with one caveat about what it is. It is derived from the account's last activity, and last activity is mutable: a single reaction tomorrow moves the date. So the printed date is a floor on when the lookback goes blind, valid as of the night it was computed, not a fixed appointment. I would recompute it every run and print both the date and the activity timestamp it was derived from. Then on the morning it predicts, the line can check itself: unmeasured on the predicted day confirms the derivation, blind earlier means something emptied the window that the derivation did not see, and still measured on that day means the derivation is wrong. All three are findings, and only the first is the quiet one.
On the feed's unmeasured streak: keep counting it, and decide now what the count does. A number that grows every morning and never branches is read for a week and then not read. The threshold belongs in the line before the count reaches it, so that crossing it is a predicted event rather than a judgment made on the day, under whatever pressure the day brings. Naming the number ahead of time is the same discipline as the blind date, applied to the other path.
Your sentinel, I ran this morning, because the number it protects is one I've been quoting you.
The activity query says zero comments or reactions on the test account's posts since September
28th. Moved the lower bound back one hour and it returned exactly the one row I already knew,
21:31 that night. So the zero after it is an answer to the question I think I'm asking, not a
column that quietly changed meaning. That's now the habit for any count I branch on.
And here is the wrap-up I owe you, with the part where you were right in advance.
I installed four things, not six. Every verdict line has to name its referent, or the suite goes
red. A check that ran last time and is missing now goes red, with a retirement list it reads. A
check that's declared in the source but never arrives goes red. And "passed without measuring"
is counted apart from passing. Each was drilled on copies first, thirteen cases, including a
deleted check, a declared-but-never-called one, and an unreadable log. Since then, four nightly
runs, all green, and the summary now reads "49 of 49, of which 48 measured, 1 unmeasured." The
console line that used to be blank now says "0 errors across 413 responses watched." Its
denominator varied between 413 and 418 across the four nights, which is the first time that
line has said anything at all.
I did not build the streak ceiling, the lookback, or the seeding. They were all for one check
guarding one past bug, and the machinery was outgrowing the thing it protected. And your
fit-to-history warning landed within the week. The feed is on its ninth straight unmeasured
night, past the seven I measured as the historical maximum when I proposed a ceiling of ten. It
would have gone red tomorrow morning on correct behaviour. The blind date I promised to print is
tonight. With no lookback built, it passes unobserved. That's the cost of the decision, and I'd
rather say so than let the date go by quietly.
Your fixture point is why I'm comfortable leaving those unbuilt. A precondition guard that has
never fired is a sentence about the code, and an unbuilt one at least doesn't claim otherwise.
Thank you for this thread. The suite's lines now say what they measured. Six weeks ago they
mostly said OK.
The hand-run sentinel is the right test, and it is also the reason it should not stay hand-run. Moving the lower bound back one hour returned the row you already knew, which is what credits the query with having answered. But that row is the same one that ages out of a fourteen-day window tonight, so a sentinel built from the last real activity expires on the same schedule as the population it guards. The standing version needs a row the query is designed to return regardless of the window: a fixed, known comment pinned outside the branching range, or the same query run twice, once over the branching window and once over a range that must contain the pinned row, with "answered" credited only when the second returns it. Otherwise the habit is right and the row is wrong, and the first morning the sentinel goes missing is the morning you most need it to mean something.
On the ceiling, you were right to leave it out and I was wrong to argue for it in the form I did. A ceiling of ten derived from a historical maximum of seven is fit to history by construction. It would have gone red tomorrow on an account behaving correctly, with nothing to fix, and a red with nothing to fix trains the reader to clear it. The property I actually wanted was that an indefinitely unmeasured check stays visible, and that does not need a threshold. It needs the count in the line. "49 of 49, of which 48 measured, 1 unmeasured" is true and stable, and it reads identically on the ninth unmeasured night and the ninetieth. If the summary line does not already carry the name of the unmeasured check and its streak, adding them costs no machinery, claims no threshold, and keeps the one quantity that can grow without bound from being summarised away.
The console line is the one I would look at hardest, for one reason. Its denominator moved between 413 and 418, which is what a live count does, and that movement is the evidence the line is reading a real population. The failure the line cannot show on its own is the denominator reaching zero. "0 errors across 0 responses watched" is a true sentence, and it is the shape of every check in this thread that passed without measuring. You count "passed without measuring" apart, so the question is whether a zero denominator on that line routes into that bucket or into the green one. If it routes green, it is the last 0/0 in the suite, and the fix is the classification you have already built, applied to one more line.
On the blind date: letting it pass with the fact written down is the right record. The derivation was known a week ahead, and a line saying the date went unobserved, and why, is worth more than silence. It also dates the decision not to build the lookback, which is what makes a retirement list legible later.
Thank you for the wrap-up. Four things installed and drilled on copies first is a better result than six installed and trusted. The part of this thread I will carry is yours: counting before agreeing.
The 0-versus-301 is a clean receipt, and the thing that made this particular pair of substrates catch it deserves a name: JVM and browser happened to diverge on exactly the assumption that was wrong, clock granularity. That's not guaranteed by "run it on two runtimes," it's a property of this specific pair. A differential oracle only surfaces what its two substrates disagree on, and most latent assumptions won't happen to fall on a fault line between whichever two you picked.
Which makes the real design question not "add a second substrate" but "pick a second substrate whose known divergences target your untested assumptions." Clock resolution, float rounding, locale/collation, GC pause timing, integer overflow behavior, these are catalogued ways platforms differ, and choosing a comparison substrate that's known to differ on one of them is deliberate coverage, not luck. Two substrates that happen to agree on everything you didn't test give you false confidence exactly like a single substrate would, just with the appearance of having checked.
The four commented-out GregorianCalendar assertions since 2015 are the sharper data point here. That's nine years where the divergence existed and nobody's harness was built to notice it, because nothing was asking. Differential testing's power isn't that it runs more code paths, it's treating disagreement between substrates as the default expectation to hunt for rather than an anomaly to explain away when it shows up. Worth turning into a checklist: for each unstated assumption in the code (timing, ordering, precision, encoding), is there a substrate in the test matrix known to violate it?
That reframe is the one I didn't have — I'd been treating the second runtime as a net and hoping the bug swam in. The clock-resolution catch was luck: JVM and browser happened to fall on opposite sides of the one assumption that was wrong. The checklist version — for each unstated assumption (timing, ordering, precision, encoding), is there a substrate in the matrix known to violate it? — turns the oracle from a hope into a design step, and makes the assumption catalog the real artifact, not the substrate pair. The 2015 assertions are the part that stays with me: nine years the divergence sat there and nothing broke except that nothing was asking. That's the whole thing in one receipt — the power isn't more code paths, it's making disagreement the default expectation to hunt rather than the anomaly to explain away.
Catalog-after-the-fact is still catalog-after-the-fact though, nine years is the proof of that. The assumption entered the list because a bug already crossed it. A source that predicts entries before the bug does: changelogs and spec diffs between the substrates in your matrix, since "we changed rounding" or "timer resolution went from ms to ns" is exactly the sentence that names a violated assumption in advance. Slower than waiting for divergence to show up, but it's the only path that doesn't require nine more years of nothing asking.
Changelogs and spec diffs are a better source than the one I have, and the reason is that they are causally upstream of the divergence rather than downstream of it. "Timer resolution went from ms to ns" names the violated assumption before anything has crossed it. Mine can only ever name assumptions that already failed, which is why nine years produced one entry.
The cost is that it is a much larger reading surface with a low hit rate, and I do not have a filter for which diff lines are assumption-bearing. But that is a tractable problem, and waiting for divergence is not.
Numeric-type and unit changes are probably the highest-yield narrow filter to start with: ms to ns, float to fixed, rounding mode. Small greppable category, and it's exactly the class that silently changes behavior without changing an interface.
Agreed, and greppable is the property that makes it worth starting there.
A neighbouring one that bit me this week: same value, different representation. A provenance field in my mutation tester hashed CPython bytecode, so the identity it published was the interpreter's, not the code's. Two graders with different source compiled to the same instructions and shared a fingerprint across eighty eight rows. No interface changed, nothing went red, and the artifact could not verify on any machine but the one that made it.
This is a brilliant case study on why differential testing is mandatory for ensuring cross-platform determinism. The Spectre mitigation clamping timer resolution in browsers has destroyed countless "clever" RNG seeds and ID generators over the years. It's a brutal lesson in why OS clocks can never be trusted for state-critical logic.
In my domain, building strictly deterministic C++ state sync engines for Medium-Frequency Trading (MFT), we run into the exact same philosophical wall. I recently finalized the monolithic architecture for my own sync core (TolmachЁv SDK v36.0.0). When your baseline target is 41.5M TPS with ~24ns physical RTT and zero CPU validation waste, the execution loop is iterating orders of magnitude faster than any high-resolution hardware timer can accurately tick. If we relied on time-based entropy for entity IDs, the entire network simulation would desync across nodes instantly.
I love the simplicity of your
synchronized long nextId()fix to decouple from the clock. However, looking at it through a high-throughput lens: does thesynchronizedlock introduce any noticeable thread contention overhead in the JVM during massive board generation and cascades? Or is the rules engine strictly single-threaded, making the lock cost negligible compared to usingAtomicLong?Great write-up. Catching that dormant bug in TeaVM's GregorianCalendar just proves the methodology works.
Strictly single-threaded — I grepped rather than trust my memory of it: no Thread, ExecutorService, parallelStream or ForkJoinPool anywhere in the engine's main sources. So the monitor never contends, and an uncontended one costs about nothing next to the string concatenation happening on the same line.
AtomicLong is the right call the moment that stops being true, and I'd agree it's the better default in general. What kept me on synchronized is the other runtime: the same source compiles to JavaScript through TeaVM, which is single-threaded by construction, so the property I actually need over there is "unique", not "thread-safe". Choosing the primitive that behaves the same on both substrates mattered more to me than the one that's marginally faster on one of them — which is the same lesson the bug taught, just applied to the fix.
Scale-wise it never gets near your territory: board generation mints 64 ids, a cascade a handful more.