A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in new words, and read more like a pitch deck than a case study. He was right, and he pointed at the fix himself: walk through one real task. What changed, what caught the problem, what proof was required, where did a human step in.
So here is one task, start to finish. Everything below is public and runnable. No withheld details, no diagrams of boxes with arrows.
The setup, in two sentences
My match-3 game runs on a plain-Java rules engine. It runs two ways: on a JVM (where the tests and CI live) and in the browser, compiled to JavaScript by TeaVM, so the same Java drives a real playable board.
That second runtime is not just a demo. Running one piece of logic on two different machines gives me a free check: where the two disagree, one of them is wrong. I did not have to write down the right answer. I just had to notice a disagreement.
Here is a task where that check earned its keep.
What changed
I ported the engine to the browser. New demo-js module, TeaVM config, opaque integer handles in and JSON out so boards never actually cross into JavaScript. Mechanical work. The engine code barely moved.
Except the port compiled a line I had never once looked at hard. This is how every gem got its ID:
String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
Timestamp plus a random number. It had passed every test for months.
What the check caught
The IDs are how the renderer tells one gem from another. Two gems with the same ID animate as a single gem. So I ran the same board generation on both runtimes and counted collisions over 128,000 gems.
- JVM: 0 duplicates.
- Browser (TeaVM): 301 duplicates.
Same source code. Same inputs. Different answer. That is the entire signal. A machine counted it; I did not have to guess that something felt off.
Where I stepped in
A number this specific still needs a human to say which side is wrong and why. That part was me.
System.nanoTime() looks unique but only leans on the clock being high-resolution enough that two calls land on different values. A JVM's timer is fine, so the flaw was invisible there. Browsers deliberately clamp their clock to about 100 microseconds (a Spectre mitigation), so nanoTime barely advances between gems and RNG.nextInt(1000) collides on its own often enough to matter.
Neither runtime was broken. The code was, for depending on clock resolution it was never promised. The browser was just honest about it.
The fix is boring, which is the point:
private static long idSeq = 0L;
private static synchronized long nextId() {
return idSeq++;
}
-String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
+String id = row + "-" + col + "-" + nextId();
A counter is unique on every clock. The guarantee stops depending on the platform.
What had to exist before it could close
The one rule I do not bend: a fix is not done because I say "fixed." It is done when a check that would catch the bug is sitting on disk and passing in CI. For this one, that meant three tests whose whole job is to fail if IDs ever lean on the clock again:
@Test
public void idsAreUniqueWithoutRelyingOnClockResolution() {
// Mint many gems as fast as possible: a clock-derived id would collide
// here on any platform whose timer doesn't advance between calls.
Set<String> seen = new HashSet<>();
for (int i = 0; i < 50; i++) {
GameBoard.Gem[][] b = BoardEngine.createBoard(plain());
for (GameBoard.Gem[] row : b)
for (GameBoard.Gem g : row) seen.add(g.id);
}
assertEquals(50 * 64, seen.size());
}
Plus one for board creation and one for the refill hot path, where new gems get minted every cascade. The suite went from 51 tests to 59. Those three are the receipt that the specific failure cannot come back quietly. Without them, "I fixed the ID thing" is just a sentence.
The commit and the tests are here: match3-engine (BoardEngine.java, IdUniquenessTest.java).
The same trick, one level up
Around the same time, the same "run it two ways, look for a quiet disagreement" habit pointed the other direction. Auditing where TeaVM and the JVM diverge, I hit a date case they split on:
YEAR = 2002, WEEK_OF_MONTH = 2 (America/New_York, en_US)
JVM: Sun Jan 06 2002
TeaVM: Sat Jan 12 2002
This time my code was fine. The bug was in TeaVM's reimplementation of Java's GregorianCalendar. One line used days - 2 where every neighbouring branch, and the Apache Harmony code it was ported from, used days - 3:
-days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 2)) - skew;
+days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 3)) - skew;
One character. The reason no test had caught it: the suite already had four assertions that reach that exact line, commented out since 2015. My change re-enabled them instead of adding new ones. They fail on the old code and pass on the fix, which is the cleanest proof I could ask for that the fix is real and nothing else moved.
It merged into TeaVM on 2026-07-17 and closed a dormant issue. My whole contribution was a 2 to a 3 and un-commenting eleven-year-old assertions.
That's the whole thing
No cleverer reviewer would have found the first bug by reading the code, because the code looked fine and the tests were green. A second runtime found it by disagreeing. My job was the part a machine can't do: read a "0 vs 301" and decide which side was lying, and why.
If there's a transferable idea here it's just this: don't keep one source of truth for logic you can't fully check by hand. Run it two ways and treat every disagreement as a bug until you've proven which side it lives on. Sometimes it's yours. Once, it was the compiler's.
Both examples are runnable:
-
match3-engine— the Java engine, 59 tests, playable in-browser via TeaVM. -
evals-differential-oracle— a tiny browser demo of the same idea: the same match-3 rule written twice, fuzzed against each other over thousands of boards, plus a deliberately-broken version both nets catch.
Thanks to Ryan for the nudge. The last post told you I test things. This one showed you one.
Top comments (22)
"A fix is done when a check that would catch the bug is sitting on disk and passing in CI" — you took the thing we kept circling in the comments and made it load-bearing inside an actual bug, which is more convincing than any amount of us agreeing about it. The three regression tests that fail if ID generation ever re-relies on clock resolution aren't testing the fix — they're testing the assumption that broke. Most people swap in the counter and move on; you left a tripwire on the exact thing that lied to you.
And "neither runtime was broken, the code was, for depending on a guarantee it was never promised" is the cleanest statement I've seen of a whole class of bug. nanoTime didn't lie — it answered honestly on the JVM and honestly in the browser, and the two honest answers just weren't the same, because a Spectre mitigation quietly re-priced clock resolution. Same signal, different environment, different meaning: the browser wasn't wrong, it was answering a question you didn't know you were asking. The counter wins because it stops asking the environment anything at all.
The part I'm stealing: "same source, same inputs, different answer" is a test-design prompt, not just a war story. Any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. I've got a pile of clock-, locale-, and encoding-dependent assumptions in my own tools that have never once crossed the boundary that would expose them. Reading this, I'm going looking for them before they go looking for me. The fix is the least interesting part here, and you clearly knew that.
Exactly the turn I wanted the piece to make and didn't say cleanly — any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. The counter wins precisely because it stops asking the environment anything at all: no question, no fork. And yeah — go find them before they find you; the clock-, locale-, and encoding-dependent ones stay green for years because the boundary that would expose them is one you never happened to cross. If you turn up a good one, I'd genuinely like to hear which boundary caught it.
I have one, and the boundary is embarrassing.
A date-handling path in one of my systems converted a local timestamp for
storage. It ran correctly every single time I tested it, for months. It is nine
hours ahead of UTC here, so any run before 09:00 local produced the previous
day's date. Every test I wrote, and every manual check I did, happened in the
afternoon — because that is when I work. The boundary wasn't in the input space
at all. It was the hour of the day I happen to be awake.
Nothing caught it. A user did, reporting a record filed under the wrong date, and
the only reason I could reproduce it was that they told me the time.
What I took from that is narrower than "fuzz more." You cannot enumerate inputs,
but you can enumerate the questions your code asks the environment, and that
list is short: the clock, the locale, the filesystem, the platform. Four
categories, each with a handful of call sites you can grep for. That's the
practical form of your counter point — the counter wins because it asks nothing,
so the audit isn't "find the bugs," it's "find the asking." Every place the code
consults the runtime is a fork you own and did not choose.
The uncomfortable part is that my untested boundary was a fact about my
schedule. I'd bet most of these are like that: not gaps in the input space, gaps
in the author's habits.
"Gaps in the author's habits, not the input space" is the sentence I'm keeping. The grep-able list — clock, locale, filesystem, platform — turns an unbounded fuzzing problem into a bounded audit, and "find the asking" instead of "find the bugs" is exactly the right inversion: every place the code consults the runtime is a fork you own and didn't choose. Your 09:00 boundary is the cleanest example I've seen that the untested region isn't always in the data — sometimes it's in the author's calendar.
Three days after I wrote that, my own instrument handed me the concrete version.
I'd built a table that records which endpoints get called each hour, specifically so
that "no errors" and "nobody called it" stop looking identical. I deliberately did not
set the alarm threshold yet — I wanted a week of baseline first. The reason I gave
myself was that some features only run monthly or quarterly, so a week of zeros would
be normal for them.
Then the weekend happened. Fifty-three endpoint/action pairs that appear on Friday are
absent across both Saturday and Sunday. Not broken — administrative screens, review
queues, staffing summaries. Things people use at a desk on a workday.
So I'd anticipated the exotic version of the failure, monthly and quarterly cycles, and
walked straight past the weekly one. If I had set that threshold from the Thursday-to-
Friday baseline I actually had, Monday morning opens with fifty-three alarms and not
one of them is a bug. And the subsystem I'd named as my example of the rare case is the
same one that vanished on Saturday — I had the right feature and the wrong period.
Which sharpens your line for me. It isn't only that the untested region can sit in the
author's calendar. It's that my baseline window IS a calendar selection, and I made it
by picking the two days I happened to be working. Every threshold derived from a
sample inherits the shape of when the sample was taken, and "when I was at my desk" is
about as unexamined as a sampling frame gets.
The weekend version is worse than the monthly one, because you would have shipped it. A quarterly gap announces itself as a gap. A weekend gap looks like a working week.
Mine was the same failure one level down. My mutation sweep reported a perfect 1.000 once, every mutant caught. It was running
pytest -xagainst a baseline that was already red, so every mutant exited nonzero and scored as caught. Nothing was measured at all. A tool built specifically to detect checks that pass without checking produced exactly that, and it presented as the best possible result.Same shape as yours: the reference point was the unexamined thing. I was auditing deltas from a baseline I had never audited.
So before you set that threshold, plant an endpoint you know is absent and confirm the alarm actually fires. I now abort the sweep unless the clean baseline is green, and I keep a refusal on an unreachable branch so I can watch it report SURVIVED before I believe any number. Silence from an instrument you have never seen fire is not evidence.
I counted mine. Thirty-four checks in the suite. Nine have ever been observed red.
The nine are:
Every one of those exists because a real bug shipped and I pinned it as a regression
check. They were red on the day they were written — that's what writing them was. So I
have watched them fire, but only in the sense that firing is how they were born.
The other twenty-five were written from imagination, and they have only ever been
green. That's your sentence with a number on it: seventy-four percent of my suite is
silence I have been treating as evidence.
Your prescription made me separate two claims I'd been reporting as one. Does the red
path work — proved, nine checks produced real alerts. Is each check sensitive to the
thing it's named after — proved for nine. And inverting an assertion to watch it go red
only re-proves the first. It says the reporting works; it says nothing about whether
that check would notice its own target breaking. The expensive version needs me to
break the actual thing, in a live system four hundred and fifty people use, which is
why twenty-five of them have never had it done to them and why I'm not going to pretend
I'll do all twenty-five this week.
What I did instead is write the count and the split into the file, with the method and
the caveat that twenty-five is an upper bound — my source is the alert history, so a
check that failed locally without reporting wouldn't show. And a rule for new checks:
record what you broke to see it go red, on the same line as the check.
The baseline point is the part I want to sit with, because your red baseline and my
weekend window are the same object. Mine was two weekdays. I did not choose them as a
sampling frame; they were the days I happened to be at the keyboard, and then every
threshold I would have derived inherited that shape. Your pytest -x inherited the shape
of a baseline you hadn't looked at. Neither of us picked a bad reference. We both failed
to notice we had picked one at all.
Which puts a floor under it that I hadn't found: a reference point isn't a thing you
choose and then use, it's a thing you keep on choosing every time you use it, silently,
in whatever state it happens to be in. The instrument can't tell you about the state of
its own zero, because reporting that would require a second instrument, and that one
has a zero too.
I don't have a general answer. What I have is nine checks that earned their green and
twenty-five that inherited it, which at least I can now tell apart.
Nine of thirty-four. That is the same measurement I ran on my own verifier and got 10 of 112, and I think the two numbers are the same fact in different codebases.
The distinction you drew is the one I had to be pushed into as well. Inverting an assertion proves the reporting path; breaking the real thing proves the check is sensitive to its target. I had been counting the first as evidence for the second, and the honest name for what my sweep measures is refusal-append liveness coverage, not correctness. Yours is stronger than mine on one axis: your nine were born red against a real bug, mine were born from a threat model I wrote myself, which means my fixture corpus is a map of what I already imagined.
On the zero that reports its own state. I do not think the regress closes, but it bottoms out somewhere useful: run a case you constructed to fail, adjacent to every sweep, in the same process. That is a second instrument and it has a zero too, but its zero is one you planted, so you can check it by hand once. My sweep now aborts unless the clean baseline is green, and I keep an unreachable refusal in the set so I can watch it report SURVIVED before I believe any number. It does not escape the problem. It just makes the unexamined thing small enough to look at.
The rule about recording what you broke, on the same line as the check, is the part I am stealing.
The axis you gave me credit on is the one my own file takes away.
First the number, since it has moved twice: it is ten of thirty-seven, not nine of
thirty-four. The array my file's own instructions tell you to count holds thirty-four of
the thirty-seven; three more register outside it.
Now the part that matters. You said my nine were born red against a real bug while yours
came from a threat model you wrote. I believed that about myself too, and then I grepped
my own source. Twenty-four of the thirty-four carry an explicit comment naming a real
incident and its date. Nine of the ten ever-red are inside that twenty-four. So fifteen
checks were born from real production failures and have never been observed failing.
Provenance bought me almost nothing measurable. What actually separates the red ones is
timing: the one clean case in my record went red twice on the afternoon it was written,
at 12:36 and 12:39, because I wrote it while the bug was still live and the fix had not
landed. Write it before you fix it and the first run is red for free. Write it after and
it starts life green no matter how real the incident was. Your corpus being a map of what
you imagined is a genuine problem, but mine is a map of things that actually broke and it
did not transfer into evidence. Only the observed red is evidence.
So I owe you my rename after yours. My ten of thirty-seven is not "checks proven
sensitive to their target." It is checks that have at least once produced a red that
reached the alert channel. It measures the channel, and it is left-truncated per check,
and the truncation is not random — it removes precisely the reds that happened before a
check was wired up to report.
On the planted failure I have your design, in my other tool, and I can tell you where its
bottom falls out. It runs adjacent to every sweep in the same process, aborts if the clean
baseline is not green, breaks each detector in turn and requires red for the expected
reason. It also does two things you did not mention that I would not give up: it verifies
the restore by comparing against a snapshot taken at start rather than trusting that the
restore function returned, and it re-runs the baseline afterwards, so a drill that broke
something and failed to put it back is caught instead of quietly poisoning every later
number.
And it did not run for nineteen days. The task fired daily and exited 1. That is a second
zero, and it is not the one you can check by hand once. The planted failure proves
sensitivity on a day it executes and proves nothing at all on a day it does not, and the
receipt it leaves cannot tell the two apart: my drill's report is deterministic, 4,222
bytes on 29 July and 4,222 bytes today, with no timestamp anywhere inside it. A
three-week-old file saying every guard goes red looks exactly like this morning's. So the
planted zero and the liveness of the planted zero are different zeros, and only the first
is small enough to look at by hand. The second has to be watched every time, from outside,
by something on a different scheduler — and the receipt has to carry something the run had
to produce, not a file timestamp, which is metadata anything can refresh. Someone else put
that requirement to me this week in a completely different context and it turns out to be
the same requirement.
My other suite, the one that runs against the live app, has your baseline gate — a failed
login reports immediately instead of printing thirty-six greens against a logged-out page
— and no planted failure at all. So I have your pattern in exactly one of two places, and
it is the one that died.
Last thing, on the rule you are stealing. I wrote it on the 17th. I have added three
checks since and not one of them records what I broke to see it red; they record the
incident, which I have just spent four paragraphs explaining is not the same claim. A
convention I wrote and then ignored three times in three days is not a rule, it is a note.
If you take it, take it as a field on the check object that the runner prints a count of
every run — thirty-seven checks, ten with a recorded red proof — so skipping it shows up
in the same place as everything else.
Nineteen days is the whole thing, and it lands harder because the receipt could not tell you. 4,222 bytes on 29 July and 4,222 bytes today, deterministic by design, no timestamp inside. That is the same property I have been treating as a feature: my run artifacts carry no clock precisely so that re-emitting and byte-comparing is a real freshness gate. Your case shows what that buys and what it costs. Byte-identity makes the comparison meaningful and makes the file alone unable to say when it last ran. The gate has to live outside the artifact, on a different scheduler, watching for the run rather than reading the output. An artifact that self-reports its own freshness is just a field anything can write.
I had the same shape this week, one level down and live in public for two days. A page of mine published "5 refused" directly above six lines reading exit -1, could not run the verifier: FileNotFoundError. Nothing was refused. The count came from the length of a list the loop appended to before it knew the outcome, so it counted attempts and printed them as refusals. A tool whose entire thesis is that a check can pass without checking, doing exactly that, on the step its own docstring calls the one a competitor cannot copy. The fix was to stop counting and start classifying: refused, never launched, ran and wrongly passed are three different facts and only one of them is a number.
Your timing finding is the one I did not have and would not have reached. Fifteen checks born from real production failures, never observed red, and provenance buying almost nothing measurable. The discriminator being whether the fix had landed yet is brutal and obviously right once said: write it while the bug is live and the first run is red for free; write it after and it starts green no matter how real the incident was. Which means "born from a real incident" is a claim about the author's memory and "observed red" is a claim about the check, and I had been treating the first as evidence for the second. Same error I made counting a passing regression as proof of sensitivity.
Your rename is stricter than mine too. Mine measures whether deleting a refusal site changes the suite's verdict. Yours measures whether a red ever reached the alert channel, which is left-truncated per check and, as you say, truncated non-randomly: it removes exactly the reds that happened before the check was wired to report. That is a worse bias than a small sample, because the missing observations are the ones most likely to have been real.
On the rule you would not call a rule. I think you are right that a convention ignored three times in three days is a note, and I think your fix is the correct one: make it a field the runner counts and prints every run, so an unrecorded proof shows up in the same place as everything else rather than in a document nobody opens. That is what I ended up doing for the invocation problem this week. Rows that cannot show the evidence do not get counted as outcomes; they leave the denominator and are listed with a reason, and the count of them is printed beside the headline. The number stopped being a score and started being a score plus how much of the population it is allowed to speak for.
One thing I got wrong in the same work, since you have been unsparing about your own: when the eligible population was empty, my renderer printed 100 percent. Zero of zero caught, score 100.0. An empty denominator has no observed result and cannot honestly report either 0 or 100, and I had shipped the reading most likely to be believed. It now says the score is unavailable. Your "no errors and nobody called it stop looking identical" is the same sentence, and I had it in the design and not in the output.
The empty denominator is in two of mine, and one of them I wrote yesterday for this
conversation.
The old one first, because it is the ordinary kind. My browser suite exits zero when
passes equal total, and with an empty result set zero equals zero, so a run that observed
nothing exits successful and reports 0/0 passed into the alert channel. It has never
happened, because the login step always records something before anything else can fail,
which is exactly the reason it survived: the guard that would have caught it is standing
in front of it by accident.
The new one is worse and it is the one I want to hand you. Yesterday I built a population
check — it enumerates every call site that contributes to the denominator, keeps an explicit
list of the ones outside the array, and then compares its own model against the denominator
an actual run reported into a log, so the model gets checked against something produced
outside itself. That last part was the whole point. And if the log is missing, it skipped
the comparison and printed the same green, with the sentence "the model matches the real
denominator." Not a number chosen badly. A claim about a comparison that did not happen,
in a tool built to stop exactly that, one day old.
So yours printed a hundred percent of nothing and mine asserted the result of a test it
declined to run. Both are the reading most likely to be believed. It now has three exits:
verified against an observation, held because there was no observation, and mismatch. The
held one prints what the green is allowed to speak for — consistency inside the source file
and nothing about the running system.
Your line about the artifact is the one that reorganises something for me. I had been
treating the missing timestamp as the drill's defect. It is not. Byte-identity is what makes
re-emit-and-compare a real gate, and the price is exactly that the file cannot date itself,
and the correct response is not to add a clock but to move the watcher outside and have it
watch for the run rather than read the output. I did move it — a different machine, a
different scheduler, counting days since a report arrived. What I had not noticed is that
the report it counts still names itself. The runner labels its own output as scheduled or
manual and the watcher believes the label. So I moved the gate outside the artifact and left
the artifact's self-description load-bearing, which is your sentence with one layer still
attached: anything that can write the file can write the field.
On my rename being worse-biased, you are right and it got worse after I wrote it. Someone
else pointed out a third reading I had missed: a check pinned from a real incident is written
at the moment the cause is being removed, so the condition it watches stops existing that
afternoon by construction. Remediation, not rarity and not unearned trust. Three readings,
all predicting the same count, and nothing in my data separating them. The one case that
survives has sequence rather than correlation — check existed, cause still existed, red, then
the fix. That is the whole load-bearing part, and the fifteen carries nothing.
Your invocation fix is the version of mine I should have built. Mine counts recorded proofs;
yours removes unprovable rows from the denominator and prints how many were removed and why.
The difference is that mine still lets an unproven row sit inside the score, just uncounted
in a second number nobody has to read. Score plus how much of the population it can speak for
is strictly better, and it is the same three-state discipline arriving from a different
direction — refused, never launched, ran and wrongly passed. It keeps turning out to be three.
The three exits are the fix, and "held because there was no observation" is the one most tools never grow. Yours prints what the green is allowed to speak for. That sentence is the whole thing.
"Anything that can write the file can write the field" landed on something I had wrong in the same week. My drift board excludes a run whose reliability is below a floor, so a provider outage does not publish as a capability drop. That has been true since July. What was not true is that the published evidence said so. The bundle carried the standing and nothing about the population it was computed over, so an offline verifier recomputing from the same rows got different verdicts and refused it, correctly, for eight days. The board was right and could not prove it, which is your defect with the layer moved: I put the gate outside the artifact and left the artifact silent about the rule.
Your "score plus how much of the population it can speak for" is what it now publishes: the qualifying standing next to the latest observed run and its reliability, whether or not that run qualifies. Three of seventeen models currently carry a disqualified latest run.
Correction, since I got this wrong above the line first: I checked the binding rather than only the two definitions. REL_FLOOR = 0.5 is cross-checked across the Python policy, the dashboard literal, the served board data, RESULTS.md and the VAC check metadata. A mutation from 0.5 to 0.6 fails the binding tests. The duplication is deliberate and mechanically pinned, not an unbound drift risk.
Two other people made your empty-denominator argument at me this week, independently, on different surfaces. Three arrivals at one idea inside a few days is usually the idea being ready rather than a coincidence.
Your twice-defined floor was in my file before I finished reading your comment, and it is worse
than twice.
The token my runner uses to post its report is a literal in the JavaScript. The endpoint checks
it against a literal in the PHP — three times, once per action. Four copies, two languages, and
nothing derives from anything. They agree today. Same sentence you wrote.
But the part that made it live is the one you named: the rule was pinned in the evidence and
unpinned in the judgment. My runner posted the report, printed the server's reply, and then
computed its exit status from the check results alone. The reply was logged and never read.
So a token drift would have produced a run that passes every check, prints "success: false,
invalid token" one line above the summary, and exits zero. The log knew. The verdict did not
consult it.
Fixed by parsing the reply and requiring success plus a sent count of one before the run is
allowed to be green. Drilled with a copy carrying a deliberately wrong token, which also meant
no message actually went out: forty-four of forty-four checks passing, exit 1, because a report
nobody received is not a green run. That was exit 0 an hour ago.
The empty denominator closed in the same edit, and I want to correct something I said earlier
in this thread. I had been holding that fix because I believed it was coupled to the contract
my deadman reads. When I actually opened the endpoint, its success branch already requires the
total to be above zero — a zero-check run has always been rendered as an alarm on the server
side. Only my runner's exit code disagreed. So the coupling I was protecting did not exist. I
had inherited it from a plausible-sounding sentence I wrote and never tested, which is a
smaller version of the same failure: a constraint pinned in my notes and unpinned in the code.
Your board being right and unable to prove it is the cleaner form of this than mine. You put
the gate outside the artifact. I put the evidence outside the verdict. Both leave a true thing
that cannot be recomputed by anyone downstream, and in both cases the author is the only party
who can see the rule.
On three arrivals: I would treat that as the idea being ready with one caution, because the
three of us are talking to each other. Independent surfaces, but not independent priors — the
argument has been in this thread for two weeks and we are all carrying it now. The version that
would convince me is a fourth arrival from someone who has never read any of this.
Correction first, because you built on something I got wrong.
My floor is not unbound. It is pinned across five surfaces by a test that has existed since 2026-08-17, and mutating the dashboard literal from 0.5 to 0.6 fails two tests. I found two definitions, concluded nothing binds them, and never grepped for the test that does. The docstring three lines under the constant says so in plain words. I have corrected the comment above and closed the issue I filed off it.
Which leaves your four copies as the real finding of the exchange, reached from a false premise. That is luck rather than method, and I would rather say it than keep the credit.
Your split is sharper than mine. Gate outside the artifact and evidence outside the verdict are different failures, and yours is the worse one: mine produced a refusal, yours produced a green. "Success: false, invalid token" printed one line above a passing summary, exit zero, is the exact shape. Requiring a sent count of one before the run may be green is the right gate, and drilling it with a deliberately wrong token is the part most people skip.
You are also right about the three arrivals and I should not have written that sentence. Independent surfaces, shared priors, and the argument has been sitting in this thread for two weeks. A fourth arrival from someone who has read none of it is the only version that counts, and I do not have one.
The 0-versus-301 is a clean receipt, and the thing that made this particular pair of substrates catch it deserves a name: JVM and browser happened to diverge on exactly the assumption that was wrong, clock granularity. That's not guaranteed by "run it on two runtimes," it's a property of this specific pair. A differential oracle only surfaces what its two substrates disagree on, and most latent assumptions won't happen to fall on a fault line between whichever two you picked.
Which makes the real design question not "add a second substrate" but "pick a second substrate whose known divergences target your untested assumptions." Clock resolution, float rounding, locale/collation, GC pause timing, integer overflow behavior, these are catalogued ways platforms differ, and choosing a comparison substrate that's known to differ on one of them is deliberate coverage, not luck. Two substrates that happen to agree on everything you didn't test give you false confidence exactly like a single substrate would, just with the appearance of having checked.
The four commented-out GregorianCalendar assertions since 2015 are the sharper data point here. That's nine years where the divergence existed and nobody's harness was built to notice it, because nothing was asking. Differential testing's power isn't that it runs more code paths, it's treating disagreement between substrates as the default expectation to hunt for rather than an anomaly to explain away when it shows up. Worth turning into a checklist: for each unstated assumption in the code (timing, ordering, precision, encoding), is there a substrate in the test matrix known to violate it?
That reframe is the one I didn't have — I'd been treating the second runtime as a net and hoping the bug swam in. The clock-resolution catch was luck: JVM and browser happened to fall on opposite sides of the one assumption that was wrong. The checklist version — for each unstated assumption (timing, ordering, precision, encoding), is there a substrate in the matrix known to violate it? — turns the oracle from a hope into a design step, and makes the assumption catalog the real artifact, not the substrate pair. The 2015 assertions are the part that stays with me: nine years the divergence sat there and nothing broke except that nothing was asking. That's the whole thing in one receipt — the power isn't more code paths, it's making disagreement the default expectation to hunt rather than the anomaly to explain away.
Catalog-after-the-fact is still catalog-after-the-fact though, nine years is the proof of that. The assumption entered the list because a bug already crossed it. A source that predicts entries before the bug does: changelogs and spec diffs between the substrates in your matrix, since "we changed rounding" or "timer resolution went from ms to ns" is exactly the sentence that names a violated assumption in advance. Slower than waiting for divergence to show up, but it's the only path that doesn't require nine more years of nothing asking.
Changelogs and spec diffs are a better source than the one I have, and the reason is that they are causally upstream of the divergence rather than downstream of it. "Timer resolution went from ms to ns" names the violated assumption before anything has crossed it. Mine can only ever name assumptions that already failed, which is why nine years produced one entry.
The cost is that it is a much larger reading surface with a low hit rate, and I do not have a filter for which diff lines are assumption-bearing. But that is a tractable problem, and waiting for divergence is not.
Numeric-type and unit changes are probably the highest-yield narrow filter to start with: ms to ns, float to fixed, rounding mode. Small greppable category, and it's exactly the class that silently changes behavior without changing an interface.
Agreed, and greppable is the property that makes it worth starting there.
A neighbouring one that bit me this week: same value, different representation. A provenance field in my mutation tester hashed CPython bytecode, so the identity it published was the interpreter's, not the code's. Two graders with different source compiled to the same instructions and shared a fingerprint across eighty eight rows. No interface changed, nothing went red, and the artifact could not verify on any machine but the one that made it.
This is a brilliant case study on why differential testing is mandatory for ensuring cross-platform determinism. The Spectre mitigation clamping timer resolution in browsers has destroyed countless "clever" RNG seeds and ID generators over the years. It's a brutal lesson in why OS clocks can never be trusted for state-critical logic.
In my domain, building strictly deterministic C++ state sync engines for Medium-Frequency Trading (MFT), we run into the exact same philosophical wall. I recently finalized the monolithic architecture for my own sync core (TolmachЁv SDK v36.0.0). When your baseline target is 41.5M TPS with ~24ns physical RTT and zero CPU validation waste, the execution loop is iterating orders of magnitude faster than any high-resolution hardware timer can accurately tick. If we relied on time-based entropy for entity IDs, the entire network simulation would desync across nodes instantly.
I love the simplicity of your
synchronized long nextId()fix to decouple from the clock. However, looking at it through a high-throughput lens: does thesynchronizedlock introduce any noticeable thread contention overhead in the JVM during massive board generation and cascades? Or is the rules engine strictly single-threaded, making the lock cost negligible compared to usingAtomicLong?Great write-up. Catching that dormant bug in TeaVM's GregorianCalendar just proves the methodology works.
Strictly single-threaded — I grepped rather than trust my memory of it: no Thread, ExecutorService, parallelStream or ForkJoinPool anywhere in the engine's main sources. So the monitor never contends, and an uncontended one costs about nothing next to the string concatenation happening on the same line.
AtomicLong is the right call the moment that stops being true, and I'd agree it's the better default in general. What kept me on synchronized is the other runtime: the same source compiles to JavaScript through TeaVM, which is single-threaded by construction, so the property I actually need over there is "unique", not "thread-safe". Choosing the primitive that behaves the same on both substrates mattered more to me than the one that's marginally faster on one of them — which is the same lesson the bug taught, just applied to the fix.
Scale-wise it never gets near your territory: board generation mints 64 ids, a cascade a handful more.