A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in new words, and read more like a pitch deck than a case study. He was right, and he pointed at the fix himself: walk through one real task. What changed, what caught the problem, what proof was required, where did a human step in.
So here is one task, start to finish. Everything below is public and runnable. No withheld details, no diagrams of boxes with arrows.
The setup, in two sentences
My match-3 game runs on a plain-Java rules engine. It runs two ways: on a JVM (where the tests and CI live) and in the browser, compiled to JavaScript by TeaVM, so the same Java drives a real playable board.
That second runtime is not just a demo. Running one piece of logic on two different machines gives me a free check: where the two disagree, one of them is wrong. I did not have to write down the right answer. I just had to notice a disagreement.
Here is a task where that check earned its keep.
What changed
I ported the engine to the browser. New demo-js module, TeaVM config, opaque integer handles in and JSON out so boards never actually cross into JavaScript. Mechanical work. The engine code barely moved.
Except the port compiled a line I had never once looked at hard. This is how every gem got its ID:
String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
Timestamp plus a random number. It had passed every test for months.
What the check caught
The IDs are how the renderer tells one gem from another. Two gems with the same ID animate as a single gem. So I ran the same board generation on both runtimes and counted collisions over 128,000 gems.
- JVM: 0 duplicates.
- Browser (TeaVM): 301 duplicates.
Same source code. Same inputs. Different answer. That is the entire signal. A machine counted it; I did not have to guess that something felt off.
Where I stepped in
A number this specific still needs a human to say which side is wrong and why. That part was me.
System.nanoTime() looks unique but only leans on the clock being high-resolution enough that two calls land on different values. A JVM's timer is fine, so the flaw was invisible there. Browsers deliberately clamp their clock to about 100 microseconds (a Spectre mitigation), so nanoTime barely advances between gems and RNG.nextInt(1000) collides on its own often enough to matter.
Neither runtime was broken. The code was, for depending on clock resolution it was never promised. The browser was just honest about it.
The fix is boring, which is the point:
private static long idSeq = 0L;
private static synchronized long nextId() {
return idSeq++;
}
-String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
+String id = row + "-" + col + "-" + nextId();
A counter is unique on every clock. The guarantee stops depending on the platform.
What had to exist before it could close
The one rule I do not bend: a fix is not done because I say "fixed." It is done when a check that would catch the bug is sitting on disk and passing in CI. For this one, that meant three tests whose whole job is to fail if IDs ever lean on the clock again:
@Test
public void idsAreUniqueWithoutRelyingOnClockResolution() {
// Mint many gems as fast as possible: a clock-derived id would collide
// here on any platform whose timer doesn't advance between calls.
Set<String> seen = new HashSet<>();
for (int i = 0; i < 50; i++) {
GameBoard.Gem[][] b = BoardEngine.createBoard(plain());
for (GameBoard.Gem[] row : b)
for (GameBoard.Gem g : row) seen.add(g.id);
}
assertEquals(50 * 64, seen.size());
}
Plus one for board creation and one for the refill hot path, where new gems get minted every cascade. The suite went from 51 tests to 59. Those three are the receipt that the specific failure cannot come back quietly. Without them, "I fixed the ID thing" is just a sentence.
The commit and the tests are here: match3-engine (BoardEngine.java, IdUniquenessTest.java).
The same trick, one level up
Around the same time, the same "run it two ways, look for a quiet disagreement" habit pointed the other direction. Auditing where TeaVM and the JVM diverge, I hit a date case they split on:
YEAR = 2002, WEEK_OF_MONTH = 2 (America/New_York, en_US)
JVM: Sun Jan 06 2002
TeaVM: Sat Jan 12 2002
This time my code was fine. The bug was in TeaVM's reimplementation of Java's GregorianCalendar. One line used days - 2 where every neighbouring branch, and the Apache Harmony code it was ported from, used days - 3:
-days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 2)) - skew;
+days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 3)) - skew;
One character. The reason no test had caught it: the suite already had four assertions that reach that exact line, commented out since 2015. My change re-enabled them instead of adding new ones. They fail on the old code and pass on the fix, which is the cleanest proof I could ask for that the fix is real and nothing else moved.
It merged into TeaVM on 2026-07-17 and closed a dormant issue. My whole contribution was a 2 to a 3 and un-commenting eleven-year-old assertions.
That's the whole thing
No cleverer reviewer would have found the first bug by reading the code, because the code looked fine and the tests were green. A second runtime found it by disagreeing. My job was the part a machine can't do: read a "0 vs 301" and decide which side was lying, and why.
If there's a transferable idea here it's just this: don't keep one source of truth for logic you can't fully check by hand. Run it two ways and treat every disagreement as a bug until you've proven which side it lives on. Sometimes it's yours. Once, it was the compiler's.
Both examples are runnable:
-
match3-engine— the Java engine, 59 tests, playable in-browser via TeaVM. -
evals-differential-oracle— a tiny browser demo of the same idea: the same match-3 rule written twice, fuzzed against each other over thousands of boards, plus a deliberately-broken version both nets catch.
Thanks to Ryan for the nudge. The last post told you I test things. This one showed you one.
Top comments (6)
"A fix is done when a check that would catch the bug is sitting on disk and passing in CI" — you took the thing we kept circling in the comments and made it load-bearing inside an actual bug, which is more convincing than any amount of us agreeing about it. The three regression tests that fail if ID generation ever re-relies on clock resolution aren't testing the fix — they're testing the assumption that broke. Most people swap in the counter and move on; you left a tripwire on the exact thing that lied to you.
And "neither runtime was broken, the code was, for depending on a guarantee it was never promised" is the cleanest statement I've seen of a whole class of bug. nanoTime didn't lie — it answered honestly on the JVM and honestly in the browser, and the two honest answers just weren't the same, because a Spectre mitigation quietly re-priced clock resolution. Same signal, different environment, different meaning: the browser wasn't wrong, it was answering a question you didn't know you were asking. The counter wins because it stops asking the environment anything at all.
The part I'm stealing: "same source, same inputs, different answer" is a test-design prompt, not just a war story. Any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. I've got a pile of clock-, locale-, and encoding-dependent assumptions in my own tools that have never once crossed the boundary that would expose them. Reading this, I'm going looking for them before they go looking for me. The fix is the least interesting part here, and you clearly knew that.
Exactly the turn I wanted the piece to make and didn't say cleanly — any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. The counter wins precisely because it stops asking the environment anything at all: no question, no fork. And yeah — go find them before they find you; the clock-, locale-, and encoding-dependent ones stay green for years because the boundary that would expose them is one you never happened to cross. If you turn up a good one, I'd genuinely like to hear which boundary caught it.
I have one, and the boundary is embarrassing.
A date-handling path in one of my systems converted a local timestamp for
storage. It ran correctly every single time I tested it, for months. It is nine
hours ahead of UTC here, so any run before 09:00 local produced the previous
day's date. Every test I wrote, and every manual check I did, happened in the
afternoon — because that is when I work. The boundary wasn't in the input space
at all. It was the hour of the day I happen to be awake.
Nothing caught it. A user did, reporting a record filed under the wrong date, and
the only reason I could reproduce it was that they told me the time.
What I took from that is narrower than "fuzz more." You cannot enumerate inputs,
but you can enumerate the questions your code asks the environment, and that
list is short: the clock, the locale, the filesystem, the platform. Four
categories, each with a handful of call sites you can grep for. That's the
practical form of your counter point — the counter wins because it asks nothing,
so the audit isn't "find the bugs," it's "find the asking." Every place the code
consults the runtime is a fork you own and did not choose.
The uncomfortable part is that my untested boundary was a fact about my
schedule. I'd bet most of these are like that: not gaps in the input space, gaps
in the author's habits.
This is a brilliant case study on why differential testing is mandatory for ensuring cross-platform determinism. The Spectre mitigation clamping timer resolution in browsers has destroyed countless "clever" RNG seeds and ID generators over the years. It's a brutal lesson in why OS clocks can never be trusted for state-critical logic.
In my domain, building strictly deterministic C++ state sync engines for Medium-Frequency Trading (MFT), we run into the exact same philosophical wall. I recently finalized the monolithic architecture for my own sync core (TolmachЁv SDK v36.0.0). When your baseline target is 41.5M TPS with ~24ns physical RTT and zero CPU validation waste, the execution loop is iterating orders of magnitude faster than any high-resolution hardware timer can accurately tick. If we relied on time-based entropy for entity IDs, the entire network simulation would desync across nodes instantly.
I love the simplicity of your
synchronized long nextId()fix to decouple from the clock. However, looking at it through a high-throughput lens: does thesynchronizedlock introduce any noticeable thread contention overhead in the JVM during massive board generation and cascades? Or is the rules engine strictly single-threaded, making the lock cost negligible compared to usingAtomicLong?Great write-up. Catching that dormant bug in TeaVM's GregorianCalendar just proves the methodology works.
The 0-versus-301 is a clean receipt, and the thing that made this particular pair of substrates catch it deserves a name: JVM and browser happened to diverge on exactly the assumption that was wrong, clock granularity. That's not guaranteed by "run it on two runtimes," it's a property of this specific pair. A differential oracle only surfaces what its two substrates disagree on, and most latent assumptions won't happen to fall on a fault line between whichever two you picked.
Which makes the real design question not "add a second substrate" but "pick a second substrate whose known divergences target your untested assumptions." Clock resolution, float rounding, locale/collation, GC pause timing, integer overflow behavior, these are catalogued ways platforms differ, and choosing a comparison substrate that's known to differ on one of them is deliberate coverage, not luck. Two substrates that happen to agree on everything you didn't test give you false confidence exactly like a single substrate would, just with the appearance of having checked.
The four commented-out GregorianCalendar assertions since 2015 are the sharper data point here. That's nine years where the divergence existed and nobody's harness was built to notice it, because nothing was asking. Differential testing's power isn't that it runs more code paths, it's treating disagreement between substrates as the default expectation to hunt for rather than an anomaly to explain away when it shows up. Worth turning into a checklist: for each unstated assumption in the code (timing, ordering, precision, encoding), is there a substrate in the test matrix known to violate it?
That reframe is the one I didn't have — I'd been treating the second runtime as a net and hoping the bug swam in. The clock-resolution catch was luck: JVM and browser happened to fall on opposite sides of the one assumption that was wrong. The checklist version — for each unstated assumption (timing, ordering, precision, encoding), is there a substrate in the matrix known to violate it? — turns the oracle from a hope into a design step, and makes the assumption catalog the real artifact, not the substrate pair. The 2015 assertions are the part that stays with me: nine years the divergence sat there and nothing broke except that nothing was asking. That's the whole thing in one receipt — the power isn't more code paths, it's making disagreement the default expectation to hunt rather than the anomaly to explain away.