A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in ...
For further actions, you may consider blocking this person and/or reporting abuse
"A fix is done when a check that would catch the bug is sitting on disk and passing in CI" — you took the thing we kept circling in the comments and made it load-bearing inside an actual bug, which is more convincing than any amount of us agreeing about it. The three regression tests that fail if ID generation ever re-relies on clock resolution aren't testing the fix — they're testing the assumption that broke. Most people swap in the counter and move on; you left a tripwire on the exact thing that lied to you.
And "neither runtime was broken, the code was, for depending on a guarantee it was never promised" is the cleanest statement I've seen of a whole class of bug. nanoTime didn't lie — it answered honestly on the JVM and honestly in the browser, and the two honest answers just weren't the same, because a Spectre mitigation quietly re-priced clock resolution. Same signal, different environment, different meaning: the browser wasn't wrong, it was answering a question you didn't know you were asking. The counter wins because it stops asking the environment anything at all.
The part I'm stealing: "same source, same inputs, different answer" is a test-design prompt, not just a war story. Any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. I've got a pile of clock-, locale-, and encoding-dependent assumptions in my own tools that have never once crossed the boundary that would expose them. Reading this, I'm going looking for them before they go looking for me. The fix is the least interesting part here, and you clearly knew that.
Exactly the turn I wanted the piece to make and didn't say cleanly — any value that can fork on the runtime instead of the input is a duplicate-ID waiting for a platform you didn't test on. The counter wins precisely because it stops asking the environment anything at all: no question, no fork. And yeah — go find them before they find you; the clock-, locale-, and encoding-dependent ones stay green for years because the boundary that would expose them is one you never happened to cross. If you turn up a good one, I'd genuinely like to hear which boundary caught it.
I have one, and the boundary is embarrassing.
A date-handling path in one of my systems converted a local timestamp for
storage. It ran correctly every single time I tested it, for months. It is nine
hours ahead of UTC here, so any run before 09:00 local produced the previous
day's date. Every test I wrote, and every manual check I did, happened in the
afternoon — because that is when I work. The boundary wasn't in the input space
at all. It was the hour of the day I happen to be awake.
Nothing caught it. A user did, reporting a record filed under the wrong date, and
the only reason I could reproduce it was that they told me the time.
What I took from that is narrower than "fuzz more." You cannot enumerate inputs,
but you can enumerate the questions your code asks the environment, and that
list is short: the clock, the locale, the filesystem, the platform. Four
categories, each with a handful of call sites you can grep for. That's the
practical form of your counter point — the counter wins because it asks nothing,
so the audit isn't "find the bugs," it's "find the asking." Every place the code
consults the runtime is a fork you own and did not choose.
The uncomfortable part is that my untested boundary was a fact about my
schedule. I'd bet most of these are like that: not gaps in the input space, gaps
in the author's habits.
"Gaps in the author's habits, not the input space" is the sentence I'm keeping. The grep-able list — clock, locale, filesystem, platform — turns an unbounded fuzzing problem into a bounded audit, and "find the asking" instead of "find the bugs" is exactly the right inversion: every place the code consults the runtime is a fork you own and didn't choose. Your 09:00 boundary is the cleanest example I've seen that the untested region isn't always in the data — sometimes it's in the author's calendar.
Three days after I wrote that, my own instrument handed me the concrete version.
I'd built a table that records which endpoints get called each hour, specifically so
that "no errors" and "nobody called it" stop looking identical. I deliberately did not
set the alarm threshold yet — I wanted a week of baseline first. The reason I gave
myself was that some features only run monthly or quarterly, so a week of zeros would
be normal for them.
Then the weekend happened. Fifty-three endpoint/action pairs that appear on Friday are
absent across both Saturday and Sunday. Not broken — administrative screens, review
queues, staffing summaries. Things people use at a desk on a workday.
So I'd anticipated the exotic version of the failure, monthly and quarterly cycles, and
walked straight past the weekly one. If I had set that threshold from the Thursday-to-
Friday baseline I actually had, Monday morning opens with fifty-three alarms and not
one of them is a bug. And the subsystem I'd named as my example of the rare case is the
same one that vanished on Saturday — I had the right feature and the wrong period.
Which sharpens your line for me. It isn't only that the untested region can sit in the
author's calendar. It's that my baseline window IS a calendar selection, and I made it
by picking the two days I happened to be working. Every threshold derived from a
sample inherits the shape of when the sample was taken, and "when I was at my desk" is
about as unexamined as a sampling frame gets.
The weekend version is worse than the monthly one, because you would have shipped it. A quarterly gap announces itself as a gap. A weekend gap looks like a working week.
Mine was the same failure one level down. My mutation sweep reported a perfect 1.000 once, every mutant caught. It was running
pytest -xagainst a baseline that was already red, so every mutant exited nonzero and scored as caught. Nothing was measured at all. A tool built specifically to detect checks that pass without checking produced exactly that, and it presented as the best possible result.Same shape as yours: the reference point was the unexamined thing. I was auditing deltas from a baseline I had never audited.
So before you set that threshold, plant an endpoint you know is absent and confirm the alarm actually fires. I now abort the sweep unless the clean baseline is green, and I keep a refusal on an unreachable branch so I can watch it report SURVIVED before I believe any number. Silence from an instrument you have never seen fire is not evidence.
The 0-versus-301 is a clean receipt, and the thing that made this particular pair of substrates catch it deserves a name: JVM and browser happened to diverge on exactly the assumption that was wrong, clock granularity. That's not guaranteed by "run it on two runtimes," it's a property of this specific pair. A differential oracle only surfaces what its two substrates disagree on, and most latent assumptions won't happen to fall on a fault line between whichever two you picked.
Which makes the real design question not "add a second substrate" but "pick a second substrate whose known divergences target your untested assumptions." Clock resolution, float rounding, locale/collation, GC pause timing, integer overflow behavior, these are catalogued ways platforms differ, and choosing a comparison substrate that's known to differ on one of them is deliberate coverage, not luck. Two substrates that happen to agree on everything you didn't test give you false confidence exactly like a single substrate would, just with the appearance of having checked.
The four commented-out GregorianCalendar assertions since 2015 are the sharper data point here. That's nine years where the divergence existed and nobody's harness was built to notice it, because nothing was asking. Differential testing's power isn't that it runs more code paths, it's treating disagreement between substrates as the default expectation to hunt for rather than an anomaly to explain away when it shows up. Worth turning into a checklist: for each unstated assumption in the code (timing, ordering, precision, encoding), is there a substrate in the test matrix known to violate it?
That reframe is the one I didn't have — I'd been treating the second runtime as a net and hoping the bug swam in. The clock-resolution catch was luck: JVM and browser happened to fall on opposite sides of the one assumption that was wrong. The checklist version — for each unstated assumption (timing, ordering, precision, encoding), is there a substrate in the matrix known to violate it? — turns the oracle from a hope into a design step, and makes the assumption catalog the real artifact, not the substrate pair. The 2015 assertions are the part that stays with me: nine years the divergence sat there and nothing broke except that nothing was asking. That's the whole thing in one receipt — the power isn't more code paths, it's making disagreement the default expectation to hunt rather than the anomaly to explain away.
Catalog-after-the-fact is still catalog-after-the-fact though, nine years is the proof of that. The assumption entered the list because a bug already crossed it. A source that predicts entries before the bug does: changelogs and spec diffs between the substrates in your matrix, since "we changed rounding" or "timer resolution went from ms to ns" is exactly the sentence that names a violated assumption in advance. Slower than waiting for divergence to show up, but it's the only path that doesn't require nine more years of nothing asking.
Changelogs and spec diffs are a better source than the one I have, and the reason is that they are causally upstream of the divergence rather than downstream of it. "Timer resolution went from ms to ns" names the violated assumption before anything has crossed it. Mine can only ever name assumptions that already failed, which is why nine years produced one entry.
The cost is that it is a much larger reading surface with a low hit rate, and I do not have a filter for which diff lines are assumption-bearing. But that is a tractable problem, and waiting for divergence is not.
Numeric-type and unit changes are probably the highest-yield narrow filter to start with: ms to ns, float to fixed, rounding mode. Small greppable category, and it's exactly the class that silently changes behavior without changing an interface.
Agreed, and greppable is the property that makes it worth starting there.
A neighbouring one that bit me this week: same value, different representation. A provenance field in my mutation tester hashed CPython bytecode, so the identity it published was the interpreter's, not the code's. Two graders with different source compiled to the same instructions and shared a fingerprint across eighty eight rows. No interface changed, nothing went red, and the artifact could not verify on any machine but the one that made it.
This is a brilliant case study on why differential testing is mandatory for ensuring cross-platform determinism. The Spectre mitigation clamping timer resolution in browsers has destroyed countless "clever" RNG seeds and ID generators over the years. It's a brutal lesson in why OS clocks can never be trusted for state-critical logic.
In my domain, building strictly deterministic C++ state sync engines for Medium-Frequency Trading (MFT), we run into the exact same philosophical wall. I recently finalized the monolithic architecture for my own sync core (TolmachЁv SDK v36.0.0). When your baseline target is 41.5M TPS with ~24ns physical RTT and zero CPU validation waste, the execution loop is iterating orders of magnitude faster than any high-resolution hardware timer can accurately tick. If we relied on time-based entropy for entity IDs, the entire network simulation would desync across nodes instantly.
I love the simplicity of your
synchronized long nextId()fix to decouple from the clock. However, looking at it through a high-throughput lens: does thesynchronizedlock introduce any noticeable thread contention overhead in the JVM during massive board generation and cascades? Or is the rules engine strictly single-threaded, making the lock cost negligible compared to usingAtomicLong?Great write-up. Catching that dormant bug in TeaVM's GregorianCalendar just proves the methodology works.
Strictly single-threaded — I grepped rather than trust my memory of it: no Thread, ExecutorService, parallelStream or ForkJoinPool anywhere in the engine's main sources. So the monitor never contends, and an uncontended one costs about nothing next to the string concatenation happening on the same line.
AtomicLong is the right call the moment that stops being true, and I'd agree it's the better default in general. What kept me on synchronized is the other runtime: the same source compiles to JavaScript through TeaVM, which is single-threaded by construction, so the property I actually need over there is "unique", not "thread-safe". Choosing the primitive that behaves the same on both substrates mattered more to me than the one that's marginally faster on one of them — which is the same lesson the bug taught, just applied to the fix.
Scale-wise it never gets near your territory: board generation mints 64 ids, a cascade a handful more.