DEV Community: Erik Hill

301 duplicate IDs in the browser, 0 on the JVM: one real bug, end to end

Erik Hill — Wed, 22 Jul 2026 00:01:43 +0000

A reader named Ryan left the sharpest comment on my last post. The gist: it was jargon-heavy, kept restating "you have to test the AI's output" in new words, and read more like a pitch deck than a case study. He was right, and he pointed at the fix himself: walk through one real task. What changed, what caught the problem, what proof was required, where did a human step in.

So here is one task, start to finish. Everything below is public and runnable. No withheld details, no diagrams of boxes with arrows.

The setup, in two sentences

My match-3 game runs on a plain-Java rules engine. It runs two ways: on a JVM (where the tests and CI live) and in the browser, compiled to JavaScript by TeaVM, so the same Java drives a real playable board.

That second runtime is not just a demo. Running one piece of logic on two different machines gives me a free check: where the two disagree, one of them is wrong. I did not have to write down the right answer. I just had to notice a disagreement.

Here is a task where that check earned its keep.

What changed

I ported the engine to the browser. New demo-js module, TeaVM config, opaque integer handles in and JSON out so boards never actually cross into JavaScript. Mechanical work. The engine code barely moved.

Except the port compiled a line I had never once looked at hard. This is how every gem got its ID:

String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);

Timestamp plus a random number. It had passed every test for months.

What the check caught

The IDs are how the renderer tells one gem from another. Two gems with the same ID animate as a single gem. So I ran the same board generation on both runtimes and counted collisions over 128,000 gems.

JVM: 0 duplicates.
Browser (TeaVM): 301 duplicates.

Same source code. Same inputs. Different answer. That is the entire signal. A machine counted it; I did not have to guess that something felt off.

Where I stepped in

A number this specific still needs a human to say which side is wrong and why. That part was me.

System.nanoTime() looks unique but only leans on the clock being high-resolution enough that two calls land on different values. A JVM's timer is fine, so the flaw was invisible there. Browsers deliberately clamp their clock to about 100 microseconds (a Spectre mitigation), so nanoTime barely advances between gems and RNG.nextInt(1000) collides on its own often enough to matter.

Neither runtime was broken. The code was, for depending on clock resolution it was never promised. The browser was just honest about it.

The fix is boring, which is the point:

private static long idSeq = 0L;

private static synchronized long nextId() {
    return idSeq++;
}

-String id = row + "-" + col + "-" + System.nanoTime() + "-" + RNG.nextInt(1000);
+String id = row + "-" + col + "-" + nextId();

A counter is unique on every clock. The guarantee stops depending on the platform.

What had to exist before it could close

The one rule I do not bend: a fix is not done because I say "fixed." It is done when a check that would catch the bug is sitting on disk and passing in CI. For this one, that meant three tests whose whole job is to fail if IDs ever lean on the clock again:

@Test
public void idsAreUniqueWithoutRelyingOnClockResolution() {
    // Mint many gems as fast as possible: a clock-derived id would collide
    // here on any platform whose timer doesn't advance between calls.
    Set<String> seen = new HashSet<>();
    for (int i = 0; i < 50; i++) {
        GameBoard.Gem[][] b = BoardEngine.createBoard(plain());
        for (GameBoard.Gem[] row : b)
            for (GameBoard.Gem g : row) seen.add(g.id);
    }
    assertEquals(50 * 64, seen.size());
}

Plus one for board creation and one for the refill hot path, where new gems get minted every cascade. The suite went from 51 tests to 59. Those three are the receipt that the specific failure cannot come back quietly. Without them, "I fixed the ID thing" is just a sentence.

The commit and the tests are here: match3-engine (BoardEngine.java, IdUniquenessTest.java).

The same trick, one level up

Around the same time, the same "run it two ways, look for a quiet disagreement" habit pointed the other direction. Auditing where TeaVM and the JVM diverge, I hit a date case they split on:

YEAR = 2002, WEEK_OF_MONTH = 2   (America/New_York, en_US)
JVM:   Sun Jan 06 2002
TeaVM: Sat Jan 12 2002

This time my code was fine. The bug was in TeaVM's reimplementation of Java's GregorianCalendar. One line used days - 2 where every neighbouring branch, and the Apache Harmony code it was ported from, used days - 3:

-days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 2)) - skew;
+days += (fields[WEEK_OF_MONTH] - 1) * 7 + mod7(skew + dayOfWeek - (days - 3)) - skew;

One character. The reason no test had caught it: the suite already had four assertions that reach that exact line, commented out since 2015. My change re-enabled them instead of adding new ones. They fail on the old code and pass on the fix, which is the cleanest proof I could ask for that the fix is real and nothing else moved.

It merged into TeaVM on 2026-07-17 and closed a dormant issue. My whole contribution was a 2 to a 3 and un-commenting eleven-year-old assertions.

That's the whole thing

No cleverer reviewer would have found the first bug by reading the code, because the code looked fine and the tests were green. A second runtime found it by disagreeing. My job was the part a machine can't do: read a "0 vs 301" and decide which side was lying, and why.

If there's a transferable idea here it's just this: don't keep one source of truth for logic you can't fully check by hand. Run it two ways and treat every disagreement as a bug until you've proven which side it lives on. Sometimes it's yours. Once, it was the compiler's.

Both examples are runnable:

match3-engine — the Java engine, 59 tests, playable in-browser via TeaVM.
evals-differential-oracle — a tiny browser demo of the same idea: the same match-3 rule written twice, fuzzed against each other over thousands of boards, plus a deliberately-broken version both nets catch.

Thanks to Ryan for the nudge. The last post told you I test things. This one showed you one.

Distinguishing wrong from absent

Erik Hill — Tue, 21 Jul 2026 19:42:32 +0000

model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.

I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.

The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model.

The design was already exclude-by-class, not blanket. Code: github.com/egnaro9/model-drift

Writing the methodology in public is how it gets read this closely.

I built agentic AI that checks its own work — a 90-second tour

Erik Hill — Tue, 21 Jul 2026 04:19:21 +0000

I built an AI dev harness that isn't allowed to trust itself

Erik Hill — Mon, 20 Jul 2026 02:12:53 +0000

Scope note (read first): this describes a system I built and operate to develop an unannounced game. The game's identity, mechanics, and assets are withheld, and so are the harness's tuned prompts, gate implementations, and internal failure specifics. What's shown here is the method and the evidence discipline — the transferable part.

Summary

Over about four months (spring–summer 2026) I built and operated, solo, an operator-supervised multi-agent development harness that builds a real, shipping product. Its defining property isn't speed — it's that nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human. This is a case study of the system and the evidence trail it leaves.

The idea

LLM coding agents are fast and unreliable. The engineering problem isn't getting output — it's trusting it. So the harness is built around one rule: an agent's work is unverified until a gate proves it. Coordination is automated; consequences are gated. And it's built to ship, not to gold-plate: every gate exists so I can move fast without shipping something broken — verification in service of velocity, not instead of it.

Architecture (concept level)

A five-role loop: Strategy → Execution → Critic → Eval → Ops. Judgment roles (Strategy, Critic, Eval) run on stronger models; execution roles on cheaper ones — cost follows the difficulty of the decision, not a flat default.
A manager / orchestration layer. Above the execution agents sits one orchestration role that I direct — it plans each unit of work, routes it to the right role and model, and holds the system's state between steps. I designed the roles, the gates, and the routing; the harness runs them. I'm not outside the loop supervising a black box — I'm the system's judgment and authority, and the manager is the layer that extends that across many parallel agents.
A cold, independent critic gate. Before a consequential change closes, it's reviewed by a Critic running on a fresh, zero-context session — a different strong model with no memory of how the code was written — so it reviews the work itself, not the author's rationale for it. It can send the change back for rework. A self-review rubber-stamps; a cold critic catches what the author already talked themselves past.
A human-in-the-loop autonomy ladder: the loop's handoffs are automated — one role hands to the next without me — but every irreversible act (deploying a build to a device, committing to git) stays behind an explicit human approval. Automate coordination; never automate the irreversible.
A differential oracle for correctness: the core logic is implemented twice and the two versions are fuzzed against each other. Where they disagree, one is wrong — no gold labels required.

The evidence discipline (the differentiator)

Every closed unit of work leaves a durable, machine-checkable proof:

NO-PROOF-NO-CLOSE gate. A work item cannot close until an automated check confirms its proof exists on disk. The loop physically cannot skip it.
Provenance-bound proof. On-device validation screenshots are sanitized (sensitive regions blacked out), and provenance manifests bind images to the exact git SHA, screen dimensions, and redaction method that produced them — so an artifact traces back to the commit it proves.
Human-gated checkpoints. Each checkpoint records scoped git staging (explicit paths only), a commit/SHA trail across the repos it touches, an artifact-registry audit, and an explicit operator approval.
Periodic self-evaluation. An independent evaluation role produces a numeric health score with a delta versus the prior period and a failure taxonomy; regressions feed a failure registry that drives fixes.

The testing oracle

The product's core logic is held to property-based invariant tests — generated inputs are thrown at the engine and a set of invariants must hold for every one (e.g. a detector must agree with an independent full re-scan, and detection must be side-effect-free). The suite runs against the authoritative implementation, so an invariant is enforced on the logic, not asserted in prose (last run: zero failures). As a standalone, fully public demonstration of the same technique, my match3-engine repo carries 16 jqwik property invariants over random inputs.

Operating record (Apr–Jul 2026, from the on-disk archive)

~200 completed work-arcs · ~190 human-gated checkpoints · 74 independent critic reviews · 13 periodic self-evaluations · a growing failure registry with per-item root-cause fixes · a ~200-file sanitized proof archive with ~90 provenance manifests.

Verifiable outcomes (all public)

An arcade game — Tap Dodge Rush, under SeraphLight Studios — shipped end-to-end to Google Play.
A one-character bug fix merged upstream into TeaVM (the Java-to-JavaScript compiler), closing a long-dormant issue.
A live public model-drift board grading 16 LLMs daily on a frozen, deterministically-graded suite — no LLM-as-judge, so a score change is real.
Ten public repos, including a differential-oracle testing project and a Model Context Protocol server built from the spec.

What I'd bring to a team

Treat AI output as unverified until proven. Build the gate before the feature. Make failures loud, not silent. Keep a human on the irreversible path. The discipline transfers to any codebase — the harness just made me practice it a few hundred times.

Full architecture case study & repo: github.com/egnaro9/agentic-dev-harness · Portfolio: egnaro9.github.io

Set membership is not pairing: a property test that was green for the exact bug it was written to catch

Erik Hill — Sun, 19 Jul 2026 05:57:09 +0000

I have a small board that tracks 16 LLMs across 5 labs on a frozen 35-task suite. The charts are the easy part. The paragraph above them is the part I did not trust, because I had written that kind of paragraph by hand twice and both times it went false — true when I typed it, falsified by a later run.

So I stopped typing it. The paragraph is generated now. Each sentence is a claim: a predicate over the current numbers plus a renderer, returning a string or None. A claim whose predicate stops holding is dropped, not reworded. The generator can be silent. It can't be wrong.

That second sentence is the part I got wrong.

The sentence that lied

One claim compares a lab's cheaper models against that same lab's flagship. On the live board it produced this:

Google's cheaper tiers match or beat its own flagship — Gemini 3.5 Flash at 100% and Gemini 3.1 Flash-Lite at 95%, against Gemini 3.1 Pro at 95%

Correct. But the first version of that renderer collected the winners, took the highest score among them, and printed one number for the group:

scores = f"{_names(winners)} at {_pct(max(r.acc for r in winners))}"

Which reads: Gemini 3.5 Flash and Gemini 3.1 Flash-Lite at 100%. Flash-Lite scored 95%. The sentence put a real number next to a model that did not earn it.

Naming several models and then printing one number is not a formatting choice. It is a claim about all of them.

The test that should have caught it

I had written a property test for exactly this class of bug. It asserted that every percentage appearing in the prose was a score some model actually got:

real = {f"{p[-1]['acc'] * 100:.0f}%" for k, p in m["series"].items()}
printed = set(re.findall(r"\d+%", out["text"]))
assert printed <= real

It was green.

It is green because the assertion is true. 100% was a real score — Gemini 3.5 Flash got it. The test asked "does this number exist in the data?" The bug was "is this number attached to the right model?" Set membership where pairing was required.

The replacement checks the pairing:

_PAIR = re.compile(r"\b([AB] (?:Big|Small|Tiny)) at (\d+)%")

for label, printed in _PAIR.findall(out["text"]):
    assert f"{actual[label] * 100:.0f}" == printed

The fixture was wrong too

This is the part I would have missed if I had stopped at the test.

A fixture with one cheap model per lab cannot reach the failing state — with a single cheap model per group, a group is always unanimous, so a renderer printing rows[0]'s score is always right. That was the shape I started from, and I fixed it before the first commit, so take this part on my word rather than on the history. The property test was not weak because of its assertion alone. It was weak because the data it ran against could not produce a disagreement.

The fixture now gives LabA two cheaper tiers under one flagship, which is the shape Google actually has (Pro / Flash / Flash-Lite):

REGISTRY = [
    {"id": "lab-a:big",   "label": "A Big",   "group": "LabA", "tier": "flagship"},
    {"id": "lab-a:small", "label": "A Small", "group": "LabA", "tier": "mini"},
    {"id": "lab-a:tiny",  "label": "A Tiny",  "group": "LabA", "tier": "nano"},
    ...
]

and the baseline board puts those two cheap tiers at different scores, both at or above their flagship.

What it cost

Eight mutants run against the new guards. Three survived the first time — the tie guard, the single-flagship rule, and a float-truncation fix — and each one needed a test written before it was really covered. The repair was less covered than it felt.

The float one is my favourite, because it is a bug in the code whose entire job was to never overstate a number. Percentages truncate rather than round, so a model on 99.6% can never print as "100%" — a perfect score it did not get, in the one place a reader checks hardest. But:

# The epsilon is not decoration: 0.58 * 100 is 57.99999999999999 in binary
# floating point, so a bare floor prints a measured 58% as "57%". [...]
return f"{math.floor(value * 100 + 1e-9)}%"

A guard against overstating had started understating instead. That is still wrong prose about real data — just wrong in the flattering direction, which is the direction you don't check.

The suite is 125 tests now, stdlib only, 0.04s. It was 91 at the fix commit, 74 when the generator first landed, and 22 before the generator existed.

What I'd do differently

Ask what state would make this assertion fail, and name it out loud. I could not have named it for the set test — not because I hadn't thought about it, but because the fixture made it unreachable and I never noticed that the two facts were connected. An assertion whose failing state you cannot describe is decoration.

Mutate the code, not just the inputs. Property tests over generated data feel rigorous enough to skip this. Three of eight mutants survived my new, careful, adversarially-reviewed guards. That is the number that convinced me.

Assume the repair is uncovered. The fix arrives believing in itself. It has no tests yet.

And one more, which I found while writing this. The README badge for that repo says tests-91. It is 125. Four screens further down, the quickstart says # 22 tests — a badge and its own README disagreeing, both hand-typed, both true on the day they were written and false since. That is the identical failure mode the generator exists to eliminate, sitting in the repo that hosts the generator.

The verification was broken, not the code. It usually is.

Repo: github.com/egnaro9/model-drift — the broken test and the real one are both in tests/test_narrative.py.
Portfolio: egnaro9.github.io

The Cold-Context Critic: A Reviewer That Never Remembers Writing the Code

Erik Hill — Thu, 16 Jul 2026 01:22:41 +0000

Anyone who has written code and then reviewed it minutes later knows the feeling: the diff looks fine. Of course it does. The person reading it is the same person who just talked themselves into every line of it. The naming made sense because the intent is still warm in memory. The shortcut felt justified because the reason for it is still sitting in working memory, uninspected.

That is the problem the cold-context critic is built to remove.

The core idea

In the harness I built to develop a game, every change passes through a review step performed by a fresh model instance that has no memory of writing the code. It did not plan the change. It did not argue for the approach. It did not feel the small relief of getting something to finally pass. It receives the diff and evaluates it on its merits, the way a stranger would.

The intuition is old and boring, which is part of why I trust it: an author rationalizes; a cold reviewer does not. A model — or a person — that just produced the work carries a bias toward the choices it already made. Strip the memory of making them, and the same mistakes stop being self-evidently correct. The reviewer has nothing to defend. It can only look at what is actually there.

I want to be precise about the claim, because it is easy to oversell. This is a discipline, an architecture choice. I am not going to tell you it caught a specific dramatic production bug, because I am not going to invent an incident to sell an idea. What I can say is the mechanism: cold review catches the class of mistakes an author talks itself into — the plausible-looking shortcut, the assumption never stated out loud, the edge case that "obviously" can't happen. Self-review is structurally bad at exactly those, because the author already accepted them once.

Where it fits

The cold critic is one layer, not the whole story. It sits alongside a few other pieces of the same distrust:

Per-role model routing. Stronger models run the judgment stages — planning, critique, evaluation — and cheaper ones handle routine execution. The reviewer isn't grading its own homework, and it isn't the same weight class picked for speed.
A differential oracle. Core game logic is implemented twice and held to shared invariants. When the two implementations disagree, at least one is wrong, and I find out without having to trust either.
A human approval gate on every irreversible action. Deploys and commits do not happen because a model decided they should. They happen because a person approves them. That gate never moves.

I think about the whole thing on an autonomy ladder, L0 through L4 — from fully hands-on toward more independence. The cold critic is what makes climbing that ladder defensible. More autonomy is only safe if the verification underneath it does not depend on the thing being verified vouching for itself.

The setup, honestly

Concretely: the critic is a separate invocation with a clean context window. It does not inherit the conversation that produced the change. It gets the diff and the standard it is being held to, and it reports back before a human sees the change. That's the shape of it. The value isn't a clever prompt — it's the missing memory. The setup exists to guarantee the reviewer never accumulated a reason to like the code.

That's also why I keep the human gate immovable. The cold critic is a good reader, not an authority. It can flag; it doesn't get to ship.

An honest note

I'm about five to six months into this, career-changing from roughly six years in professional kitchens, no CS degree — I learned through freeCodeCamp and by building. The game is real and published to Google Play, but it is not serving millions, and I'm not running production RAG or fine-tuning pipelines. I'd rather tell you what this is than dress it up.

So the honest framing is this: I don't have the scars yet that teach senior engineers where the bodies are buried. What I can do is build the distrust in as a habit from the start — assume the author is biased, including when the author is me or a model I'm running, and make something with no stake in the answer do the checking.

If you want to see the same idea in code you can actually run, the differential oracle is public: github.com/egnaro9/evals-differential-oracle. It's the same move as the cold critic — distrust your own output — implemented as two independent versions of the logic checked against each other, plus invariant tests over thousands of random boards, with a deliberately buggy implementation included to prove the checks bite. Clone it, run pytest, watch it catch the planted bug.

More at egnaro9.github.io and github.com/egnaro9.

A differential oracle: making agentic code prove its own correctness

Erik Hill — Wed, 15 Jul 2026 02:58:44 +0000

Six months ago I was running a kitchen. I taught myself to build agentic systems, and the thing I'm proudest of is the part nobody demos: the evaluation layer.

Coding agents are easy to demo and hard to trust. The moment an agent touches a real codebase — deploys, commits, user-facing changes — you need what any production system needs: review, tests, and a way to catch your own regressions.

The problem: correctness you can't hand-check

I build a match-3 game. Match-3 resolution has thousands of edge cases — cascades, chain reactions, special-piece rules — where the "right answer" isn't obvious and you can't check them all by hand.

The differential oracle

So instead of trusting one implementation, I use two — written independently. The same game logic runs in a React build and a native Java engine, and logic-invariant tests hold both to the same rules. If the two ever disagree on a board state, one of them is wrong. Agreement between two independently-built systems is a far stronger signal than either one passing its own tests — it has caught real bugs I'd never have found by playing.

Runnable repo: https://github.com/egnaro9/evals-differential-oracle — two implementations + invariant tests, plus a deliberately-buggy version that both nets catch. pytest passes over ~9k random boards.

Invariants as a second net

The rules themselves are also tests — e.g. a special gem is generated only by a direct match of 4+, never a 3. When I planted a bug that awarded one for a 3-run, the differential oracle disagreed and the invariant checker flagged it.

The bigger system

The oracle sits inside an autonomous dev harness: a self-reviewing loop (Strategy → Execution → Critic → Evaluation → Ops) with a cold, independent critic, per-role model routing, and a human-in-the-loop autonomy ladder that automates the launches but never the approval on anything irreversible.

Write-up: https://github.com/egnaro9/agentic-dev-harness · portfolio: https://egnaro9.github.io

What I take from it

Evals and oracles are the hard, valuable part; getting a demo to work once is not the job.
Two independent implementations that must agree beats one with a big test suite.
Be honest about what's objectively measurable vs. what needs human judgment — don't fake a metric for the second.

I'm a self-taught engineer looking for a remote (US) role in agentic / AI / evals engineering. If this is what your team cares about, I'd love to talk.