A differential oracle: making agentic code prove its own correctness

Erik Hill — Wed, 15 Jul 2026 02:58:44 +0000

Six months ago I was running a kitchen. I taught myself to build agentic systems, and the thing I'm proudest of is the part nobody demos: the evaluation layer.

Coding agents are easy to demo and hard to trust. The moment an agent touches a real codebase — deploys, commits, user-facing changes — you need what any production system needs: review, tests, and a way to catch your own regressions.

The problem: correctness you can't hand-check

I build a match-3 game. Match-3 resolution has thousands of edge cases — cascades, chain reactions, special-piece rules — where the "right answer" isn't obvious and you can't check them all by hand.

The differential oracle

So instead of trusting one implementation, I use two — written independently. The same game logic runs in a React build and a native Java engine, and logic-invariant tests hold both to the same rules. If the two ever disagree on a board state, one of them is wrong. Agreement between two independently-built systems is a far stronger signal than either one passing its own tests — it has caught real bugs I'd never have found by playing.

Runnable repo: https://github.com/egnaro9/evals-differential-oracle — two implementations + invariant tests, plus a deliberately-buggy version that both nets catch. pytest passes over ~9k random boards.

Invariants as a second net

The rules themselves are also tests — e.g. a special gem is generated only by a direct match of 4+, never a 3. When I planted a bug that awarded one for a 3-run, the differential oracle disagreed and the invariant checker flagged it.

The bigger system

The oracle sits inside an autonomous dev harness: a self-reviewing loop (Strategy → Execution → Critic → Evaluation → Ops) with a cold, independent critic, per-role model routing, and a human-in-the-loop autonomy ladder that automates the launches but never the approval on anything irreversible.

Write-up: https://github.com/egnaro9/agentic-dev-harness · portfolio: https://egnaro9.github.io

What I take from it

Evals and oracles are the hard, valuable part; getting a demo to work once is not the job.
Two independent implementations that must agree beats one with a big test suite.
Be honest about what's objectively measurable vs. what needs human judgment — don't fake a metric for the second.