Six months ago I was running a kitchen. I taught myself to build agentic systems, and the thing I'm proudest of is the part nobody demos: the evaluation layer.
Coding agents are easy to demo and hard to trust. The moment an agent touches a real codebase — deploys, commits, user-facing changes — you need what any production system needs: review, tests, and a way to catch your own regressions.
The problem: correctness you can't hand-check
I build a match-3 game. Match-3 resolution has thousands of edge cases — cascades, chain reactions, special-piece rules — where the "right answer" isn't obvious and you can't check them all by hand.
The differential oracle
So instead of trusting one implementation, I use two — written independently. The same game logic runs in a React build and a native Java engine, and logic-invariant tests hold both to the same rules. If the two ever disagree on a board state, one of them is wrong. Agreement between two independently-built systems is a far stronger signal than either one passing its own tests — it has caught real bugs I'd never have found by playing.
Runnable repo: https://github.com/egnaro9/evals-differential-oracle — two implementations + invariant tests, plus a deliberately-buggy version that both nets catch. pytest passes over ~9k random boards.
Invariants as a second net
The rules themselves are also tests — e.g. a special gem is generated only by a direct match of 4+, never a 3. When I planted a bug that awarded one for a 3-run, the differential oracle disagreed and the invariant checker flagged it.
The bigger system
The oracle sits inside an autonomous dev harness: a self-reviewing loop (Strategy → Execution → Critic → Evaluation → Ops) with a cold, independent critic, per-role model routing, and a human-in-the-loop autonomy ladder that automates the launches but never the approval on anything irreversible.
Write-up: https://github.com/egnaro9/agentic-dev-harness · portfolio: https://egnaro9.github.io
What I take from it
- Evals and oracles are the hard, valuable part; getting a demo to work once is not the job.
- Two independent implementations that must agree beats one with a big test suite.
- Be honest about what's objectively measurable vs. what needs human judgment — don't fake a metric for the second.
I'm a self-taught engineer looking for a remote (US) role in agentic / AI / evals engineering. If this is what your team cares about, I'd love to talk.
Top comments (0)