The Cold-Context Critic: A Reviewer That Never Remembers Writing the Code

Erik Hill — Thu, 16 Jul 2026 01:22:41 +0000

Anyone who has written code and then reviewed it minutes later knows the feeling: the diff looks fine. Of course it does. The person reading it is the same person who just talked themselves into every line of it. The naming made sense because the intent is still warm in memory. The shortcut felt justified because the reason for it is still sitting in working memory, uninspected.

That is the problem the cold-context critic is built to remove.

The core idea

In the harness I built to develop a game, every change passes through a review step performed by a fresh model instance that has no memory of writing the code. It did not plan the change. It did not argue for the approach. It did not feel the small relief of getting something to finally pass. It receives the diff and evaluates it on its merits, the way a stranger would.

The intuition is old and boring, which is part of why I trust it: an author rationalizes; a cold reviewer does not. A model — or a person — that just produced the work carries a bias toward the choices it already made. Strip the memory of making them, and the same mistakes stop being self-evidently correct. The reviewer has nothing to defend. It can only look at what is actually there.

I want to be precise about the claim, because it is easy to oversell. This is a discipline, an architecture choice. I am not going to tell you it caught a specific dramatic production bug, because I am not going to invent an incident to sell an idea. What I can say is the mechanism: cold review catches the class of mistakes an author talks itself into — the plausible-looking shortcut, the assumption never stated out loud, the edge case that "obviously" can't happen. Self-review is structurally bad at exactly those, because the author already accepted them once.

Where it fits

The cold critic is one layer, not the whole story. It sits alongside a few other pieces of the same distrust:

Per-role model routing. Stronger models run the judgment stages — planning, critique, evaluation — and cheaper ones handle routine execution. The reviewer isn't grading its own homework, and it isn't the same weight class picked for speed.
A differential oracle. Core game logic is implemented twice and held to shared invariants. When the two implementations disagree, at least one is wrong, and I find out without having to trust either.
A human approval gate on every irreversible action. Deploys and commits do not happen because a model decided they should. They happen because a person approves them. That gate never moves.

I think about the whole thing on an autonomy ladder, L0 through L4 — from fully hands-on toward more independence. The cold critic is what makes climbing that ladder defensible. More autonomy is only safe if the verification underneath it does not depend on the thing being verified vouching for itself.

The setup, honestly

Concretely: the critic is a separate invocation with a clean context window. It does not inherit the conversation that produced the change. It gets the diff and the standard it is being held to, and it reports back before a human sees the change. That's the shape of it. The value isn't a clever prompt — it's the missing memory. The setup exists to guarantee the reviewer never accumulated a reason to like the code.

That's also why I keep the human gate immovable. The cold critic is a good reader, not an authority. It can flag; it doesn't get to ship.

An honest note

I'm about five to six months into this, career-changing from roughly six years in professional kitchens, no CS degree — I learned through freeCodeCamp and by building. The game is real and published to Google Play, but it is not serving millions, and I'm not running production RAG or fine-tuning pipelines. I'd rather tell you what this is than dress it up.

So the honest framing is this: I don't have the scars yet that teach senior engineers where the bodies are buried. What I can do is build the distrust in as a habit from the start — assume the author is biased, including when the author is me or a model I'm running, and make something with no stake in the answer do the checking.

If you want to see the same idea in code you can actually run, the differential oracle is public: github.com/egnaro9/evals-differential-oracle. It's the same move as the cold critic — distrust your own output — implemented as two independent versions of the logic checked against each other, plus invariant tests over thousands of random boards, with a deliberately buggy implementation included to prove the checks bite. Clone it, run pytest, watch it catch the planted bug.

More at egnaro9.github.io and github.com/egnaro9.

A differential oracle: making agentic code prove its own correctness

Erik Hill — Wed, 15 Jul 2026 02:58:44 +0000

Six months ago I was running a kitchen. I taught myself to build agentic systems, and the thing I'm proudest of is the part nobody demos: the evaluation layer.

Coding agents are easy to demo and hard to trust. The moment an agent touches a real codebase — deploys, commits, user-facing changes — you need what any production system needs: review, tests, and a way to catch your own regressions.

The problem: correctness you can't hand-check

I build a match-3 game. Match-3 resolution has thousands of edge cases — cascades, chain reactions, special-piece rules — where the "right answer" isn't obvious and you can't check them all by hand.

The differential oracle

So instead of trusting one implementation, I use two — written independently. The same game logic runs in a React build and a native Java engine, and logic-invariant tests hold both to the same rules. If the two ever disagree on a board state, one of them is wrong. Agreement between two independently-built systems is a far stronger signal than either one passing its own tests — it has caught real bugs I'd never have found by playing.

Runnable repo: https://github.com/egnaro9/evals-differential-oracle — two implementations + invariant tests, plus a deliberately-buggy version that both nets catch. pytest passes over ~9k random boards.

Invariants as a second net

The rules themselves are also tests — e.g. a special gem is generated only by a direct match of 4+, never a 3. When I planted a bug that awarded one for a 3-run, the differential oracle disagreed and the invariant checker flagged it.

The bigger system

The oracle sits inside an autonomous dev harness: a self-reviewing loop (Strategy → Execution → Critic → Evaluation → Ops) with a cold, independent critic, per-role model routing, and a human-in-the-loop autonomy ladder that automates the launches but never the approval on anything irreversible.

Write-up: https://github.com/egnaro9/agentic-dev-harness · portfolio: https://egnaro9.github.io

What I take from it

Evals and oracles are the hard, valuable part; getting a demo to work once is not the job.
Two independent implementations that must agree beats one with a big test suite.
Be honest about what's objectively measurable vs. what needs human judgment — don't fake a metric for the second.

I'm a self-taught engineer looking for a remote (US) role in agentic / AI / evals engineering. If this is what your team cares about, I'd love to talk.

DEV Community: Erik Hill