Adding a second model to review your agent's work sounds like free safety. It only fixes the mistakes the reviewer does not share, and its false alarms can break work that was already right.
Engineer's view. This is running two replicas of the same software for safety. Both crash on the same leap second, because they share the bug. A reviewer model trained like the drafter misses the same failures. Pick a reviewer that fails differently, and measure what it catches on your drafter's failures, not on its own.
TL;DR. A reviewer model can only fix errors it would not have made itself, and models share many blind spots. Measure what it fixes and what it breaks on your own tasks, then review only where the drafter often fails.
Explain Like I'm 5
You write a story for school. Before you hand it in, your twin reads it to look for mistakes.
Your twin is smart, but your twin learned spelling from the same teacher. The words you get
wrong are the words your twin gets wrong too. So those mistakes slip through.
A friend from another school spots them, because they learned differently. But sometimes the
friend says a right word is wrong. If you change it, your story gets worse.
So the best helper is someone who makes different mistakes than you do. And they should only
read the hard stories, where you make the most mistakes.
The Problem
You have shipped this bug already, in a system with no model in it.
Your service ran two replicas for safety. If one crashed, the other would take the traffic. Then
a leap second arrived. Both replicas crashed in the same minute, because both ran the same code
with the same bug. The backup was real, and it protected you against nothing that mattered.
Engineers call this a common-mode failure: one cause takes out every copy at once.
Redundancy only helps against failures the copies do not share.
A reviewer model is the same idea applied to an agent. One model, the drafter, writes a
plan or a patch. A second model reads it and lists concerns. The drafter then revises once.
Some coding tools now run this step for you. You can add it to your own agent in an afternoon.
It costs at least two extra model calls per task. Whether it pays depends on two things the
setup hides. The reviewer misses whatever blind spots it shares with the drafter. And when it
raises a false alarm on a correct draft, the revision can break it.
The fix: measure what the reviewer fixes and breaks on your drafter's own tasks, pick one
that fails differently, and call it only where the drafter often fails.
Read the full post on The AI Commit
How to Tell Whether a Second Model Reviewing Your Agent's Work Pays for Itself carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.
Top comments (0)