Mayur Agnani

Posted on Oct 30 Edited on Oct 30 Originally published at theaicommit.com

How to Tell Whether a Second Model Reviewing Your Agent's Work Pays for Itself

#ai #testing #productivity #agents

Engineer's view. This is running two replicas of the same software for safety. Both crash on the same leap second, because they share the bug. A reviewer model trained like the drafter misses the same failures. Pick a reviewer that fails differently, and measure what it catches on your drafter's failures, not on its own.

TL;DR. A reviewer model can only fix errors it would not have made itself, and models share many blind spots. Measure what it fixes and what it breaks on your own tasks, then review only where the drafter often fails.

Explain Like I'm 5

You write a story for school. Before you hand it in, your twin reads it to look for mistakes.

Your twin is smart, but your twin learned spelling from the same teacher. The words you get
wrong are the words your twin gets wrong too. So those mistakes slip through.

A friend from another school spots them, because they learned differently. But sometimes the
friend says a right word is wrong. If you change it, your story gets worse.

So the best helper is someone who makes different mistakes than you do. And they should only
read the hard stories, where you make the most mistakes.

The Problem

You have shipped this bug already, in a system with no model in it.

Your service ran two replicas for safety. If one crashed, the other would take the traffic. Then
a leap second arrived. Both replicas crashed in the same minute, because both ran the same code
with the same bug. The backup was real, and it protected you against nothing that mattered.
Engineers call this a common-mode failure: one cause takes out every copy at once.

Redundancy only helps against failures the copies do not share.

A reviewer model is the same idea applied to an agent. One model, the drafter, writes a
plan or a patch. A second model reads it and lists concerns. The drafter then revises once.
Some coding tools now run this step for you. You can add it to your own agent in an afternoon.

It costs at least two extra model calls per task. Whether it pays depends on two things the
setup hides. The reviewer misses whatever blind spots it shares with the drafter. And when it
raises a false alarm on a correct draft, the revision can break it.

The fix: measure what the reviewer fixes and breaks on your drafter's own tasks, pick one
that fails differently, and call it only where the drafter often fails.

The Fix: Measure Fixed and Broken, Then Review Only the Hard Tasks

What does one review pass add?

Count the four outcomes above. The gain is the fixed tasks minus the broken ones. As rates:

net = fail × catch × fix − (1 − fail) × false alarm × break

Here fail is how often the drafter alone gets it wrong. Catch is how often the reviewer flags
those failures. Fix is how often the revision then gets it right.

Why isn't the reviewer's own accuracy enough?

Because the catch rate is measured on the drafter's failures, and those are the hard cases. A
study of over 350 models found that when two models both get a question wrong, they pick the
same wrong answer about 60% of the time. Same developer, same architecture and similar size all
raised that overlap.

When does review hurt?

On easy tasks. If the drafter fails 4% of the time, there is little to fix. A 10% false-alarm
rate with a 25% break rate then costs more than review gains.

What This Means for You

When this matters. Your agent writes code, plans or answers, and you are adding a second model
to check them. Or you turned on a tool's built-in critic and it now costs more. It matters most
on long tasks, where one missed error ruins the run. GitHub found Rubber Duck helped most on
problems spanning three or more files that take 70 or more steps.

How it affects you. A reviewer from the drafter's own family feels safe and catches less than
its benchmark score suggests. A reviewer on every task pays two extra calls on work that was
already right. Its false alarms then rewrite correct code. You will not see either in a demo,
because demos show the fixed case.

What to do about it. Start with no new code. Take 20 tasks your agent failed last month.
Paste each failing draft into a reviewer from another model family and ask for its top three
concerns. Count how many name the real bug. That is a first estimate of catch.

Then run the full check. Run a few hundred tasks twice from the same start, alone and with
review. Count fixed and broken. Split them by a signal you have before review runs, such as
files touched. Turn review on only for the buckets where fixed clearly beats broken.

Implementing It

The change.

Three people touch this. The second one decides whether it ships.

Role 1, whoever writes the agent loop. Add one review pass. Give the reviewer the task and the
draft only, with no tools and no write access. HydraFusion does the same: its critic is
read-only, runs in a tool-less context, and comes from a different model family. Then let the
drafter revise once.

REVIEW = ("You are reviewing another engineer's change. List at most three concerns, "
          "most important first: bugs, wrong assumptions, missed edge cases. "
          "If you find nothing serious, reply exactly: LGTM.")

def draft_with_review(task, drafter, reviewer):
    draft = drafter.run(task)                        # full agent loop, with tools
    notes = reviewer.complete(system=REVIEW,          # one call, no tools
                              user=f"Task:\n{task}\n\nChange:\n{draft.diff}")
    if notes.strip() == "LGTM":
        return draft, {"flagged": False}
    revised = drafter.run(task, extra=f"A reviewer raised:\n{notes}\n"
                                      "Fix what is real. Ignore what is wrong.")
    return revised, {"flagged": True, "notes": notes}

The "ignore what is wrong" line matters. It is your only defense against the broken case.

Role 2, whoever owns evals. Run paired trials. Every task runs twice from the same commit:
drafter alone, then drafter with review. Grade both with the task's own tests. Only the tasks
whose outcome changed carry evidence, so test those with McNemar's test.

fixed  = sum(1 for r in runs if not r.alone_ok and r.reviewed_ok)
broken = sum(1 for r in runs if r.alone_ok and not r.reviewed_ok)
helps  = fixed - broken > 2 * math.sqrt(fixed + broken)   # about 95% confidence

In the demo, 400 tasks give 27 fixed and 8 broken overall. Split by files touched, one-file tasks
give 2 fixed and 0 broken, too few to trust. Two-file tasks give 4 and 6. Tasks with three or more
files give 21 and 2, the only bucket that clears the test.

Role 3, whoever owns the routing. Gate review on that signal. Files touched is the one GitHub
reports. Plan length, step count or a past failure on the same module also work.

def needs_review(task, plan):
    return len(plan.files) >= 3 or plan.steps >= 70

If you drive Copilot CLI, Rubber Duck is behind /experimental and pairs a Claude drafter with a
GPT reviewer. The same paired check tells you whether it earns its cost on your repo.

How you know it worked.

Rerun the paired trials a week later on fresh tasks. Fixed minus broken should still clear the
test in the buckets you review. Your success rate should rise while total model spend rises
less than the review share would suggest. Log every false alarm. If broken climbs, tighten the
prompt before you widen the gate.

When a Reviewer Model Is the Wrong Tool

When the task has a test that runs. A failing test is a reviewer with no blind spots and no
false alarms. Run it, feed back the failure, and save the second model for what tests miss.

When the drafter rarely fails. On easy work there is little to catch. A few false alarms then
cost more than review gains, and every call costs the same.

When you can only get a reviewer from the drafter's family. It still catches some errors, but
fewer than its score suggests. A stronger single model may cost less than two weaker ones.

Latency is real too. HydraFusion holds the draft until review and revision finish. A reader
waiting on the answer feels every extra call.

Three questions before you add one:

What share of your agent's failed tasks does this reviewer actually flag?
How often does it flag a draft that was already right?
Which signal, known before review, separates the tasks where it helps?

Glossary

common-mode failure — one cause that takes out every redundant copy at once, because they share it.
reviewer model — a second model that reads another model's work and lists concerns, without changing it.
drafter — the model that does the task and revises after review.
catch — the share of the drafter's failures that the reviewer flags.
false alarm — the reviewer flagging a draft that was already correct.
McNemar's test — a check on paired runs that uses only the tasks whose outcome changed.

DEV Community