The Reviewer Is the Untested Component: A Three-Layer Gate for AI Refactors

This week's DEV feeds keep asking what humans do while AI codes. The sharper question is about the review layer. A parallel discussion asks whether the reviewer was ever tested. AI writes patches; developers review them. Nobody reviews the review.

A refactor is the hardest review case. A new feature can be smoke-tested by running it. A refactor must prove that before and after behave identically. Characterization probes are the proof. Dead probes are the silent failure.

This article builds a three-layer gate for AI-proposed refactors. It includes a runnable Python artifact and a decision table. Core claim: test the review before you trust the refactor.

Layer 1: Pin behavior with probes

Characterization tests record what the code does, not what it should do. Run the legacy code. Capture the outputs. Assert them exactly.

Three rules:

1. One probe per observable behavior.
2. Boundary values get dedicated probes.
3. Never assert desired behavior. Assert recorded behavior.

The model that proposes the refactor should not write the probes alone. A second pass from a fresh context reduces correlated blind spots. Then verify the probes. Writing them is not enough.

Layer 2: Kill one line

A probe that cannot fail is dead. Dead probes pass. Dead probes protect nothing.

Mutation checking finds dead probes. Change one implementation line. Run the suite. A surviving mutant means the probes cannot detect that change. That line is unprotected.

Consider the condition cart_total > 100. Flip it to cart_total >= 100. No probe feeds it exactly 100. The mutant survives. This is how AI refactors silently break boundaries.

Layer 3: Cap the diff

Small changes stay reviewable. Large changes hide behavior drift. Set a diff budget before the refactor starts, not after it lands.

Normalize formatting first. Whitespace is noise. Then count changed lines. Exceeding the budget means splitting the refactor. One transformation per commit.

The artifact: a reference gate

The demo uses a small legacy module with a hidden boundary. First, the legacy code:

TAX_RATE = 0.08

def price(cart_total, promo_code=None):
    if cart_total is None or cart_total < 0:
        return None
    if promo_code == "SAVE10":
        cart_total = cart_total * 0.9
    elif promo_code and len(promo_code) > 5:
        cart_total = cart_total * 0.95
    if cart_total > 100:
        cart_total = cart_total - 10
    return round(cart_total * (1 + TAX_RATE), 2)

Next, the characterization probes:

from legacy import price

def test_none_input():
    assert price(None, None) is None

def test_negative_input():
    assert price(-5, None) is None

def test_plain():
    assert price(50, None) == 54.0

def test_save10():
    assert price(100, "SAVE10") == 97.2

def test_long_code():
    assert price(100, "ABCDEF") == 102.6

def test_volume_discount():
    assert price(150, None) == 151.2

Now the mutation gate. It runs the suite, applies one mutation, and reports survivors:

import subprocess
import sys
from pathlib import Path

MUTATIONS = [
    ("legacy.py", "if cart_total > 100:", "if cart_total >= 100:"),
]

def suite_passes():
    result = subprocess.run(
        [sys.executable, "-m", "pytest", "-q"],
        capture_output=True, text=True,
    )
    return result.returncode == 0

for path, old, new in MUTATIONS:
    source = Path(path).read_text()
    assert source.count(old) == 1, "mutation target is ambiguous"
    Path(path).write_text(source.replace(old, new))
    killed = not suite_passes()
    mutated = Path(path).read_text()
    Path(path).write_text(mutated.replace(new, old))
    print(f"mutation {'KILLED' if killed else 'SURVIVED'}: {old!r} -> {new!r}")

Run the gate against the current suite. The mutant survives. No probe exercises cart_total == 100. Any refactor can flip that boundary silently. The gate blocks the refactor before it is ever applied.

An AI model proposes a cleaner version. It extracts the discount logic:

def _discount(total, code):
    if code == "SAVE10":
        return total * 0.9
    if code and len(code) > 5:
        return total * 0.95
    return total

def price(cart_total, promo_code=None):
    if cart_total is None or cart_total < 0:
        return None
    total = _discount(cart_total, promo_code)
    if total > 100:
        total -= 10
    return round(total * (1 + TAX_RATE), 2)

The full suite passes. A normal review approves. The gate refuses. One boundary probe is missing:

def test_volume_discount_boundary():
    assert price(100, None) == 108.0

Add the probe. Rerun the mutation check. The mutant dies. Now the refactor is safe to review.

The diff budget closes the loop:

BUDGET = 12

def changed_lines(a, b):
    return len(set(a.splitlines()) ^ set(b.splitlines()))

before = Path("legacy.py").read_text()
after = Path("legacy_refactored.py").read_text()
cost = changed_lines(before, after)
print(f"diff cost: {cost} lines (budget {BUDGET})")
assert cost <= BUDGET, "split the refactor"

This count is an approximation. It ignores line order and duplication. Use difflib for a strict gate. The point is the budget, not the counter.

The decision table

Probes pass	Mutants killed	Diff within budget	Verdict
yes	yes	yes	Approve. Read the diff, then merge.
yes	no	yes	Add boundary probes. Rerun the gate.
yes	yes	no	Split the refactor into smaller commits.
no	—	—	Reject. Behavior changed. Revert.

Four rows. Four actions. No ambiguity.

Run the gate on every proposal

One gate run is a demo. A gate run per change is a process. Scheduled runs need a server, not a laptop. The script is plain Python with pytest. It runs anywhere CI runs.

This is where MonkeyCode's free model access and free server option enter the workflow. The free model can draft the first probe batch. The free server can run the scheduled gate. MonkeyCode is an open-source project; at the time of writing (August 2026), its README states a 10-million-token free allotment and a free server option. Verify the current numbers before building on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

Characterization tests pin bugs as features. If the legacy behavior is wrong, this gate locks it in.

Mutation checking is slow on large suites. Sample high-risk lines instead of exhausting every mutant. Mutation targets must be unambiguous; the gate enforces that.

Diff budgets count lines, not semantics. A one-line behavioral change can outweigh a forty-line rename. Pair the budget with a human diff read.

Free quotas change. The 10-million-token figure is a point-in-time claim, not a contract. Check the README. Assume nothing is permanent.

Skip this gate for greenfield code. There is no legacy behavior to pin. Skip it for mid-rewrite modules where behavior is changing on purpose. Skip it for throwaway scripts where the gate costs more than the change.

Nobody reviews the review

The model is not the riskiest part of an AI refactor. The reviewer is. This gate tests the review: probes pin behavior, mutations expose dead probes, and a diff budget keeps changes small.

Run the gate before your next AI refactor. Let the surviving mutant surprise CI, not production. If you need a free model for the first probe draft and a free server for the gate, MonkeyCode's README describes both — check the current terms, then try the workflow.