Review Comments Are Hypotheses: An Audit of an AI Code Review

The conclusion first: a free model reviewed a 120-line C++ patch and produced 14 comments. I applied each comment as a patch and ran the test suite. Eight comments were correct. Two broke the build. One made the benchmark slower. Two changed nothing. One could not be turned into a patch at all. The model's confidence was not correlated with correctness.

A code review comment is a hypothesis about the code. The cheapest way to test a hypothesis about code is to apply it and run the tests. Reading a review and nodding is pattern-matching. The audit I describe below is measurement.

Background: a patch that needed a second pair of eyes

The patch was a small optimization to a config_loader class. The original parser used std::istringstream to split each line:

std::vector<std::pair<std::string, std::string>> parse(std::istream& in) {
    std::vector<std::pair<std::string, std::string>> out;
    std::string line;
    while (std::getline(in, line)) {
        std::istringstream ss(line);
        std::string key, value;
        std::getline(ss, key, '=');
        std::getline(ss, value);
        out.emplace_back(std::move(key), std::move(value));
    }
    return out;
}

The replacement scanned the string manually with find and substr:

auto eq = line.find('=');
if (eq == std::string::npos) continue;
out.emplace_back(line.substr(0, eq), line.substr(eq + 1));

About 120 lines changed. The unit tests passed. The micro-benchmark showed a 2.3x speedup on a 10,000-line config file.

The patch was fine. The question was whether a free model's review of it was fine too.

The method: every comment is an experiment

I sent the diff and the surrounding file to a free model via MonkeyCode's API and asked for specific, actionable review comments. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model returned 14 comments. Each one went through three steps.

Step 1: turn the comment into a patch. A comment is only actionable if it can be expressed as a diff. "This is not thread-safe" is not a patch. "Take std::string_view instead of std::istream&" is a patch. Comments that could not become a diff went into the unverifiable bucket.

Step 2: apply the patch to a clean checkout. Each comment became its own branch. The test suite and the micro-benchmark ran against that branch.

Step 3: record the outcome. Five outcomes: passes (tests green, benchmark not worse), fails_tests (build or tests broke), fails_benchmark (tests green, benchmark regressed), no_op (nothing observable changed), and unverifiable.

The artifact: review_audit.sh

The audit script is deliberately dumb. It loops over a directory of patch files, applies each one to a clean checkout, runs the test command, and writes a CSV.

#!/usr/bin/env bash
# review_audit.sh <repo> <patches-dir> <test-cmd> <bench-cmd>
set -euo pipefail

REPO_DIR="$1"
PATCHES_DIR="$2"
TEST_CMD="$3"
BENCH_CMD="$4"
RESULTS="audit_results.csv"

echo "comment_id,status,notes" > "$RESULTS"

for patch in "$PATCHES_DIR"/*.patch; do
    id="$(basename "$patch" .patch)"
    git -C "$REPO_DIR" checkout -- .
    git -C "$REPO_DIR" clean -fdq
    if ! git -C "$REPO_DIR" apply --check "$patch" 2>/dev/null; then
        echo "$id,does_not_apply,"
        continue
    fi
    git -C "$REPO_DIR" apply "$patch"
    if ! (cd "$REPO_DIR" && eval "$TEST_CMD" >/tmp/audit_test.txt 2>&1); then
        echo "$id,fails_tests,$(tail -1 /tmp/audit_test.txt)"
        continue
    fi
    if ! (cd "$REPO_DIR" && eval "$BENCH_CMD" >/tmp/audit_bench.txt 2>&1); then
        echo "$id,fails_benchmark,$(tail -1 /tmp/audit_bench.txt)"
        continue
    fi
    echo "$id,passes,"
done >> "$RESULTS"

column -t -s, "$RESULTS"

The BENCH_CMD in this run was a small wrapper that compared the patched binary against the baseline binary and exited non-zero if the median regressed by more than 2%. The script automates the expensive part: build, test, and benchmark for every comment. The cheap part — reading each passes diff to separate a real fix from a cosmetic rename — stays with the human.

What the audit found

The 14 comments broke down like this:

outcome	count	example
passes	8	"You don't handle \r\n line endings"
fails_tests	2	"Take std::string_view instead of std::istream&"
fails_benchmark	1	"Reserve 1024 entries before the loop"
no_op	2	"Rename parse_key_value to parse_pair"
unverifiable	1	"Consider a state machine instead"

After the script finished, I read each passes diff. Two were pure renames with no behavioral change, so I reclassified them as no_op. The script's job is to filter out failures cheaply. The human's job is to read the survivors.

The three failures were the interesting ones.

Failure 1: the string_view comment. The model claimed parse() should take std::string_view because "callers already have the config text in memory." They do not. The callers open a file and pass an std::ifstream. The patch changed the signature, and every call site in the test suite stopped compiling.

Failure 2: the const overload comment. The model suggested adding an overload for const std::istream&. It does not compile. std::getline needs a mutable stream, and a const std::istream& cannot bind to the non-const getline overloads. The model forgot how the standard library works.

Failure 3: the reserve comment. The model suggested out.reserve(1024) before the loop because "config files can be large." The patch applied cleanly. The tests passed. The benchmark got 18% slower: reserving 1024 entries for a 47-line test config allocated more than the vector ever needed. It passed the test suite and failed the performance check.

That last one is the most important lesson. A comment can pass every test and still be wrong. The audit is only as good as the suite it runs — which is why the benchmark is part of the audit, not an afterthought.

What the wrong comments had in common

The model did not misread the diff. It invented a context around the diff. It assumed callers held strings in memory. It assumed a const stream was a meaningful type. It assumed config files were large. Each assumption was plausible. Each assumption was false.

The correct comments were different. They pointed at concrete lines and concrete inputs: \r\n handling, the double copy in parse_key_value, the missing trim of trailing whitespace. They were grounded in the code that exists, not the code that might exist.

Confidence was not a signal. The string_view comment was the most confident one in the review. It was also the most wrong.

Why the audit beats reading the review

Reading a review comment is fast. Too fast. A confident wrong comment feels correct until you run it, and by then you have already spent the mental energy integrating it into your model of the code. The audit moves that cost out of your head and into the build system.

The free server made this cheap. Fourteen comments meant fourteen clean checkouts, fourteen builds, fourteen test runs, fourteen benchmarks. The whole loop ran unattended on MonkeyCode's free server while I did something else. The CSV was waiting when I came back.

Limitations: who should not use this

The audit measures what the test suite and benchmark measure. If the suite is weak, the audit is weak. A comment that fixes a bug the tests do not cover will be marked passes, and you will be none the wiser.

The audit cannot judge architecture. "Consider a state machine instead" is unverifiable by construction. That does not make it wrong. It makes it a conversation, not a claim.

Do not use this workflow for security-sensitive code without a human reading every fails_tests result. A broken build is not proof the comment is bad. It is proof the comment conflicts with the current code. Sometimes the comment is right and the code is wrong.

The takeaway

The next time a model reviews your code, ask it to submit a patch instead of a paragraph. Then run the patch. The test suite is a better judge of a review comment than the reviewer who wrote it — especially when the reviewer is very confident.