DEV Community

Mizoko
Mizoko

Posted on

I Tested Three Claude Code Review Skills for Six Months. The Simplest One Won.

TL;DR

  • I recounted 1,306 review comments from three Claude code review skills over six months, using one fixed definition of "fixed."
  • Accuracy was the same across all three: about 10% of the comments were wrong in each. What differed was the fix rate. The simplest skill reached 68.9%, the most elaborate one 59.2%.
  • Counting comments turned out to be the wrong metric. What matters is how critical each comment is.

The three skills

Over six months I built three skills that review pull requests for a mobile app and a backend. review-pr is the basic one: it comments line by line on a PR. deep-review is the elaborate one: it runs sub-agents in parallel, one per viewpoint, and the viewpoints grew from 11 to 14. nitpick-pr only posts minor, nit-level comments.

How I counted

I fixed the definitions first and did not change them midway. A comment counts as "fixed" when the code actually changed in response to it, judged from the replies and the commits that followed. The scope is merged PRs where the PR author replied to the comments: 1,306 of the 1,691 comments in the period (April 1 to November 13, 2026). I could not attribute 36% of the comments to a specific skill, and that caveat applies to every per-skill number below.

Results

Skill Comments Per PR Fixed Fix rate Wrong comments
review-pr 354 4.4 244 68.9% 9.6%
nitpick-pr 114 2.8 76 66.7% 9.6%
deep-review 368 7.8 218 59.2% 9.8%
Overall (incl. unattributed) 1,306 9.7 789 60.4% 11.2%

The most elaborate skill had the lowest fix rate. The wrong-comment rate was about 10% for all three, so accuracy does not explain the gap. The result also matched my gut feeling: in day-to-day work, review-pr had always seemed to leave the most useful comments. The surprise was not the ranking, but that the feeling held up under a recount.

Why the elaborate one lost

Splitting the not-fixed comments by reason shows where the gap came from. In deep-review, "correct but low priority" and "out of scope for this PR" together made up 17.4% of its comments. In review-pr the same total was 6.2%, and in nitpick-pr 16.7%. deep-review also posted 7.8 comments per PR against review-pr's 4.4, roughly 1.8 times as many. A correct comment still gets ignored when it does not belong to that PR, and volume turns the rest into noise.

A month where the simple one slipped

In August, review-pr's wrong comments spiked: 15 in total, and 8 of them sat in just two PRs. In one mobile PR, 3 pointed at code that an older commit had already fixed, and 2 flagged behavior that was intentional. In one backend PR, 3 were based on a wrong precedent and were withdrawn. From September on, the rate settled back to between 0% and 5.6%. I have no record of what changed, so I will not guess at a cause.

I stopped the nitpick skill despite decent numbers

nitpick-pr had a 66.7% fix rate, second place. But its comments were formatting-level, and it cannot detect critical problems. I stopped running it entirely. A good fix rate and being worth running turned out to be different things.

Limits of these numbers

Three caveats. The scope (merged PRs with replies) pushes the rates higher than an all-comments count would. The "fixed" judgment has noise: in a spot check of 8 comments, 2 were changes unrelated to the comment. And 36% of the comments could not be attributed to a skill, especially in August.

Takeaway

What survived six months is the simplest skill, running on a schedule. The elaborate one and the nitpick one are both retired. When you compare review skills, the wrong-comment rate alone will not separate them. Look at the fix rate and the number of comments per PR together. And stop counting comments as the goal: the next thing to measure is how critical each comment is. I kept the per-comment data and the counting script, and I rerun the same count every month.

Top comments (2)

Collapse
 
nilesharma0108 profile image
Nilesh Sharma •

Counting fix rate alongside false positives is a better signal than raw finding volume. I’d also separate outcomes by severity so a low-value nit does not look equivalent to a correctness or security issue. Could tracking reviewer overrides or reverted fixes help estimate the long-term value of a comment?

Collapse
 
izgorodin profile image
Edward Izgorodin •

Fix rate and wrong-comment rate both look at the comments that were posted. The question you end on, how critical each comment is, has a second half that neither column can see: the problems none of the three skills commented on. A skill can post fewer, better comments and still miss the same bugs as the others, and nothing in the table would show it.

Your data may already hold a rough way to count that side. For the merged PRs in the period, take the later commits that fix or revert lines those PRs changed, say within 30 days, and check whether any review comment touched those lines. A fix with no comment on its lines is a miss. Split by skill, that gives a number to set against the fix rate, and it answers the criticality question from the other direction: a comment that sits on lines that later needed a hotfix was critical, whether or not the author took it at the time.

The 36% of comments without attribution would limit this the same way it limits the current table. And the 2 of 8 unrelated changes in your spot check suggest that the fixed side and the miss side share one hazard: a commit near a comment is not always a commit because of it.