DEV Community

Mizoko
Mizoko

Posted on

I Automated My Claude Code Review. Then 46% of Its Comments Got Ignored.

TL;DR

  • Of the three Claude review skills I built over six months, the only one still running is the one wired into a scheduled, automated flow. Every skill that waited to be invoked by hand fell out of use.
  • Automating had a side effect. The share of review-pr comments that got no reaction at all rose from 20% in the manual period to 46% in the automated one.
  • My working hypothesis, not proven by the data: in the automated runs, the context — the existing comments and the discussion on the PR — is not being passed in properly, so runs can force out comments or duplicate ones that already exist. Whether the asynchronous execution is part of the cause is still uninvestigated; that is what I will look into next.

How the manual skills ended

The last time I invoked deep-review by hand was August 5, 2026. In the usage logs from September 29 to November 13, it appears zero times.

I let it go because its fix rate was the lowest, both in feel and in the recount. Its wrong-comment rate was 9.8%, level with review-pr at 9.6%. It was not wrong more often. Its comments just did not get fixed.

Maintenance did not stop. Even after August 5, 17 commits went into the deep-review skills. I kept improving something nobody was invoking anymore.

What survived

What survived was not a skill but a flow. Since October 16, an automated review flow runs on weekday slots: 116 runs out of 122 scheduled. Inside it, review-pr ran 23 times and a recheck skill 49 times, with no human deciding whether to invoke them.

The pattern was consistent. Skills that had to be called died. The one embedded in an existing routine kept running.

The side effect: comments nobody reacts to

Counting all review-pr comments, the share with no reaction (no reply, no fix) was 20% in the manual period, April to May (34 of 166), and 46% in the automated period, October to November (52 of 114). These counts include comments on PRs that were never merged.

The data does not say why. My guess is that the context is not handed over properly in the automated runs: a run does not take the existing comments and the discussion so far into account, so when the flow runs multiple times against the same PR, it forces some comments out or duplicates what is already there. The asynchronous way these runs execute may be part of it, but I have not investigated that yet.

What I changed and what is left

I settled on keeping one skill, running on a schedule, inside the flow. Automation itself was the right call. What is left are two open problems. The first is context: the automated runs need to receive the existing comments and the discussion properly, and why that is not happening is still uninvestigated — the asynchronous execution may be involved. The second is measurement: my monthly recount tracks the fix rate and the wrong-comment rate, but not whether anyone reacted. That is the number I need to add.

Takeaway

  • Do not build skills that wait to be invoked. What survives is what gets embedded into an existing flow.
  • When you automate output, measure the receiving side too, not just what the system produces.

Top comments (2)

Collapse
 
parsam profile image
Parsa • • Edited

your context theory is easy to test with what you already have. look at the ignored comments and see how many are near copies of an earlier comment on the same PR. if it’s most of them, it’s the handoff. if not, people may just react less to feedback they didn’t ask for. at tomosu we show each finding as a before and after change you accept or reject, so every one gets a yes or no you can count

Collapse
 
sweety717 profile image
JavaCoder7 •

I think there's an important distinction between automating the review process and making the review context-aware.

One thing I'd investigate is whether each new review run knows which findings were already reported, dismissed or addressed in previous runs.

A useful approach could be to maintain finding fingerprints across commits, then revalidate existing findings against the latest diff instead of generating them from scratch every time.

That could help distinguish genuinely new issues from repeated feedback and prevent developers from losing trust in automated reviews.

Do you currently preserve finding identity across reruns, or does each execution treat the PR as a fresh review?