run the same AI reviewer on the same pull request twice and you'll often get different comments. people notice this and stop trusting the tool, which is fair.
it happens because the reviewer is a language model sampling an answer each time. some findings show up every run, the obvious ones. the borderline ones come and go. that's fine for suggestions, and a real problem if anyone treats "no comments" as a pass.
the fix that holds up is splitting the job. let the AI reviewer do what it's good at, pointing out things a person should look at. move anything that must hold into checks that give the same answer every time, a test, a lint rule, a semgrep rule, a CI step that fails the PR.
that split is also how we think about Tomosu. it gives each pull request a Production Reliability Index score against the whole repo, and every finding is a specific before and after code change you accept or reject, so you're reviewing a concrete change rather than a comment that might not show up next run.
whatever tool you use, the test is simple. rerun it on an unchanged PR and see what moves.
Top comments (0)