When a team asks how to cut pull request review time with AI, the answer they get is usually a vendor number. Atlassian says Rovo cut PR cycle time by 45% in their dogfooding. That number is real, but it's self-attested: no harness, no fixed window, no control group, nothing you can reproduce against your own repo. That doesn't make the claim a lie. It makes it useless as a forecast for your team.
The fix isn't a better pitch. It's measuring your own before-and-after with a protocol that doesn't let the mean hide the real story.
Here's the trap most teams fall into. They roll out an AI reviewer, watch average cycle time drop, and call it done. But a mean drops when the tool eats the easy PRs and the hard ones keep sitting. A 30% drop in mean cycle time can be the tool clearing the 90-line single-file PRs while the 2,000-line cross-module changes got slower. The aggregate looks great on the dashboard and the regressions live in the tail the whole time.
So before you adopt anything, fix the measurement.
Pick a fixed slice to compare against itself. Two weeks of PRs from the same team before the tool, then the same two weeks with it. Keep the reviewer pool unchanged across both. Don't compare a busy team's fall to a different team's spring.
Split the comparison by size and risk, not just total. Separate small low-risk changes from large or boundary-crossing ones. AI review tools are cheap to the point of being free on a one-file bugfix and genuinely slow when context spans modules. If your team lives in the large changes, a tool that only speeds up the small ones hasn't bought you anything.
Measure the tail, not just the mean. Track p90 and p95 cycle time, and review turnaround per change size. A mean can improve while the worst PRs get worse.
One more signal that usually goes unflagged: reviewer load. A tool that hands every reviewer forty new inline comments a day hasn't reduced review time, it has moved it from merge latency to reading time. If your reviewers spend more time unscrolling AI nitpicks than they used to spend scrolling diffs, the cycle-time number is lying to you.
And keep the attribution clean. Time-to-merge catches everything upstream of the button: the tool, the linter, the CI queue, and whether your senior reviewer happens to be on vacation. A drop that lines up with a CI speedup isn't evidence for the AI anything. When you see a delta, ask what changed at the same moment before you credit the tool.
The 45% number is a floor, not a promise. It's what one team saw on their repo, their rules, their context. Yours will be different, and the only way to know by how much is to run the fixed window, split it by size and risk, and read the tail alongside the average.
Top comments (1)
The measurement fix is right, and there is one confounder it does not cover that tends to break the before and after comparison in exactly the teams most likely to run it.
Holding the reviewer pool fixed is half of it. The other half is the authoring process, which is usually the thing being changed in the same quarter. If a team adopts an AI reviewer while it is also writing more of its code with an assistant, window B does not contain the same population of pull requests as window A. A 200 line change somebody wrote by hand and a 200 line change that came out of an agent in one pass are different review objects carrying the same label, so the size and risk buckets end up comparing things that only look alike. Either freeze the authoring side across both windows, which nobody will agree to, or record provenance per pull request and carry it as a covariate rather than letting it sit in the noise.
Second, on the window itself. If you are reading p90 and p95, two weeks probably cannot carry them. A team merging four or five pull requests a day gives you roughly fifty in the window, so p95 is two pull requests and it moves if one reviewer takes a long weekend. The tail is the right place to look. It needs either a longer window or a count rather than a percentile, something like the share of changes still open after three days, which is stable at that sample size and is easier to defend to whoever asked for the number.
The measure I would add sits underneath all of them: whether review changed anything. Track the share of pull requests where review produced an edit to the diff, and separate behavioural edits from cosmetic ones. If cycle time falls and that share falls with it, review did not get faster, it got shallower, and the cycle time number will keep improving all the way to rubber stamping. Your reviewer load signal has a cheap version of the same idea, since comments that led to an edit against comments resolved with no edit comes out of the same API data and is a fair signal to noise measure for the tool.
That is the one I would put in front of a sceptical manager, because faster is easy to sell and it is the only number that separates the two ways of getting there.