Whether you’re trying to build a software factory or already running one, you’ve probably heard some version of the pitch: agents can write the code now, and humans are the bottleneck. Get people out of the loop and let the agents cook.
And for most teams, review is exactly where things get stuck. At the top 10% of companies, agents are now writing a majority of merged PRs, but a person still signs off on the vast majority of them before they merge.
If people are the bottleneck, it’s fair to ask what all that human attention is actually buying us. Does it make agent code better?
At Jellyfish we analyze AI telemetry and engineering workflow data across 300,000 developers at 1,300 companies. When we looked at agent-written PRs that merged and checked how often they got reverted within a month, the answer depended on how involved a person had been along the way:
- Nobody touched it: 0.5% reverted
- A person approved it: 1.2%
- A person stepped in (left comments, asked for changes, pushed a commit): 2.2%
The more a human got involved, the more often the code got rolled back, roughly doubling at each step. So are humans making agent code worse?
Well… not quite. What you’re actually seeing is the effect of triage, powered by good human engineering judgment.
Engineers are good at spotting risk, so they wave the easy PRs through, approve the middling ones, and roll up their sleeves on the perilous ones. And since hard things are hard, the riskiest PRs get reverted much more often, even with a person helping. (These gaps hold up when you compare PRs of the same size within the same company, so this isn’t just true for big PRs or messy orgs.)
The triage is working, too. When we compare agent PRs that nobody reviewed against similar ones that were reviewed, there’s no statistically meaningful difference in reverts, follow-up fixes, or bugs. For the work they’re choosing to let through, engineers are holding the line on quality.
Holding the line, by hand
The problem is that at most orgs, engineers are indeed holding the line, but at great effort and expense. This triage that keeps quality in check is maintained by hand, one PR at a time, at the very end of the process (which is the most expensive place to do it). Since early 2025, the typical developer is doing 26% more code reviews per week, and essentially all of that growth is work a coding agent helped write. Review has become a second job stacked on top of the first one.
You might expect AI review to pick up the slack, but so far it hasn’t. When a repo switches on an AI reviewer, people still do about two reviews per PR, and it still takes about three hours to merge. The bot is typically one more voice on the PR, not a decision that offloads work from a human.
Move the judgment up front
The good news is that the hard part, knowing which changes are risky, already exists. It just lives in your engineers’ heads. So get it out:
- Write down the triage. What do your engineers wave through, and what do they always dig into?
- Move it up front. Put that risk call into the ticket or spec before the agent starts, instead of discovering it in review after the agent is done.
- Turn that judgment into checks, one kind of work at a time. Run a check behind a person, track how often the person overrules it, and when that gets close to zero, let the check decide for that kind of work.
The fastest way past the review bottleneck probably isn’t fewer humans or more bots. It’s getting the judgment your engineers already have out of their heads and into the process.
Top comments (0)