A 78-page job market paper landed last week that puts field numbers on the argument everyone has been having from vibes. Fiona Chen and James Stratton at Harvard wrote it using a proprietary dataset from the engineering analytics platform Jellyfish, and the abstract states the finding flatly: both AI assistants and AI agents raise coding productivity, and the gains do not fully pass through to software output or employment. For agents specifically, the reason is a bottleneck in code review.
I read the full paper rather than the coverage, because the methodology is the only reason to believe any of it.
What they measured: about 300 million work events across 718 firms and roughly 700,000 workers, from January 2021 through March 2026. Five sources merged into one panel. HR records, version control activity from GitHub, Jira issue management, Google Calendar, and generative AI usage telemetry for GitHub Copilot, Cursor and Claude Code. Identification is a staggered difference-in-differences design that exploits the fact that firms adopted these tools at different times. That is a real instrument. It is a long way from asking developers whether they felt faster.
Assistants and agents are not the same intervention
The paper separates the two and the split is the most useful thing in it.
AI assistants, the autocomplete-style tools, produced +12% lines of code, +9% commits and +5% pull requests. Only the commit effect was statistically significant. Assistants barely moved the review process at all: review length effect of -0.38 days against a baseline of 9.34 days, and the change-request and comment effects were small and insignificant.
AI agents are a different animal. They produced +30% lines of code, +20% commits and +23% pull requests, all significant. Then the review process reacted.
- Review length, measured as days from pull request opened to merged: +3.45 days, a 49% increase over the baseline of 7.03 days.
- Share of PRs with a formal change request: +0.12 against a baseline of 0.13. That is close to doubling.
- Review comments per PR: +0.58 against a baseline of 1.66, a 35% increase.
If you have been treating "AI coding tool" as one category in your rollout plan, the data says you are averaging two very different things. The assistant era was a mild throughput bump that review absorbed. The agent era is a much larger throughput bump that review visibly does not absorb.
The review regression isn't bigger diffs
The obvious objection to those review numbers is that PRs got bigger, so of course reviewers leave more comments. The authors checked. Mean PR size, measured in lines added, did not change significantly for agents (-67.99, s.e. 126.91 against a baseline of 1167.97) or for assistants.
So the review burden went up while the average diff stayed the same size. That kills the size explanation and points at the rate and the composition. More changes arriving, and more of them landing in the category that needs a human to actually reason about intent. There's a quote in the paper from a junior engineer that will sound familiar to anyone on a platform team: only one senior engineer was allowed to approve changes on a project, he could review once a week, and after some AI-generated commits caused outages the team added rules requiring senior approval for AI-generated code, which tightened the bottleneck further.
The output doesn't move
Here's the part that should end up in a slide. Resolution of Jira issues and epics, the paper's proxy for finished software, showed no statistically significant change after AI tool introduction. The authors also checked whether the composition of those issues shifted toward smaller or simpler work. It didn't. Employment moved neither up nor down in any significant way, including by seniority.
The model section explains why. Software production is two sequential stages: write draft code, then verify it before it ships. AI raises productivity in stage one and can also raise the bug rate. If verification capacity is fixed, the extra draft code gets spread across the same review time per unit, so a smaller share of it makes it through without additional expected losses from undetected bugs. The shortfall term is larger when detection has sharper diminishing returns and when review already eats a bigger share of cost. That is the formal version of a thing most teams have felt: you sped up the stage that wasn't the constraint.
Your AI reviewer is touching one PR in ten
This is the number I'd put on the slide. By March 2026, nearly 80% of firms in the sample had used AI tools somewhere in the code review process. Adoption is basically universal.
Usage is not. Only 23.3% of all review comments were generated by AI, and only 10.8% of pull requests received at least one AI-generated comment. So the review stage is where nearly everyone has a tool installed, and it is still where humans do almost all of the work.
Two readings. One is that AI review tooling is immature and the coverage will climb. The other is that teams are deliberately keeping humans on the read stage because the cost of shipping a bad change is high, which the paper suggests as well. Either way, if you are buying a reviewer on the promise that it fixes the review bottleneck, 10.8% coverage is the baseline you're starting from, and you should be measuring your own number instead of assuming a demo generalizes.
There's a note about whether AI reviewer comments are worth it at all in our earlier look at review-time claims, and the short version is that most of the cycle-time speedup people attribute to AI review comes from clearing the wait for a reviewer, which is real, and not from making the read faster, which mostly doesn't happen.
What to measure on your own repos
The paper's manager-facing advice is to stop trusting intermediate metrics. Lines of code and commit counts are exactly the stage AI accelerates. They tell you nothing about the constraint. Concretely, here's what I'd wire up:
- Split your rollout by tool class. Agents and assistants belong in separate cohorts, because they produce different review load. If your dashboard lumps them together you will see an average that describes neither.
- Track review length, change-request share and comments per PR, with mean PR size as a control. If review burden rises and diff size is flat, you're seeing the bottleneck the paper describes, not a size artifact.
- Count coverage, meaning the share of your PRs your AI reviewer actually comments on. That's the number vendors don't put on the pricing page, and it's the one that determines whether the tool is doing anything.
- Keep an output measure in the loop. Whatever "finished work" means for you, issues closed, deploys, features resolved, watch it while you scale agents. If it's flat, you have a capacity problem and buying more code generation makes it worse.
For the coverage measurement in step three, the comparison set matters. CodeRabbit, Greptile, Qodo, Kodus and the rest all demo well on a single prepared diff. The thing that separates them is what fraction of your real PR stream they touch without flooding reviewers, and how they route the diffs that genuinely need a human. That's a measurable property and you can build a small eval set for it from PRs you already reviewed by hand. Our 9-pillar directory standard scores tools on public documentation, and it will not tell you your coverage number. Only your own telemetry will.
The uncomfortable reading
The firms in this sample spent real money on agents, got 30% more code for it, and did not get measurably more software out the other end. That isn't a claim that agents are useless. It's a claim that the constraint moved, and the org chart and the metrics didn't.
The Ars Technica writeup on the same study frames it as a double-edged sword. I think the more useful framing is that the constraint was always downstream and now you can see it in the data. The paper's own conclusion is that the returns to AI require evaluating the whole pipeline, and that realizing the benefit means moving labor and attention to the stage that just became the bottleneck. Jellyfish published a summary of the research as well, which is worth a skim for how the vendor side is reading its own numbers.
For anyone who runs evals on this stuff, the takeaway is to stop evaluating the generator and start measuring the pipeline. Generation is the part that got fast. The part that decides whether the code ever ships is the part you should be putting under test.
Top comments (0)