DEV Community

fable7
fable7

Posted on

AI Coding Agents Write 30% More Code. Shipped Software Doesn't Move.

#ai #programming #productivity #discuss

Here is the number that should be on every engineering leader's dashboard right now:

+30% code. 0% more shipped software.

That's the headline from a Harvard working paper by Fiona Chen and James Stratton, Artificial Intelligence in the Firm: Bottlenecks in Software Production, which Ars Technica surfaced this month. The study is built on engineering telemetry, not surveys: ~300 million work events from 718 firms and roughly 700,000 engineers, January 2021 through March 2026, using a staggered difference-in-differences design comparing firms before and after they adopted coding agents.

What actually moved

Metric Change after agent adoption
Lines of code per worker ~+30% (≈1,500 extra lines/worker-month)
Commits +20%
Pull requests +23%
Resolved Jira issues / completed epics no statistically significant change
Firm-level engineering headcount no significant change

Read that table again. Every number on the top half measures activity. The two rows that would measure outcome are flat.

The gains didn't disappear. They moved downstream:

  • Average PR submit → merge time: +49% (about 3.45 days)
  • Revision requests per PR: nearly doubled
  • Comments per PR: +35%
  • Share of workers pulled into code review: +14%

And the obvious escape hatch isn't working yet. By March 2026, roughly 80% of firms were running an AI code review tool, but only 23.3% of review comments were AI-generated — and only 10.8% of PRs received at least one AI-generated review comment. The humans are still doing the quality control, on a bigger pile.

Two quotes from the study's interviews say it better than I can:

"People could make a huge number of commits quickly, but code review was still the bottleneck." — a junior engineer

"There is only one bottleneck at a time. If you speed up coding, the bottleneck shifts somewhere else." — a senior engineer

Why this isn't a "AI tools are overhyped" post

It would be easy to file this under agents don't work. That's not what the paper says, and the caveats matter:

  • It's observational, not a randomized trial. Firms that adopt early may differ from firms that adopt late in ways the model can't fully absorb.
  • It's firm-level averages. "No significant change in resolved issues" doesn't mean no team benefited. It means the average didn't move.
  • Adoption is timestamped off API usage and agent commit signatures — so unsigned personal use is invisible.
  • The data ends in March 2026. Six months of new models and new review tooling are outside the window.
  • There's at least one competing result: a separate Zhou et al. study of ~200 developers reported shorter review times around an AI assistant rollout. Different design, different population — but it's a real counterexample, and "AI always lengthens review" is too strong a claim.

So the honest reading is narrower and more useful: at the average firm, adding generation capacity to a pipeline whose review stage didn't scale produces more code, not more software.

October's releases already agree with the paper

The interesting part is that the vendors shipped as if they'd read it.

  • Microsoft rebuilt Copilot around Home / Code / Autopilot (announced Sept 25, still not GA as of Oct 8) and — more tellingly — moved Cowork, Code, Autopilot and long-running agent jobs onto usage-based Copilot Credits while everyday chat stays on the per-seat licence. The metered part is exactly the part that burns tokens unsupervised.
  • OpenAI's Codex CLI 0.162 (Oct 9–11) added managed Git worktree tools and Command Center task pinning, plus reusable cloud dev environments and a repo-scanning security product. Worktrees and task pinning are review-and-isolation features.
  • Anthropic's Claude Code 2.1.296 (Oct 9) added workflow-level subagent model pinning — you can force every subagent in a pipeline onto one chosen model. That's cost and predictability control, not generation speed.
  • Google's CodeMender line keeps iterating on automated security-finding triage and deduplication — review-side work again.
  • JetBrains open-sourced Mellum2.1 (Apache 2.0, 12B MoE, 2.5B active, 131k context): the pitch is a fast, cheap worker model for agent subtasks, not a frontier replacement. Its own benchmark table shows Qwen3.5-9B still ahead on SWE-bench Verified (50% vs 47%) and SWE-bench Pro (38% vs 28%); Mellum2.1 wins on LiveCodeBench v6 (82%) and throughput. Local review triage and compaction are plausible jobs for it. Fixing real repo bugs unattended is not, per its own numbers.

Notice the pattern: nobody is selling "generate faster" this month. Everybody is selling review, isolation, governance, and metering.

What I'd actually do about it

  1. Measure your own merge time before you buy more seats. The paper's 49% is an average; your number is the one that decides whether the next licence pays for itself.
  2. Treat review capacity as a budget line. If you can add one thing, add review throughput — better PR sizing, faster CI, clearer ownership — not another generator.
  3. Check whether your review agent is actually reducing merge time or just adding comments. "AI review is enabled" is not the same metric as "PRs merge faster." The paper's own split (80% adoption vs 23% AI-written comments) suggests a lot of teams have the first without the second.
  4. Size your drafts. Agents that open 400-line PRs walk straight into the bottleneck.

Sources

The uncomfortable version of this post is one sentence: you can't merge your way out of a review bottleneck by generating more. If your team's productivity story this quarter is "we're producing way more code," that's a leading indicator of a backlog, not of delivery.

Top comments (0)