24% more merged PRs: Microsoft's 2026 Claude Code study, and what it does not prove

Summary. On 1 July 2026, three Microsoft researchers published the first field study to track a company-wide rollout of agentic command-line coding tools with developer-level telemetry. Studying tens of thousands of engineers across a 16-week window, Emerson Murphy-Hill, Jenna Butler and Alexandra Savelieva found that adopters of Claude Code and GitHub Copilot CLI merged about 24% more pull requests than the model's estimate of what they would have merged otherwise: +24.0% in PRs per engineer per day, 95% CI [+14.5%, +33.7%], p<0.001. The lift did not fade between February and April 2026. Buried in the same paper is a number almost nobody has quoted: split by tool, Copilot CLI adopters saw +24.9% and Claude Code adopters saw +11.4%, a gap of roughly 2.2 times. The authors are Microsoft employees, and Microsoft owns GitHub. At organisational scale, they note, token spend "can run into millions of dollars annually", which is why reading this paper correctly is worth an afternoon of a CTO's time.

The paper is arXiv:2607.01418. It matters because most of what engineering leaders have to work with is vendor marketing, self-reported surveys and a single much-cited randomised trial from 2025 that its own authors have since labelled out of date. This is the first time anyone has published telemetry from a real rollout at this scale, with a pre-registered-looking design, a placebo test and an honest limitations section.

It is also being misread. The 24% figure is travelling through newsletters and LinkedIn posts stripped of the four conditions that make it meaningful. Below is what the study measured, what it explicitly did not measure, how it squares with METR's contradictory result, and a measurement plan you can run on your own team in one quarter.

What Microsoft actually measured

The design has two halves, and conflating them is the first mistake readers make.

The adoption study asks who tries Copilot CLI and who keeps using it. It covers Microsoft software engineers who were all eligible to try Copilot CLI on the rollout date. Retention is defined as recording activity on at least 5 of the 14 days beginning with first use.

The outcomes study asks whether adopters of either tool merged more code. It runs in two parts. Part 1 builds a synthetic counterfactual with CausalImpact, comparing adopters against a non-adopter control on merged PRs per engineer per day. Part 2 holds each engineer as their own control with a fixed-effects Poisson model, comparing weeks in which an engineer used a tool against weeks in which the same engineer used nothing.

The windows are specific. The pre-period runs 1 October 2024 to 4 January 2026, which is 461 days. The post-period runs 5 January to 29 April 2026, which is 115 days. The authors treat 5 January 2026 as "the rollout", the point at which internal use of both tools took off. GitHub announced general availability of Copilot CLI on 25 February 2026; Microsoft had earlier access through a preview programme. Claude Code arrived on a narrower path, reaching specific divisions and individuals through a managed programme from late 2025.

PRs were counted from Azure DevOps only, with a 28-day merge window.

The headline number, and the check that supports it

CausalImpact put the post-period lift at +24.0% in PRs per engineer per day, 95% CI [+14.5%, +33.7%], p<0.001.

The reason to take that seriously is the placebo test. The authors re-ran the same synthetic-control model with a fake intervention date of 6 October 2025, a point at which nothing happened. It returned -1.1%, 95% CI [-10.6%, +8.6%], an interval comfortably straddling zero. A model that manufactures lifts out of seasonal noise would have produced a spurious effect there. It did not.

Measure	Estimate	95% interval
Overall lift, PRs per engineer per day	+24.0%	[+14.5%, +33.7%]
Placebo intervention, 6 Oct 2025	-1.1%	[-10.6%, +8.6%]
February 2026 bucket	+29.4%	[+17.7%, +44.4%]
1 March to 29 April 2026 bucket	+20.0%	[+7.4%, +35.9%]
Copilot CLI, any-use weeks	+24.9%	[+23.0%, +26.8%]
Claude Code, any-use weeks	+11.4%	[+9.4%, +13.6%]

The lift did not fade

Prior work on Cursor in open source found an early velocity gain that decayed within a few months. Microsoft's data does not show that pattern. Splitting the post-period into buckets, February came in at +29.4% and the 1 March to 29 April stretch at +20.0%. Those posterior intervals overlap heavily and both exclude zero, so the drop in the point estimate sits inside sampling noise. Within a 16-week window, this is not a novelty effect.

The honest caveat is in the paper: sixteen weeks is "long enough to surface an early fade but not a long-horizon decline." If you are underwriting a three-year tooling budget, this study does not cover your time horizon.

The per-tool split nobody is quoting

Here is the finding that should change how you read every summary of this paper.

In the within-person analysis, where each engineer is compared against their own zero-tool weeks, the two tools did not perform alike:

• Copilot CLI, any use: +24.9% [+23.0%, +26.8%]
• Claude Code, any use: +11.4% [+9.4%, +13.6%]

Copilot CLI adopters showed roughly 2.2 times the lift of Claude Code adopters, at p<0.0001. Every headline of the form "Claude Code delivers 24% more PRs" is quoting an aggregate that the paper itself decomposes into a much less flattering per-tool result for Claude Code.

Before anyone reaches for a procurement decision, three things complicate that comparison, and the paper flags all of them.

First, exposure was asymmetric by design. Copilot CLI was available to every eligible engineer at rollout. Claude Code arrived through a narrower managed programme whose access list kept changing. Shortly after the observation window closed, an internal announcement said Claude Code licences would be discontinued for most engineers within about a month, with those engineers directed to Copilot CLI. The authors deliberately ended the analysis before that migration, because otherwise "migrating engineers would look like organic Copilot CLI adopters, and Claude Code usage would fall for policy reasons rather than engineer preference." That is careful work. It also means the two tools were never on equal footing inside the sample.

Second, the tool comparison excludes multi-tool users, so anyone running both is absent from that particular figure.

Third, and most importantly, the authors say this themselves:

"The authors are Microsoft employees; Microsoft sells AI tools, encourages their use, and owns GitHub, the maker of Copilot CLI. Though under no explicit pressure, our proximity may have shaped our questions, design, and interpretation."

That is Emerson Murphy-Hill, Jenna Butler and Alexandra Savelieva, researchers at Microsoft, writing the positionality statement in their own paper. Publishing a result that favours your employer's product is not disqualifying, and disclosing it plainly is the correct scientific behaviour. But a 2.2x advantage for the in-house tool, measured by the in-house team, on the in-house PR system, is the one result in this paper that deserves independent replication before it moves any money.

Throughput scales with how often the tool is used

The dose-response curve is the most actionable part of the study for anyone planning a rollout. Comparing an engineer's weeks by number of tool-use days against their own zero-day weeks, the curve is monotone and well separated:

• Three tool-use days per week: +15.0%
• Five or more tool-use days per week: +50.1%

The gains concentrate in habitual users. A licence sitting unused returns nothing, and light use returns little. This is the practical case for treating rollout as a behaviour-change problem rather than a procurement one.

Adoption itself spread socially. Among engineers eligible for Copilot CLI, exposure through reviewer peers, the colleagues someone regularly trades code with, moved the odds of first use substantially; the top exposure bucket carried a headline odds lift of around +54%. Retention, by contrast, tracked how much code an engineer was already writing rather than their demographics. The more pull requests an engineer created per week before the rollout, the more likely they were to stick with the tool.

The authors also caution that heavier-use weeks may simply carry a lighter task mix, so they read the dose-response as an association rather than a cause.

Who benefited most

Evaluated at three tool-use days per week, mid-career individual contributors saw a +21.3% [+19.6%, +23.0%] lift, and engineers with 5 to 15 years of tenure saw +21.5% [+19.9%, +23.1%]. Both moderator families show a "C" shape: the most junior individual contributors and the most senior managers picked up larger lifts than the middle of the distribution.

The survey responses point at a mechanism. One developer described the tools as "an especially good fit for experienced, senior developers, who can break work" into pieces an agent can execute. Another described the shift in what work became possible at all, saying the tool gave them "the ability to make larger changes that I never would have taken on in the past, like splitting our huge test project into five separate files."

That is the more interesting claim hiding under the PR count. Some of the lift is not the same work done faster. It is work that would previously have been skipped.

The METR contradiction, and why it mostly dissolved

The standard rebuttal to any AI productivity claim is METR's randomised controlled trial. It is worth handling properly, because the version circulating in 2026 is out of date.

In July 2025 METR reported that 16 experienced open-source developers working 246 real issues in their own mature repositories took 19% longer when allowed to use AI tools, 95% CI [+2%, +39%]. The repositories averaged more than 22,000 stars and over 1 million lines of code. Developers were paid $150 per hour. The tooling was primarily Cursor Pro with Claude 3.5 and 3.7 Sonnet, between February and June 2025. The striking part was the perception gap: developers forecast a 24% speedup beforehand and still believed they had been sped up 20% afterwards.

METR now labels that result historical. On 24 February 2026 the organisation published an update from a second study begun in August 2025, covering 57 developers across 143 repositories and more than 800 tasks, paid $50 per hour, with a median of 10 years of experience. The raw results reversed sign. For the subset of developers returning from the original study, METR estimates a speedup of -18%, 95% CI [-38%, +9%]. For newly recruited developers, -4%, 95% CI [-15%, +9%]. Negative numbers here mean faster.

METR does not present those figures as a clean win, and neither should anyone else. The organisation says the data "gives us an unreliable signal", because between 30% and 50% of developers reported withholding tasks they did not want to attempt without AI, and an increasing share declined to participate at all. Their own reading: "we believe it is likely that developers are more sped up from AI tools now, in early 2026, compared to our estimates from early 2025. However, because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase."

So the two studies are not really in conflict. They measure different things, in different years, with opposite biases.

Dimension	Microsoft field study	METR RCT
Design	Observational telemetry, synthetic control plus within-person	Randomised at task level
Population	Tens of thousands of Microsoft engineers	16 developers (2025); 57 developers (late 2025)
Period measured	5 Jan to 29 Apr 2026	Feb to Jun 2025; Aug 2025 onward
Tooling	Copilot CLI, Claude Code	Cursor Pro with Claude 3.5/3.7 Sonnet
Outcome metric	Merged PRs per engineer	Self-reported task completion time
Headline result	+24.0% merged PRs	+19% time (2025); -18% and -4% (late 2025)
Main weakness	Adopters self-select; vendor-affiliated authors	Tiny sample; severe task and participant selection
Authors' own caveat	A merged PR is not the value it delivers	Results labelled out of date by METR

The pattern across both, plus METR's May 2026 survey of 349 technical workers reporting a median 1.4x to 2x self-reported change in the value of work, is that self-reports run hot and controlled measurements run cold. In that survey METR notes its own staff gave the lowest estimates of any subgroup, and that in the 2025 experiment people overestimated AI's effect on their time by more than 40 percentage points.

Treat any number sourced from a developer survey, including your own internal one, as an upper bound.

Four things this study does not prove

The paper is unusually clear about its own limits, and these are the sentences that get cut from the summaries.

Merged PRs are not value. The authors say so directly, acknowledging that a merged PR "is not the same as the value it delivers." Merged-PR counts reward small, frequent PRs, and the study may miss quality costs such as added complexity. A team that responds to this paper by optimising PR count will get more PRs and learn nothing about whether it shipped more product.

This is not a finding about AI tools in general. Microsoft engineers already had GitHub Copilot in IDEs for code completion, chat and agent mode for several years before this window. The paper states that readers "should not interpret our findings broadly as being about AI tools, but rather as about the adoption and impact of agentic command line tools specifically in a context where non-CLI tools were already available." The 24% is incremental on top of an existing Copilot baseline, not measured against a team with no AI at all. If your team is starting from zero, this number does not describe your situation.

Adopters chose themselves. In Part 1, the engineers who adopted are not a random sample of engineers. The placebo test mitigates the seasonal-confound worry, and the within-person design in Part 2 removes adopter-versus-non-adopter bias entirely, but self-selection into adoption remains.

The measurement is partial. PRs came from Azure DevOps only, undercounting engineers on other ecosystems. Career stage came from a single HR snapshot, so promotions during the post-period are invisible. Retention thresholds were somewhat arbitrary, though the model held at 3-of-14 and 7-of-14 as well as the headline 5-of-14.

A measurement plan you can run in one quarter

Copying Microsoft's design is not realistic for a 40-engineer team; the statistical power is not there. Copying its discipline is. Here is a plan sized for a normal engineering organisation.

Set a pre-period before you buy anything. Microsoft used 461 days of history. You need at least one full quarter of clean baseline telemetry before the first licence is issued. If you turn on the tool first and start measuring second, you have no counterfactual and the exercise is theatre.

Randomise at the team level, not the individual level. Individual randomisation leaks: engineers talk, pair, and review each other's code, which is exactly the social diffusion Microsoft measured. Two comparable squads, one with licences and one without for a quarter, is cruder but survives contact with reality.

Instrument the dose, not just the seat. The dose-response curve is the whole story. Track tool-use days per engineer per week, because a team averaging one day per week and a team averaging five sit at opposite ends of a curve that runs from roughly +15.0% to +50.1%. Seat counts tell you nothing. Standard OpenTelemetry instrumentation for agent traffic makes this tractable; our guide to OpenTelemetry GenAI conventions for LLM and agent tracing covers the span attributes worth capturing.

Pair throughput with at least two counterweight metrics. Because merged PRs reward small, frequent PRs, track change failure rate and median PR review time alongside them. If PR count rises 20% while review time rises 40%, you have moved work from authors to reviewers rather than created capacity.

Budget the tokens before the pilot, not after. The paper's opening warning is about cost: at organisational scale, token spend "can run into millions of dollars annually", and misreading adoption "can make a rollout expensive without changing engineering velocity". Model the spend against the dose you actually want. Our breakdown of Microsoft and Uber's Claude Code token costs and the mechanics of Copilot AI credit pools for enterprise cost control cover the two spending models most teams choose between. Teams standardising on free tooling first can start with our list of free tools to measure LLM costs.

Seed adoption through reviewers, not through announcements. This is the study's clearest operational finding. First use spread through reviewer peers and skip-level peers, with the top exposure bucket carrying roughly a +54% odds lift. A launch email is the weakest possible intervention. Getting four or five widely-read reviewers using the tool visibly, in public PRs, is the strongest.

Decide your isolation posture before week one. Agentic CLI tools execute commands and edit files. If you are granting that to a coding agent across production repositories, the sandboxing question comes before the productivity question; see our VM isolation guide for AI coding agents. Teams running several agents at once should also read our notes on parallel agents and git worktrees.

The real cost of a CLI agent rollout is usually the measurement discipline, not the licences.

India-specific considerations

Two points apply with extra force to engineering organisations operating in India.

The first is data protection. This study runs on developer-level telemetry: per-engineer tool-use days, PR activity, career stage and tenure pulled from HR records. Building the equivalent dataset in India means processing employee personal data, which sits squarely inside the Digital Personal Data Protection Act 2023. Notice obligations, purpose limitation and the rules on processing employee data all apply, and "we are measuring productivity" is a purpose that needs stating up front rather than discovering later. Aggregate at the team level wherever the analysis allows it. Our DPDP Act engineering playbook for Indian startups covers the notice and consent mechanics.

The second is the composition of the workforce. The study's tenure result, a +21.5% lift for engineers with 5 to 15 years of tenure and a "C" shape with larger gains at both ends, maps awkwardly onto Indian service and product teams, which typically skew far more junior than Microsoft's engineering population. If most of your team sits below three years of experience, you are extrapolating from a subgroup the paper measured thinly. Run your own pilot rather than importing the number.

Token pricing is charged in dollars for every major vendor, so a rupee-denominated budget carries currency risk on top of usage risk. Model both.

FAQ

What did the Microsoft study actually find?

Studying tens of thousands of engineers between 5 January and 29 April 2026, Microsoft researchers found that adopters of Claude Code and GitHub Copilot CLI merged about 24% more pull requests than a synthetic counterfactual predicted, with a 95% confidence interval of +14.5% to +33.7% and p below 0.001.

Is the 24% figure specific to Claude Code?

No. The 24.0% is the aggregate lift across both tools. Split by tool in the within-person analysis, Copilot CLI adopters showed +24.9% and Claude Code adopters showed +11.4%, roughly a 2.2 times difference at p below 0.0001. Exposure to the two tools was not symmetric inside Microsoft.

Does this contradict METR's finding that AI made developers slower?

Less than it appears. METR's 19% slowdown came from 16 developers using early-2025 tooling, and METR labels it out of date. Its February 2026 update reports raw speedups of 18% for returning developers and 4% for new recruits, while cautioning that selection effects make the signal unreliable.

Did the productivity gain fade over time?

Not within the window measured. The lift was +29.4% in February 2026 and +20.0% across 1 March to 29 April 2026. Those intervals overlap substantially and both exclude zero, so the decline sits within sampling noise. The authors note 16 weeks cannot rule out a longer-horizon decline.

How much does usage frequency matter?

A great deal. Comparing each engineer against their own zero-tool weeks, the lift was +15.0% at three tool-use days per week and +50.1% at five or more days. The benefit concentrates in habitual users, which makes unused licences the most expensive line in a rollout budget.

Should I trust a study written by Microsoft employees about Microsoft tools?

Read it with the authors' own disclosure in hand. They state that Microsoft sells AI tools, encourages their use and owns GitHub, and that their proximity may have shaped the questions, design and interpretation. The placebo test and within-person design are genuine safeguards, but the per-tool comparison warrants independent replication.

What is the biggest limitation for my team?

Microsoft engineers already had GitHub Copilot in their IDEs for years before this rollout. The paper says findings should not be read as being about AI tools generally, but about agentic command-line tools where non-CLI tools were already present. Starting from no AI tooling at all puts you outside the measured setting.

Are merged pull requests a good productivity metric?

The authors say plainly that a merged PR is not the same as the value it delivers, and that the metric rewards small, frequent PRs while potentially missing quality costs such as added complexity. Pair any throughput count with change failure rate and review latency before drawing conclusions.

How eCorpIT can help

eCorpIT runs AI tooling pilots for engineering organisations the way this paper runs its analysis: baseline first, dose instrumented, counterweight metrics in place before the first licence is issued. Our senior engineering teams set up the telemetry, design a team-level comparison that survives real-world contamination, and model token spend against the usage pattern you actually want rather than the seat count a vendor quotes. We also handle the DPDP-aligned data handling that developer-level measurement requires in India. Talk to us about scoping a one-quarter measurement pilot before you commit to an organisation-wide rollout.

References

1. Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI (arXiv:2607.01418, abstract)
2. Full text of arXiv:2607.01418v1, including the outcomes study, dose-response results and threats to validity
3. METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (10 July 2025)
4. Becker, Rush, Barnes and Rein, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089)
5. METR, We are Changing our Developer Productivity Experiment Design (24 February 2026)
6. METR, Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity (11 May 2026)
7. GitHub Changelog, GitHub Copilot CLI is now generally available (25 February 2026)
8. METR, task substitution and uplift research
9. METR, dataset for the early-2025 developer productivity study
10. METR, dataset for the late-2025 developer productivity study
11. InfoQ, GitHub Copilot CLI Reaches General Availability

Last updated: 21 July 2026.