Manu Shukla

Posted on Jul 26 • Originally published at ecorpit.com

AI coding agents: 19% slower or 55% faster? A 2026 measurement playbook

#aicodingagents #developerproductivit #dorametrics #dxcore4

AI coding agents: 19% slower or 55% faster? A 2026 measurement playbook

Summary. In a 2022 GitHub controlled study, developers using Copilot finished a task 55% faster, 1 hour 11 minutes against 2 hours 41 minutes. In a 2025 METR randomized trial, experienced developers were 19% slower with AI, yet believed they were 20% faster. Both results are real. The 2025 DORA report, built on nearly 5,000 professionals, reconciles them: AI amplifies whatever a team already is, lifting delivery throughput while pushing stability down. So "are we faster?" cannot be settled by a vendor demo. This playbook shows how to measure the real impact of AI coding agents in 2026 using DORA, the DX Core 4, and Amazon's CloudWatch coding agent insights, launched on 20 July 2026, with a 30/60/90-day rollout and the token-cost signals to watch.

If you lead an engineering org, you are almost certainly paying for AI coding agents already. The open question is whether that spend shows up as faster, safer delivery or just as more code. Measuring it well is now a leadership skill, not a nice-to-have.

Why "are we faster?" is a hard question

Two of the most-cited studies point in opposite directions, and understanding why is the whole game.

The optimistic number comes from a GitHub and Microsoft Research controlled experiment. Recruited developers were asked to build an HTTP server in JavaScript; the group with Copilot finished 55% faster, averaging 1 hour 11 minutes against 2 hours 41 minutes for the control group, a result significant at P=.0017. It is a clean, isolated task with a clear finish line, which is exactly where an AI assistant shines.

The sobering number comes from METR, an independent research group. Its 2025 randomized trial put 16 experienced open-source developers through 246 real tasks on mature repositories they had worked on for about five years. With AI tools allowed, they were 19% slower, not faster. The twist that matters most for leaders: those same developers expected AI to speed them up 24%, and even after the trial they still believed it had made them 20% faster. Perceived speed and measured speed moved in opposite directions.

Neither study is wrong. They measured different work. Greenfield tasks with no existing context favour AI; deep changes in a large, familiar codebase, where the human already holds the model in their head, often do not. That is why a single anecdote, or a developer's own sense of speed, is the worst possible basis for a rollout decision.

The 2025 DORA report: AI is an amplifier

The largest recent dataset comes from the 2025 DORA report, "State of AI-assisted Software Development," drawing on survey responses from nearly 5,000 technology professionals and more than 100 hours of qualitative data.

Its headline finding is blunt: "AI doesn't fix a team; it amplifies what's already there." Strong teams get faster and better; struggling teams find their existing problems intensified. Concretely, the 2025 data shows a positive relationship between AI adoption and both software delivery throughput and product performance, a shift from the more negative 2024 picture. But AI adoption continues to have a negative relationship with software delivery stability. More change, delivered faster, without strong safety nets, breaks more often.

Three supporting numbers are worth pinning to the wall. AI adoption is near-universal at 90% of respondents; more than 80% believe it has increased their productivity; and 30% still report little or no trust in the code AI generates. The report also found that 90% of organisations have adopted at least one internal platform, and that platform quality correlates directly with an organisation's ability to turn AI into real value. The lesson: measure stability and delivery together, never throughput alone.

The frameworks that actually measure it

You do not need to invent metrics. Four established frameworks cover the ground, and the newest one unifies the rest.

Framework	Core dimensions	What it is best for
DORA four keys	Deployment frequency, lead time for changes, change failure rate, failed-deployment recovery time	A delivery-performance baseline for a team or org
SPACE	Satisfaction, Performance, Activity, Communication, Efficiency	Balancing output against developer well-being
DevEx	Flow state, feedback loops, cognitive load	Diagnosing the friction AI is supposed to remove
DX Core 4	Speed, Effectiveness, Quality, Impact	A single unified scorecard over DORA, SPACE and DevEx
Raw activity counts (PRs, diffs, commits)	Individual output volume	A warning sign to watch, not a target to chase

The DX Core 4 is the one to standardise on for an AI rollout. Created by Laura Tacho, CTO at DX, and Abi Noda, its co-founder and CEO, together with the authors of DORA, SPACE and DevEx, and refined across more than 300 organisations, it rolls the older frameworks into four dimensions: Speed, Effectiveness, Quality and Impact. Speed carries DORA's delivery metrics plus developers' perceived delivery rate. Effectiveness is captured through the Developer Experience Index, a 14-item survey covering code quality, focus time and CI/CD friction. Quality tracks reliability and stability. Impact ties the work back to business value. Measuring all four at once is what stops a team from celebrating a Speed gain that quietly wrecked Quality.

The last row of the table is the trap. Counting pull requests, diffs or commits per developer is easy, which is exactly why it is dangerous: optimise for it and you get more, smaller, less-reviewed changes. Treat activity as a diagnostic signal, never a goal.

Instrument the agents: CloudWatch coding agent insights

Frameworks tell you what to measure; you still need the telemetry. That is the gap Amazon CloudWatch coding agent insights, launched on 20 July 2026, is built to fill.

It integrates with the Claude apps gateway for AWS to collect telemetry from Claude Code without additional instrumentation, and it also supports other agents including OpenAI's Codex and GitHub Copilot. The data model is OpenTelemetry metrics emitted by the agents, presented alongside your existing CloudWatch operational data, so agent usage sits next to the delivery and reliability signals you already track. Standard CloudWatch OpenTelemetry metric-ingestion pricing applies, and the feature is available in all AWS commercial regions except the UAE, Bahrain and Israel (Tel Aviv).

What it lets a leader actually do: track token spend and its trend, set proactive token billing alerts, correlate agent adoption with improvements in commit throughput and pull-request velocity, and identify which models deliver the best cost-to-output ratio for a given workload. In other words, it connects the two questions FinOps and engineering usually ask separately, "what are we spending?" and "is it working?", into one dashboard.

Signal to track	Source	Question it answers
Token spend and trend	OpenTelemetry from Claude Code, Codex, Copilot	Are we within budget, and which team is overspending
Adoption correlated with commit and PR velocity	CloudWatch plus delivery data	Where are agents actually accelerating delivery
Cost-to-output ratio by model	Per-model telemetry	Which model earns its keep for this workload
Adoption by team	Gateway telemetry	Which teams should get expanded access
Delivery stability beside agent usage	DORA change-failure data	Is the extra speed costing us reliability

The token-cost angle is easy to underrate. At frontier prices, a model such as Claude Opus 5 runs at $5 per million input tokens and $25 per million output tokens, so an agent left on maximum effort for routine work burns budget fast. Our breakdown of Claude Opus 5 cost per task for coding agents and the wider LLM inference cost picture for enterprise AI show why per-task cost, not sticker price, is the number to instrument.

A measurement stack you can stand up

Put the pieces together into something a platform team can build in a quarter.

Start with a DORA baseline before you scale any agent, so you know your deployment frequency, lead time, change failure rate and recovery time without AI in the picture. Layer the DX Core 4 on top so Speed and Quality are read together, and run the Developer Experience Index survey to capture the human side that logs cannot see. Then wire agent telemetry through OpenTelemetry into CloudWatch coding agent insights (or your existing observability stack) so token spend and model choice sit beside those delivery metrics. Close the loop with change-failure and incident data, because that is where the DORA stability warning shows up first. For teams that already run model evals, fold agent quality into the same observability practice rather than standing up a parallel one; our note on AI evals and observability covers that integration.

The discipline that makes this work is comparison. Measure a team for a few sprints without agents, then with them, and compare the same metrics rather than trusting the before-and-after feeling. METR's perception gap is the reason: the story your developers tell about their speed is not evidence.

The traps that make measurement lie

Four failure modes quietly ruin AI-productivity dashboards.

The first is Goodhart's law: the moment PRs-per-developer becomes a target, it stops measuring anything useful, because people optimise the number. The second is measuring individuals instead of the system; the DORA data shows individual output can rise while organisational delivery stays flat, so a per-developer metric can look great while the product ships no faster. The third is watching throughput without stability, which is precisely how AI's amplifier effect turns a speed win into an incident spike. The fourth is cost blindness: without token-level tracking, a team can double its cloud AI bill with no matching delivery gain, and never notice until finance does. Tie token spend to delivery outcomes from day one; the Microsoft and Uber study on Claude Code token cost is a useful reference for what disciplined tracking looks like.

India-specific considerations

For Indian engineering teams and the global capability centres (GCCs) that now run large AI-assisted delivery, the measurement case is sharper, because these centres are judged on demonstrable output value to a global parent.

Two practical points. First, cost is billed in US dollars against rupee budgets, so token-spend tracking is also FX-risk tracking; instrument it with the same rigour you would apply in a cloud FinOps managed service rather than reviewing it once a quarter. Second, coding-agent telemetry frequently captures repository names, prompts and sometimes fragments of source or customer data, which brings it under the Digital Personal Data Protection (DPDP) Act 2023 once personal data is involved. Decide what the agent gateway may log, apply minimisation, and keep that telemetry inside your governed observability boundary. CloudWatch coding agent insights operating on OpenTelemetry metrics rather than raw prompt contents helps, but the retention and access decisions remain yours.

A 30/60/90-day rollout

Days 1 to 30: capture a clean DORA baseline and pick your framework (DX Core 4 is the safe default). Agree which activity counts you will watch as warning signs, not targets, and set token budgets per team.

Days 31 to 60: enable agents for one or two pilot teams, wire OpenTelemetry into CloudWatch coding agent insights, and run the Developer Experience Index survey. Compare pilot teams against a control group on the same metrics.

Days 61 to 90: review Speed and Quality together. Expand access only where the data, not the enthusiasm, shows a real gain, and fortify safety nets, automated tests and review gates, before you scale, because that is where the DORA stability risk bites.

The honest verdict from the evidence is that AI coding agents can make a strong team meaningfully faster and a weak team faster at shipping bugs. Which one you get is a measurement and platform problem, not a licensing one.

FAQ

Do AI coding agents actually make developers faster?

Sometimes. A 2022 GitHub controlled study found Copilot users 55% faster on an isolated task, but a 2025 METR field trial found experienced developers 19% slower on real work in mature repositories. The gain depends on the task and the team, which is why measurement, not vendor demos, should drive the decision.

Why did the METR study find AI made developers slower?

METR tested 16 experienced developers on 246 real tasks in codebases they knew well, where the human already holds the context AI has to rebuild. Developers were 19% slower yet believed they were 20% faster. The perception gap is the key lesson: self-reported speed is unreliable evidence for a rollout.

What is the best way to measure AI coding agent impact?

Use an established framework rather than inventing metrics. Start with a DORA baseline, then adopt the DX Core 4, which unifies DORA, SPACE and DevEx across Speed, Effectiveness, Quality and Impact. Read Speed and Quality together so a delivery gain that damages stability cannot hide.

What is CloudWatch coding agent insights?

Amazon CloudWatch coding agent insights, launched on 20 July 2026, collects OpenTelemetry telemetry from agents like Claude Code, Codex and GitHub Copilot and presents it beside your operational data. It tracks token spend, correlates agent adoption with commit and pull-request velocity, and flags which models give the best cost-to-output ratio.

What does the 2025 DORA report say about AI?

The 2025 DORA report, based on nearly 5,000 professionals, concludes that AI amplifies a team's existing strengths and weaknesses. AI adoption relates positively to delivery throughput and product performance but negatively to stability. It also found 90% of respondents use AI and 30% still distrust AI-generated code.

Which metrics should you avoid when measuring AI coding tools?

Avoid raw activity counts such as pull requests, diffs or commits per developer as targets. By Goodhart's law, once an output count becomes a goal, people optimise the number rather than the outcome, producing more, smaller, less-reviewed changes. Treat activity as a diagnostic signal, never a performance target.

How long before AI coding agents show measurable gains?

Plan for a full quarter. A workable path is 30 days to set a DORA baseline and budgets, 30 days to pilot agents against a control group with telemetry wired in, and 30 days to compare Speed and Quality before scaling. Expand only where the data, not enthusiasm, shows a gain.

How should Indian teams and GCCs measure AI coding ROI?

Track token spend as an FX-exposed cost against rupee budgets with FinOps rigour, and correlate it with the same DORA and DX Core 4 metrics a global parent expects. Keep agent telemetry within a governed boundary under the DPDP Act 2023, applying data minimisation to anything the agent gateway logs.

How eCorpIT can help

eCorpIT helps engineering organisations in India and worldwide roll out AI coding agents and prove whether they work. We stand up DORA and DX Core 4 baselines, wire agent telemetry through OpenTelemetry into CloudWatch or your existing observability stack, and tie token spend to delivery and stability metrics so speed gains do not come at the cost of reliability. If you want a measurement stack and a safe rollout rather than a licence and a hope, see our work on enterprise AI agent development and AI evals and observability, or contact us to scope a pilot.