The AI Agents Were Done. The Delivery Clock Wasn’t.

The metric that stops at generated code hides review, rework, and acceptance — the work that decides whether the change can ship.

Cover illustration: the metric stops at generated code while delivery continues through review, rework, validation, and acceptance.

Both AI agents had finished. Their assigned checks passed. Then their patches were combined, and eight of the 24 acceptance methods turned red.

On sequence 21 of a paused coding study, the integrator needed 19.516 seconds to reconstruct shared branches before all 24 methods passed. If the timer had stopped when the agents reported done, the run would have looked successful. The delivery clock told a different story.

AI coding productivity has a finish-line problem. Generated code, a locally green test, and an opened pull request are useful events. None of them proves that a change survived the handoff into the system that must accept it.

AI does not become productive when output appears. It becomes productive when accepted work moves through the system without exporting more verification than it saves.

AI coding productivity and the seductive finish line

AI makes activity visible almost immediately. A terminal fills. A diff appears. A worker says the task is complete. Those signals feel like speed because the old blank space has disappeared.

The study used a production software delivery boundary: the timer stopped only after the integrated tree passed all 24 frozen methods and the evidence receipt was sealed.

There are at least four plausible places to stop a coding clock:

1. the first code is generated;
2. the assigned local checks pass;
3. the pull request is opened;
4. the integrated change reaches its agreed acceptance state.

The first three are useful telemetry. The fourth is delivery.

When a team reports only time-to-first-output, it can improve the reported metric while moving work into code review, integration, security checks, rollback preparation, or another person’s queue. The work did not disappear. It changed owners.

What one failed merge revealed

Sequence 21 was small enough to inspect. Two agents worked on separate requirement partitions, but both patches touched a shared module. Each patch was locally valid against its assigned checks.

The seam failed only after integration. Applying the second patch removed branches required by the first combined tree. The first full validation exposed eight failures. The integrator rebuilt the missing paths, reran the suite, and reached 24 of 24.

No single worker had an obvious local failure. The workflow had a delivery failure.

That distinction matters because dashboards often reward the event nearest to generation. A faster PR can coexist with a longer review. More accepted suggestions can coexist with larger batches. More completed agent tasks can coexist with a slower path to deployable software.

The research is messier — and more useful — than a slogan

In a 2025 randomized study, METR observed 16 experienced open-source developers completing 246 real issues in repositories they knew well. In that setting, developers took 19% longer when AI tools were allowed. Before the study, they expected a 24% speedup; afterward, they still believed AI had made them 20% faster.

That result does not prove that AI slows most developers. METR explicitly limits the claim to the studied people, tools, repositories, and period. It does show that perceived acceleration and measured task completion can point in opposite directions.

The follow-up became even more revealing. In February 2026, METR said it was changing the experiment design because adoption had altered who would participate and which tasks they would submit. Some developers would not work without AI. Others withheld tasks they did not want randomized into a no-AI condition. Multi-agent use also made time reporting unreliable for part of the sample.

The honest conclusion was not “AI won” or “AI lost.” It was that the measurement boundary no longer captured the work cleanly.

The same shift appears outside coding. A Microsoft Research study of 319 knowledge workers found that AI-assisted work changes where critical effort is applied: away from direct production and toward verification and integration. Generation becomes cheaper while judgment remains attached to the human accepting the result.

Measure the acceptance gap

The simplest correction is to add one interval:

Acceptance gap = accepted delivery timestamp − first generated completion timestamp

The gap is not automatically waste. Review, validation, and release decisions can be the work that makes an output safe. The purpose of measuring it is not to force it to zero. The purpose is to stop pretending it is free.

The same logic appears in production AI evaluation, quality gates, and monitoring: a plausible output is an intermediate state, while prompt versioning, shadow-mode validation, drift detection, and rollback procedures determine whether it can move forward.

The acceptance gap begins when generated work first looks complete and ends only at the agreed delivery boundary.

This is compatible with established delivery measurement. DORA’s current framework carries change lead time to deployment and pairs throughput with instability measures. It also warns against choosing one metric to rule them all.

An AI-specific measurement should preserve the same tension: speed matters, and the accepted outcome matters.

A five-event measurement card

You do not need a new analytics platform. Capture five timestamps for one recurring task type:

1. Generation complete: the agent first presents a supposedly complete change.
2. Pull request opened: the change enters the shared review system.
3. First full validation: the complete acceptance suite runs against the integrated state.
4. Last rework complete: required corrections stop changing the result.
5. Accepted or deployed: the team’s real delivery boundary is crossed.

Then record three companion values: the number of rework cycles, the number of acceptance failures, and whether the change caused rollback or unplanned follow-up.

Compare the same task class over time. Do not compare a documentation edit with a payment-system migration and call the difference an AI effect.

Three questions before calling AI productive

Where does the clock stop? If it stops at generation, local green, or PR creation, label the metric accordingly. Do not present an intermediate event as delivery time. The name of the metric should reveal its boundary.

Who absorbs verification? An agent can reduce the author’s typing while increasing a reviewer’s reconstruction work. Count the human and machine effort needed to reach acceptance, even when it occurs in another queue.

What counts as failure? A rejected patch, rollback, reopened issue, or second validation cycle must remain in the record. Removing inconvenient runs makes the tool look faster by redefining the work after it happens.

When the shorter clock is honest

Sometimes AI really does shorten delivery.

A narrow task with a frozen interface, fast tests, low shared-state risk, and a clear reviewer can move from request to acceptance faster because generation was the actual constraint. If accepted lead time falls while rework and failure do not rise, call it a productivity gain.

The acceptance gap is not an anti-AI metric. It is what allows a positive result to survive scrutiny.

Keep the finish line after the handoff

The useful question is not whether AI writes code faster. It often does. The useful question is whether the system accepts valuable changes faster after review, integration, and failure recovery are included.

For one recurring workflow, capture the five events. Keep the task class stable. Preserve failed runs. Then look at both generated speed and accepted delivery.

That is a narrower claim than “AI transformed productivity.” It is also a definition of AI coding productivity an engineering team can verify.

The delivery clock keeps running after the AI says done.