AI-assisted development can shorten one stage while leaving the system constraint elsewhere. The useful question is not whether a coding step became faster; it is where work now waits, and what evidence supports that conclusion.
That distinction matters because a local productivity observation does not establish a team-level delivery result.
One study is not a universal productivity number
A 2025 METR study of experienced open-source developers working in repositories they knew reported that the developers took 19% longer when using the AI tools in that study, despite expecting a speed increase. The result was specific to that sample, task design, tools, and measurement period. It does not establish that all developers are slower or that all AI-assisted work has the same effect.
METR's February 2026 update describes selection and measurement issues that affect how the original result should be interpreted. That update is a reason to keep the claim scoped, rather than replace one universal story with another.
The practical lesson is narrower and more useful: record the task, population, tool, comparison period, and measure before generalising from a productivity result.
This is a flow-metrics question
Coding speed is one stage of a delivery system. It does not tell us whether work reaches a usable outcome sooner. If implementation produces work faster than review, testing, or decision-making can absorb it, the queue moves downstream.
A team can therefore see more generated code while its end-to-end lead time stays flat or becomes harder to explain. That is a systems hypothesis to test with the team's own observations, not a claim about every organisation.
Useful questions include:
- Where does work wait between ready and released?
- Which review or test decisions are repeatedly revisited?
- How much of the change was generated, and how much was verified?
- What happens to change failure, rework, and recovery when the input or revision changes?
The measure should answer a decision. Deployment frequency alone cannot answer whether an agent-assisted change was well understood or safely recovered.
Review can become the visible constraint
AI-generated code can be plausible without carrying the reasoning a reviewer would normally infer from the author's work. That can increase the validation burden even when the typing burden falls.
The response is to instrument the review surface rather than declare a universal rule. A team might record the proposed revision, the tested revision, the input fixtures, the unresolved assumptions, the decision owner, and the recovery path. Those are proposed evidence fields; their presence does not prove that the control is complete or that the change is safe.
A useful test is simple: change the proposed revision after the evidence was recorded. If the review still passes, the boundary is not tied to the condition that was actually inspected.
Requirements can become the earlier constraint
When implementation is slow, vague requirements can remain hidden inside the build. A fast implementation path exposes that ambiguity sooner. The remedy is to make the contract, state, expected effects, and unresolved assumptions explicit before asking an agent to act.
That is a design choice, not a new job title or a prediction about the labour market. The important question is whether the team can inspect the decision before the generated change reaches a consequential boundary.
What a manager can measure
Start with a small, comparable observation period. Define what counts as AI-assisted work, select one flow question, and record the denominator. For example, a team could compare review wait time for a labelled sample of changes while also recording revision churn and the reason for rework.
Do not infer causality from a single dashboard snapshot. Keep the observation period, task mix, and limitations with the result. If the sample changes, the conclusion changes with it.
The honest version of the productivity discussion is therefore modest: a faster stage can move the constraint. The work is to find the new constraint, measure it with a defined question, and keep the evidence tied to the system that produced it.
For a concrete, zero-credential exercise, the Stopline team evaluation guide walks through one proposed browser-agent action and records the decision, evidence owner, and recovery question. It is an experimental evaluation aid, not a production control or certification.
Top comments (4)
This is the honest AI productivity story. The bottleneck rarely disappears; it moves from implementation speed to decision quality, review capacity, integration, and ownership of edge cases.
That is still valuable, but only if the team notices the new constraint. Otherwise everyone celebrates faster output while the real queue quietly moves to architecture, QA, or product judgment.
That's exactly it, and the failure mode I see most is a team celebrating the throughput number without checking where the queue actually moved to. If nobody's watching architecture, QA, or product judgment as a constraint, the bottleneck doesn't disappear. It just goes invisible until something breaks downstream of the part everyone was measuring.
Interesting - thanks! Requirements specification (a shall do b) versus system architecture (high level design) versus system design (low level design). Out of experience, setting LLMs loose without the system design (database schemas, API contracts, class diagrams) was begging for trouble.
Agreed, and that layering is underrated as a prerequisite rather than a nice to have. An agent working from "a shall do b" without the schema, contracts, and class diagrams underneath it is filling in the low level design itself, invisibly, one file at a time, and nobody signed off on those decisions. Making the system design explicit before turning an agent loose isn't extra process. It's the same information a human engineer would have needed anyway, just usually carried in someone's head instead of written down.