A Right AI Answer Can Still Cost You Time. Here's the Arithmetic and What the Studies Say

#ai #programming #productivity #softwaredevelopment

A simple break-even model, three kinds of evidence, and one study that told us its own data was too compromised to trust.

A recent Coder Legion post by BizzAppDev, "AI Gives the Right Answer. But Does It Actually Save Time?", makes a point I think is right: an AI feature should be judged by how much work is left after the answer appears, not by how good the answer is. The post has no numbers, so I went looking for some, and I added a small calculation of my own.

Here is what I found.

The arithmetic of a "right" answer

Take a task that takes 30 minutes by hand. With AI you pay for four things: generating the answer, getting it into the right place, reviewing it, and fixing it when it is wrong. I'll assume 2 minutes to generate, 4 to hand it off, 5 to review, and 20 to fix a wrong answer. These are my assumed numbers, not measurements.

Plug them in and AI saves 17 minutes when it is right 90% of the time, 13 minutes at 70%, 9 at 50% and 5 at 30%. So far, so good.

Now change one thing. If careful review of code you did not write takes 12 minutes instead of 5, the break-even point moves from "right 5% of the time" to "right 40% of the time". If the hand-off takes 10 minutes instead of 4, because someone has to open another screen, find the record, copy the answer and update it by hand, break-even moves to 35%. That hand-off is exactly the scenario the Coder Legion post describes.

Accuracy is one term in the sum. Review and hand-off are the others, and they are the ones that rarely get measured.

What controlled studies found

A 2023 preprint studying GitHub Copilot found that developers with the AI pair programmer finished one task, writing an HTTP server in JavaScript, 55.8% faster than a control group. It was a clean, self-contained task, which is where AI tends to look best.
In mid-2025, METR ran a randomized trial with 16 experienced open-source developers working on 246 real issues in repositories they knew well. Each issue was randomly assigned to allow or forbid AI. With AI allowed, the developers took 19% longer. They had expected a 24% speedup, and afterwards they still believed AI had made them 20% faster. The authors list limits themselves: a small group, one kind of work, and tools that may not have been used in the best way.
In February 2026, METR published a follow-up with 57 developers across 143 repositories and more than 800 tasks. The raw numbers pointed the other way: tasks took 18% less time for the 10 returning developers and 4% less for the 47 new ones. Both ranges include "no change" (from 38% less to 9% more, and from 15% less to 9% more).

METR then said its own data was too compromised to rely on. Developers increasingly refuse to work without AI, so the most optimistic people drop out of the no-AI condition, and 30% to 50% said they held back tasks they did not want to do without it. METR still believes developers are probably more sped up now than in early 2025, but calls its data only very weak evidence of that. My reading: the honest answer today is "it depends on the task, the tool and the year", and measuring it well is hard. One thing was solid in 2025, though: people felt faster while the clock said slower.

What developers say

The 2025 Stack Overflow Developer Survey asked what frustrates people about AI tools. The top answer, chosen by 66% of respondents, was "AI solutions that are almost right, but not quite". Another 45.2% said debugging AI-generated code is more time-consuming. On accuracy, about a third said they trust it and about 46% said they distrust it, while 47.1% use AI tools every day.

My reading of "almost right": it is the most expensive case in the arithmetic above. A clearly wrong answer is rejected in seconds. An almost-right one passes a quick look, and you pay the review cost and then the fix cost.

What I would measure instead

Measure time to done, not time to answer. Start the clock when you begin the task and stop it when the result is where it needs to be.
Count how often you had to fix or redo the AI's output, and how long that took.
Count the manual steps left after the answer appears. BizzAppDev's three questions are a good checklist: is there less work afterwards, did the information reach the right place, and did the task finish without unnecessary manual steps?

The 2025 DORA report on AI-assisted software development describes AI as an amplifier of an organization's existing strengths and weaknesses, which fits: the same tool saves time in a team with a clean workflow and costs time in one without.

The METR result is the reason to write numbers down. For a week, note your own estimate for each AI-assisted task and then the real time including review and fixes. In the 2025 METR study, the developers' feeling and the clock disagreed.

What I did not test

I ran no experiment on real people or real tasks. The 30-minute model uses numbers I assumed, so change them to match your own work and see where your break-even lands. The studies above are about writing code, while the Coder Legion post is about AI features inside business tools, which is a related but different case. I read METR's two posts, the Stack Overflow survey results page and the arXiv abstract page for the Copilot experiment, but only the overview page of the DORA report, not the full report.

How do you measure whether an AI tool really saves you time? A commenter on the original post asked the same thing, and I still don't have a method I would call reliable.

DEV Community