Ask sixteen experienced open-source developers how much faster AI tools would make them, and they'll say about 24%. Ask them again right after they finish the task, and they'll still swear they were roughly 20% faster. The stopwatch says otherwise: they were actually 19% slower.
That gap is the headline result from a 2025 randomized controlled trial run by METR, a group that studies how AI systems actually perform in the wild rather than on benchmarks. Instead of testing beginners on toy problems, the study put experienced developers to work on real, mature codebases they already knew well — their own open-source repositories, with years of accumulated context already living in their heads.
The 39-point gap
Here's the setup: developers picked real issues from their own projects, then were randomly assigned to either use AI tools or not for each task. Before starting, they predicted AI would cut their completion time by about 24%. After finishing, they still believed they had come out roughly 20% ahead.
The actual clock time told a different story. Tasks done with AI assistance took about 19% longer than tasks done without it. That's a 39 percentage-point gap between what these developers believed happened and what a stopwatch recorded — in professionals who write code for a living.
Why the most experienced developers lost the most time
The slowdown wasn't evenly spread. It was strongest on exactly the codebases the developers knew best. That's the counterintuitive part: you'd expect deep familiarity to make AI assistance additive, a second pair of hands on a system you already understand. Instead, the tool mostly added overhead — prompting, reviewing suggestions, and correcting output — layered on top of work the developer could already do quickly from memory.
Ten years of shipping code builds a fast, private, mental model of a system: which functions are safe to touch, which abstractions leak, where the real bugs hide. That model runs in the background and is basically free. An AI assistant doesn't have it, so every suggestion has to be checked against it, and checking takes time the raw "assistance" never shows up as.
The same illusion shows up in floating point
There's a smaller, more mechanical version of this mismatch that any developer runs into on day one:
>>> 0.1 + 0.2
0.30000000000000004
Nothing is broken. Binary floating point can't represent 0.1 or 0.2 exactly, so the addition is off by a sliver too small to notice by eye but large enough to break an equality check. The computer isn't wrong — your intuition about what "0.1 + 0.2" means is just miscalibrated against how the machine actually stores numbers.
That's the same shape as the METR result. Your internal sense of speed, like your internal sense of what 0.1 plus 0.2 equals, is a model — usually good enough, occasionally confidently wrong, and only correctable by actually checking the value against a ground truth instead of trusting the feeling.
What actually worked
The METR study also found the developers who avoided the slowdown weren't the ones who resisted AI tools altogether. They were the ones who stopped guessing and started timing themselves — treating "does this actually help me" as a measurable question instead of a vibe.
The other habit that held up: writing down why the code looks the way it does, before that reasoning quietly leaves with whoever holds it in their head. A decade of experience is mostly an unwritten, unmeasured model of a system. AI tools are good at producing plausible-looking code fast; they are not a substitute for that model, and treating them as one is exactly how an experienced developer ends up slower while feeling faster.
If you want to know whether a tool is actually helping you, the fix is unglamorous: time it, on your own real tasks, more than once.
Here's the 60-second version of this story:
If you've caught yourself feeling faster while a task actually dragged on, that's the whole study in miniature — worth checking with a clock next time before you trust the feeling.
Top comments (0)