Most AI productivity benchmarks test developers on code they've never seen before.
METR ran a randomized controlled trial in July 2025. 16 experienced developers, 246 real tasks, in their own OSS repos - projects they'd owned for years. AI allowed vs. not allowed. The AI group took 19% longer.
The same devs predicted a 24% speedup before the trial. After it ended, they still believed they'd been ~20% faster. They weren't.
That 43-point gap between felt speed and measured speed is the actual story. "It feels faster" is not a measurement - it's a sensation, and AI tools are very good at triggering it.
Where AI wins: greenfield projects, boilerplate, unfamiliar tech. In a codebase you already hold in your head, verifying and steering suggestions costs more than just writing the fix yourself.
We apply "trust but verify" to every other benchmark that could drive a hiring decision or an architecture change. Why not this one?
Measure the tool against your work, on your codebase, on problems you already know how to solve. Not the conference talk.
Is your productivity signal speed on day 1 in a new repo, or month 6 in the system you own? Those measure different things.
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Top comments (0)