In July 2025, METR published a study that nobody in a boardroom wanted to read.
They took 16 experienced open-source developers and 246 real iss...
For further actions, you may consider blocking this person and/or reporting abuse
“Use daily, trust rarely” can be healthy only while review capacity scales with generation.
The hidden subsidy in many of these numbers is experienced engineers spending judgement to turn fast drafts into accepted changes. I’d measure accepted changes per review hour and the classes of defects that escape, rather than the share of code generated by AI.
Otherwise a team can report more AI output while quietly moving both the bottleneck and the risk into senior review.
Thanks, "the hidden subsidy is experienced engineers spending judgement to turn fast drafts into accepted changes" is the best one-line summary of the article I've seen. Accepted-changes-per-review-hour is a far better leading indicator than "% of code generated," and it maps onto the Fastly finding that just under 30% of seniors edit AI output enough to cancel out the time saved. The bottleneck moves into senior review, and you're right that most dashboards would record that as a win.
Accepted changes per review hour is useful precisely because it prices the judgement subsidy instead of celebrating generated volume. I would split it once more: waiting time, active review, rework after review and recovery after merge. Two teams can have the same ratio while one is constrained by reviewer availability and the other by low-quality drafts that keep coming back. Have you seen any data that separates those two failure modes rather than averaging them into one productivity number?
AI has gotten shockingly good at coding. So I'd say judging it like a 'mark bad quota' isnt the right approach. AI can write an entire codebase, as long as it's thoroughly tested and working, including error handling, then where's the problem? When I was working in a team who didnt use AI, I was the only one who actually wrote unit tests... Because I used AI, while they didnt. Sure, I had to correct alot back then, but that was fine, because in the end, my code was the only code tested in the 1.5m LOC codebase... Let that sink in, a 1.5m LOC ERP system, without any test coverage. They relied purely on 'click button, button click WOW it works' as opposed to empirical proofs that show it works under concurrency and it fails gracefully. So 'pre-AI' developers need to up their game and update their standards too.
I agree that the standard has to apply to both human and AI-written code. A pre-AI team with 1.5M LOC and no real test coverage is not the benchmark anyone should defend.
My concern is less about an AI “bad output quota” and more about who supplies independent evidence that the result is acceptable. Tests generated in the same loop are useful, but the highest-risk behaviors still need an oracle that can disagree with both the model and the developer: failure handling, concurrency, permissions, and rollback.
AI makes thorough verification cheaper too. That should raise the bar for everyone, not create a separate lower bar for AI-written code.
That's why I lean more towards deterministic rule-driven test generators. That way you spot a bug once and it's flagged across every repo you have. Essentially turning unit testing into a package, that generates tests that test not only the happy path, but the boundaries and fail states too.
Definitely agree on the oracle. Eg. with the pgrust to wasmer port I did, I use native postgres as an oracle (not just native pgrust), that way I can guarantee that original intended behavior is preserved. Helped me spot a bug nobody woulda found. ClockID in libc uses a pointer, vs pgrust's clockid uses an int, so while all tests printed green, a deep dive into the logs showed all 0 times. You'd think it's a packaging issue, but it was a standard library incompatibility, that needed an interpreter. So definitely, always use an oracle where you can!
I learned to code before AI chatbots were a thing. I built AI driven apps before AI had a way to code. That doesnt mean that I want to be judged on syntax and jargon... Pro-proof, in my opinion, is a real-world task, done competently. Not scrutinized till they cant answer the question, plain and simple proof of work. If it runs, it doesnt have bugs, if it has limitations that they're documented. Then that's sufficient, because that's all that actually matters day-to-day.
Thanks — I agree proof-of-work beats jargon, and a documented limitation is worth more than a clean-looking diff. The one caveat the data keeps hammering: the top frustration in the Stack Overflow survey (66%) is "almost right, but not quite," which is the exact case where it runs and still pages you three weeks later. So "if it runs, it doesn't have bugs" is the assumption the mid-career skeptics are pushing back on — and why they document and test the seam.
good ai overview
interesting approach