Sixteen senior open-source developers agreed to something most engineers would find uncomfortable: hand over their next few weeks of real work to a coin flip. Half their tasks would be done with AI tools. Half without. Every minute would be logged.
These weren't hobbyists. They averaged five years on the exact codebases they'd be working in, projects with over 22,000 GitHub stars and years of accumulated context in their heads. If anyone was positioned to use AI well, it was this group.
Before the study started, they predicted AI would cut their task time by 24 percent. After finishing, even having lived through the actual work, they still believed AI had saved them 20 percent. The stopwatch disagreed. They were 19 percent slower.
That's not a story about junior developers leaning on autocomplete they don't understand. It's a story about the exact profile the term "vibe engineering" was built to describe.
The Distinction Everyone Quotes, Few Apply Correctly
In late 2025, developer Simon Willison drew a line that spread fast through engineering circles. Vibe coding, he argued, is building software by prompting an AI and shipping whatever comes out, without reading it, without understanding it, without caring whether it's right. Vibe engineering is the opposite end of the same spectrum: experienced developers using tools like Cursor, Claude Code, or Devin to accelerate implementation, while staying fully accountable through testing, planning, code review, and disciplined source control.
The distinction is useful. The way most teams apply it isn't.
Somewhere between Willison's original post and a hundred LinkedIn recaps, "vibe engineering" quietly turned into shorthand for "an experienced engineer using AI." Seniority became the qualifying credential. If you'd been coding for a decade, the thinking goes, you were doing it responsibly by default. That's not what Willison described. He described a set of practices. Practices are something you do on a given task, not a status you carry into every one.
That gap between what the term means and how it gets used is where the METR study becomes relevant, and where most coverage of it gets the application wrong.
What the Study Actually Measured
The trial in question, run by AI research group METR between February and June 2025, was a randomized controlled trial, the same experimental design used to test drugs. Researchers recruited 16 experienced developers, gave them 246 real issues from their own repositories, and randomly assigned each issue to either allow or forbid AI tool use. When allowed, most developers reached for Cursor Pro paired with Claude 3.5 or 3.7 Sonnet, the frontier models at the time.
The gap between prediction, perception, and reality is the actual finding here, not the 19 percent on its own. These developers didn't just guess wrong going in. They kept believing the wrong thing after the work was already done. That's a harder problem than a bad estimate. It suggests experience doesn't automatically produce accurate self-assessment of AI's effect on your own output.
Access Is Not the Same as Practice
Here's where the study gets misapplied. METR's "AI-allowed" condition meant developers could use AI however they wanted. Nobody was required to write tests first, plan before prompting, or review every generated line the way Willison's vibe engineering describes. The trial measured what happens when experienced developers get access to AI tools. It did not measure what happens when they follow a specific disciplined practice around that access.
Access is not practice.
That distinction matters for anyone leading a team. A policy that amounts to "our engineers are senior, they'll use AI responsibly" is not vibe engineering. It's just access, with a job title attached as a substitute for process. Handing an experienced woodworker a table saw doesn't make the years of training show up automatically. The saw still requires the technique. Without it, an experienced hand can lose a finger just as easily as a beginner, sometimes more easily, because confidence outpaces caution.
The METR developers had the tenure. What the study suggests they didn't reliably have, in that six-month window with those specific tools, was a workflow that turned access into the disciplined outcome Willison's term promises.
The Study That Couldn't Be Repeated
There's a wrinkle here that most articles quoting the 19 percent figure leave out, and it surfaced almost a year later. In February 2026, METR published an update explaining why they were changing how they run this research. They'd started a follow-up experiment in August 2025 with a larger developer pool and newer tools, expecting to track how the effect changed over time. Instead, they found the new data unreliable. A growing number of developers were refusing to participate specifically because they didn't want to work without AI assistance, even temporarily, which skews any comparison between AI-allowed and AI-disallowed conditions.
Read that carefully, because the obstacle wasn't that AI got so good the slowdown vanished. It's that the population willing to go without it shrank enough to break the experiment's design. That's a different and arguably more interesting finding than the original 19 percent, and it's had a fraction of the coverage.
The practical takeaway: the original study is a snapshot of a specific six-month window, specific tools, and a specific group of 16 people. Treating it as a permanent verdict on experienced developers and AI is the same overreach as treating "vibe engineering" as a permanent status once you've hit five years of tenure. Both claims need re-proving under current conditions, not just cited from an older post.
What This Means for the Team You're Actually Running
If you manage engineers, or you're evaluating your own output, the practical question isn't whether your team is vibe coding or vibe engineering. It's whether anyone has actually checked. The developers in the METR study were confident, experienced, and wrong about their own speed, at the same time. That combination is the dangerous one, not inexperience on its own.
Teams that reward "ships fast with AI" in performance reviews without a pre-AI baseline to compare against are running the same experiment METR ran, just without the stopwatch. The senior engineer who says AI made them dramatically faster might be right. Or they might be describing the exact overconfidence sixteen experienced developers displayed under controlled measurement. Without a baseline, there's no way to tell which.
Vibe engineering isn't a rank you earn once and carry forward. It's a claim you have to keep proving, task by task, the same way a junior developer's output gets checked. The tenure doesn't remove that requirement. It just makes people less likely to ask.
Top comments (0)