TL;DR
AI coding assistants are excellent at producing code fast, but the research increasingly shows that speed does not automatically translate into better engineering, deeper understanding, or stronger long term skill. Several controlled studies now suggest that leaning on AI to generate code, rather than to interrogate your own approach, can quietly erode the reasoning ability that makes a developer effective in the first place. This post walks through what the data actually shows and lays out a different way to use these tools, one where AI works as a reviewer of your thinking instead of a replacement for it.
The Productivity Story Everyone Already Knows
The headline numbers around AI coding tools are impressive and widely repeated. GitHub Copilot now has 15 million users, a 400 percent increase in a single year, and is writing 46 percent of the average developer's code, reaching as high as 61 percent in Java projects. GitHub's own controlled trial found a 56 percent velocity gain on a complex coding task for developers using Copilot, and 85 percent of developers reported feeling more confident in the code they produced.
This is the version of the story most engineering teams have already internalized. AI writes code faster, developers ship more, everyone moves on. But a second, less publicized body of research complicates this picture considerably, and it is worth taking seriously before you build your workflow entirely around code generation.
The Slowdown Nobody Expected
In mid-2025, the AI safety research group METR ran a randomized controlled trial that produced a genuinely surprising result. Researchers recruited 16 experienced developers from large, mature open source repositories, each averaging over 22,000 stars and more than a million lines of code, and had them complete 246 real issues, half with AI assistance allowed and half without.
Before starting, developers forecast that AI would reduce their completion time by 24 percent. After finishing the study, they still believed AI had made them roughly 20 percent faster. The actual measured result was the opposite: allowing AI tools increased completion time by 19 percent.
The gap between what developers believed and what actually happened is the most important part of this finding. The slowdown stemmed largely from the time developers spent reviewing and correcting AI suggestions that were often directionally correct but not exactly what the task required. In other words, the AI produced plausible looking code fast, and the real cost showed up later, in the review and correction cycle that developers underestimated while it was happening.
This matters because it points to a pattern that shows up across the rest of the research too: AI is very good at producing output that feels helpful in the moment, and much less reliable at producing output that survives scrutiny.
What Happens When You Skip the Struggle
Speed and correctness are one axis of this problem. Learning and skill retention are another, and the evidence here is more concerning for anyone who treats AI as a default first step rather than a tool used with intention.
In early 2026, Anthropic published a study that randomized 52 mostly junior developers, all with at least a year of Python experience, into two groups learning Trio, an unfamiliar async library. One group worked with AI assistance, the other worked by hand. On an immediate comprehension quiz afterward, the AI assisted group scored roughly 17 points lower than the hand coding group, 50 percent versus 67 percent.
The broader finding echoed this: developers who used AI assistance completed tasks significantly faster, but demonstrated markedly lower skill retention and problem solving ability once the AI was taken away.
It is worth being precise about what this study does and does not claim. It measures skill formation in learners encountering something new, not raw productivity in experienced developers working in a stack they already know well, and a senior engineer shipping features in familiar territory is doing different cognitive work than a junior developer learning an unfamiliar library for the first time. But the underlying mechanism it points to, that generating a working answer is not the same as building the mental model behind it, lines up closely with what the MIT cognitive debt research found in writing tasks and what the METR study found in production codebases: the parts of the process AI skips over are often exactly the parts where understanding gets built.
Correctness and Security Take a Hit Too
It is not only learning that suffers when AI output goes unquestioned. Code quality and security research tells a similar story.
One controlled study found that developers given an AI assistant while solving cryptography problems produced substantially less secure code than developers working without assistance, and were more confident in those insecure solutions despite the lower quality.
Other controlled experiments have found similarly mixed results: one study reported that developers using Copilot failed tasks more often than developers using standard autocomplete, due to incorrect code, despite preferring the AI assisted workflow anyway. Preference and performance pulled in opposite directions, which is a pattern worth remembering any time a tool feels good to use.
Research has also found that the effectiveness of AI coding assistants depends heavily on a developer's existing expertise, since less experienced developers often lack the skills needed to properly evaluate whether an AI generated suggestion is actually correct. This is the uncomfortable core of the issue. The people who benefit most from having their thinking challenged are often the same people least equipped to catch it when the AI's suggestion is wrong.
Why This Keeps Happening
Across all of these studies, a consistent mechanism shows up. AI coding tools are optimized to produce something that looks complete and reasonable quickly. That fluency creates a strong pull toward acceptance rather than scrutiny. Developer surveys back this up directly: developers mainly use AI assistants to reduce keystrokes and recall syntax rather than to reason through a problem, and one large scale study found developers keep only 52 percent of AI suggestions after actually reviewing them.
That 52 percent figure is worth sitting with. Roughly half the time, a careful developer looks at what the AI produced and decides it is not good enough to keep. The question this raises is what happens the other half of the time, when a suggestion gets accepted quickly because it looks plausible rather than because it was actually verified.
A Different Way to Use These Tools
None of this is an argument for abandoning AI coding assistants. The productivity gains are real in many contexts, and outright avoidance is not a realistic strategy for most teams. The research instead points toward a specific shift in how these tools get used.
The core idea is simple. Instead of asking AI to generate the solution, ask it to interrogate your solution. This flips the tool from something that replaces your reasoning into something that pressure tests it, and it changes which parts of the thinking process you actually do yourself.
Write the approach first, then ask AI to critique it. Draft your function signature, your data model, or your algorithm outline before generating anything. Then prompt the AI specifically to find flaws in your approach rather than to produce its own version. This keeps the design reasoning with you, where the actual skill building happens.
Ask for edge cases, not implementations. A prompt like "what inputs would break this function" produces a very different cognitive workout than "write this function." The first forces you to think through your own code more rigorously. The second replaces that thinking entirely.
Treat every accepted suggestion as a code review, not a shortcut. Given that developers keep only about half of what AI suggests after review, build that review step into your process explicitly rather than trusting fluency as a proxy for correctness.
Use AI to explain unfamiliar code, not to avoid learning it. When you hit a library or pattern you do not know, ask AI to walk through why it works, then implement a version yourself. This preserves the comprehension gains the Anthropic study found were missing in the AI assisted group.
Be honest about the time you are actually spending. The METR result shows developers can be wrong about their own speed by a wide margin. Track review and correction time explicitly for a few weeks rather than relying on how fast the workflow feels.
The Bigger Shift
The pattern running through all of this research is not that AI produces bad code. Often it produces code that works. The pattern is that offloading the reasoning behind the code, the part where you consider edge cases, weigh trade offs, and build a mental model of why something works, comes with a cost that does not show up immediately. It shows up later, in comprehension quizzes, in security audits, in the time spent correcting suggestions that looked right at first glance, and in the moment a junior developer has to debug something they never actually understood.
The fix is not to stop using AI. It is to stop treating it as a code generator and start treating it as a sparring partner. Ask it to poke holes in your design. Ask it what could go wrong with your approach. Ask it to explain the unfamiliar thing instead of writing the unfamiliar thing for you. The code still gets written either way. The difference is whether you understand it when you are done, and whether you will still understand it three months from now when it breaks in production and the AI is not there to fix it for you.
Top comments (0)