Friday afternoon. A ticket lands in your queue: new endpoint, some validation, a small migration. The kind of task that used to eat half your day.
You describe it to your AI assistant, skim the diff, run the tests, ship it. Twenty minutes, done.
Three months later that endpoint breaks under some load pattern nobody planned for. You open the file and you don't quite recognize it. You wrote it, technically. You accepted it, tested it, merged it. But you never built a real model of how it works, because the part that would have forced you to think it through got handled for you.
That raises a question: if AI writes more of our code, are we still learning how that code actually works, or just getting faster at shipping things we don't fully understand?
The productivity studies don't agree, and the disagreement is the useful part
In 2023, Microsoft, GitHub, and MIT Sloan ran a randomized trial where developers wrote a standardized HTTP server in JavaScript. The Copilot group finished 55.8% faster, and more of them finished at all, 78% versus 70%, though that completion-rate gap wasn't statistically significant on its own. It measured speed on a fairly contained task, not whether the resulting code held up over time.
Two years later, METR tested something closer to real engineering work. They recruited sixteen experienced open source developers and watched them complete 246 tasks in their own large, familiar repositories, some with AI access and some without. The result ran the other direction: developers using AI took 19% longer. Before starting, they'd predicted AI would speed them up. Afterward, they still believed it had, estimating a roughly 20% improvement that the data didn't support.
At first glance these look contradictory. But the studies tested different kinds of work. A contained task with an unfamiliar tool is where AI assistance tends to show up well. A codebase someone already carries in their head, built over years, is a much harder place for a generic tool to add value, and an easier place for review and correction overhead to eat whatever time it saves. The studies tell us something different about productivity in different settings. METR's result is the one worth sitting with: developers expected AI to save them time, and the measured result went the other way.
The quiz that tested what developers actually learned
A more direct test of the real question came from Anthropic, in a study of 52 developers, mostly junior, learning an unfamiliar Python library called Trio. One group had an AI assistant during the coding tasks. The other coded by hand. Afterward, everyone took the same quiz, no AI allowed, covering debugging, code reading, and conceptual understanding. It's a small study of people learning something new, not a representative sample of senior engineers working on production systems, but the results are worth examining.
The AI-assisted group averaged 50% on that quiz, versus 67% for the hand-coding group, a gap of 17 percentage points. The biggest difference between the groups showed up specifically on debugging questions. There's another important detail, though: the AI group only finished about two minutes faster on average, a difference too small to call statistically meaningful. They didn't even get much speed in exchange for the gap in understanding.
Looking back at screen recordings, the researchers sorted participants into six observational patterns of how people actually used the assistant. Three patterns, built around handing the task over and accepting what came back, averaged under 40% on the quiz. Three other patterns, where people asked why something worked or requested explanations alongside the code, scored between 65% and 86%. The researchers are careful that this is a correlation found after the fact, not proof that a given pattern of use caused the score. Still, the association is hard to ignore: people who stayed cognitively engaged with the assistant tended to retain more than people who didn't.
What shows up later, in actual codebases
GitClear analyzed roughly 153 million lines of changed code from 2020 through 2023 and projected that code churn, meaning lines reverted or rewritten within two weeks of being written, would roughly double by 2024 compared with its 2021 baseline. Copy-pasted code was also rising relative to properly refactored code. GitClear shows that churn increased during the years AI tools became common, but that doesn't prove AI caused it, and GitHub has pushed back publicly on that reading. One possible explanation is that developers are merging code before they've fully understood it, though the data alone can't establish that.
On security specifically, Stanford researchers found that developers with access to an AI coding assistant wrote measurably less secure code than a control group, and were more confident their code was secure. The tools have changed since that 2022 to 2023 experiment, so its results shouldn't be treated as a verdict on today's assistants. But it highlights a risk worth considering: developers can be confident in code they haven't fully evaluated.
The 2025 Stack Overflow survey, built on more than 49,000 responses, found 84% of developers either using AI tools already or planning to, a figure that mixes current users with people who haven't started yet. Among respondents overall, 46% said they didn't trust the accuracy of AI output, up from 31% the year before, and 66% named "almost right, but not quite" answers as their top frustration with the tools. Of those who answered the frustrations question, 45% said debugging AI-generated code was more time-consuming than expected. That's self-reported experience, not a controlled measurement of actual debugging time, but it lines up with the other findings here.
Where the tool genuinely helps
The Copilot study's speed gains were real, and some people in the Anthropic study using the exact same tool on the exact same task learned just as much as the control group. AI tends to help most with the parts of the job that are mechanical once you already understand them: boilerplate, syntax you've half forgotten, turning a plan you've already worked out into code, explaining an unfamiliar error faster than a search engine would. It's also useful as a second opinion, something to push back against your own design before committing to it. None of this says AI damages learning by default. It says learning depends on what a developer does while the tool is open, not just whether it's open.
Rate limiting: two ways to use AI
Picture two developers given the same task: add rate limiting to an API with an AI assistant available to both. This is a made-up scenario to keep the point concrete, not a documented incident.
The first developer asks for an implementation, gets a working solution using a sliding-window counter backed by Redis, confirms the tests pass, and ships it. Weeks later, under a burst of traffic, the Redis connection pool starts timing out. They're trying to understand the retry logic and connection-pool behavior while the service is degraded, without knowing whether the problem is in their implementation, their Redis configuration, or both.
The second developer asks the same initial question, then follows up. Why this approach over an alternative. What happens if Redis briefly drops. They try implementing a small piece by hand first, compare it against what the assistant produced, and only merge once they can explain why the retry logic is structured the way it is. The task takes longer. When the same Redis issue shows up later, they already have a working model of the failure, because they built one on purpose while it was still possible to.
Neither developer is permanently stuck this way. The first could still learn the retry logic properly before anything breaks, and plenty of developers do exactly that. What separates them isn't talent or diligence in some abstract sense. It's that one of them paid a small cost upfront, and the other is likely paying a larger one later, probably at a worse time.
A short list of things worth trying
Sketch your own approach before asking the assistant for one, even roughly. Ask for the reasoning behind a suggestion, not only the code. Review generated code like you're the one debugging it alone later, because you probably will be. Write tests that check actual behavior rather than confirming a demo ran once. When something breaks, spend a few minutes forming your own guess before asking the AI to fix it. Try explaining a piece of merged code out loud before you consider the task finished. The Anthropic findings offer a reason to take these habits seriously: the people who asked questions and worked to understand the code generally scored better than those who just let AI do the work.
The real variable
If AI saves you an hour, use some of that time to actually understand what it produced. If you just move to the next ticket, you might finish more work today without being any better prepared to debug it tomorrow.
Pick one piece of AI-assisted code you shipped this month. Could you walk a teammate through why it's built the way it is, without reopening the chat?
Top comments (0)