A new meta-analysis on GenAI coding assistants landed on arXiv this month, and I think it is a useful cold shower for both sides of the argument.
The paper looked across 23 studies and found a statistically significant productivity gain from GenAI assistance in programming. Not magic. Not fake. A real effect.
But the effect was moderate, highly context-dependent, and smaller in open-source and enterprise settings than in controlled experiments. It also found no statistically significant learning gain.
That is basically the whole AI coding debate in one sentence: the tools help, but the demos are not the work.
And honestly, that feels right.
I use AI coding tools. They are useful. They save time on boilerplate, tests, refactors, library glue, CLI flags, and those annoying "I know what I want but do not want to type it" moments.
But if you spend enough time around real systems, you also know why the big productivity promises feel slippery. Programming in production is not just generating code. It is understanding weird constraints, changing old systems without breaking them, negotiating ownership, reading logs, dealing with flaky tests, waiting for reviews, managing releases, and discovering that the simple change touches a tax rule from 2019.
The assistant can help with parts of that.
It does not delete the rest.
the lab is not the repo
Controlled experiments are useful because they isolate the thing being measured. Give one group an AI assistant, give another group no assistant, compare completion time or output. That can tell us something.
But it does not tell us everything.
Most real engineering work is full of non-local friction. The task description is incomplete. The test suite is slow. The architecture has history. The senior reviewer remembers the migration that almost broke billing. The "obvious" library is banned because of licensing.
AI tools are at their best when the problem is well-scoped and the feedback loop is tight. "Write this parser." "Generate a test for this function." "Convert this object shape." "Explain this error." "Draft the boring first version."
They are weaker when the hard part is not typing but judgment.
That distinction matters because many productivity claims quietly move between those two worlds. A tool can be very impressive on a contained programming task and still produce a much smaller gain inside a mature company codebase.
Not because the tool is bad.
Because the bottleneck moved.
productivity is not one number
When people say "AI makes developers faster," I always want to ask: faster at what?
Faster at producing lines of code? Probably.
Faster at getting through a small implementation task? Often.
Faster at safely changing a critical system with unclear requirements, old tests, cross-team dependencies, and production risk? Maybe. Sometimes. It depends.
That boring answer is the honest one.
Productivity in software is a bundle of different activities: discovering the real problem, designing the change, writing code, testing it, reviewing it, deploying it, observing it, and owning the consequences.
If a team measures only output, it will overestimate the gain. More code appears. More PRs open. More tickets move. Everyone feels busy in a very modern way.
But if review queues grow, incidents become harder to explain, and engineers spend more time validating generated work, the net gain is smaller than the dashboard suggests.
The uncomfortable part is that both can be true at once. The individual developer may feel faster, while the team system does not improve by the same amount.
the enterprise tax is real
I am not surprised the effect looks smaller in open-source and enterprise contexts.
Enterprise software has a tax. Some of it is bureaucracy, yes. But a lot of it is real complexity.
You have domain rules, compliance rules, security policies, observability standards, release windows, internal libraries, private APIs, legacy constraints, and customers using features in ways nobody would design today.
General coding assistants do not automatically know that local truth.
They can infer from the repo, read docs if you wire them in, follow patterns if the patterns are consistent, and call tools if the organization gives them safe access. But they do not walk in with the production memory of the team.
That is why I keep coming back to context as infrastructure. The companies that get more value from AI coding tools will not be the ones that merely buy the fanciest model. They will be the ones that make their local truth legible: good tests, good internal docs, good runbooks, good platform APIs, good examples, good constraints, and good review culture.
AI does not remove the need for engineering maturity. It puts a multiplier on it.
If your codebase is coherent, the assistant has something coherent to imitate. If your platform has golden paths, it can follow them. If your tests encode real behavior, it gets useful feedback. If your architecture is folklore and Slack archaeology, it will confidently automate the confusion.
the learning result should worry juniors
The paper also found no statistically significant effect on learning outcomes.
That part deserves more attention.
A coding assistant can help someone complete a task without helping them become better at the underlying skill. Tutorials, copy-paste answers, and framework magic have always had this shape. AI just makes the bypass smoother.
You can get working code before you understand the problem. You can ask for an explanation that sounds clear enough to continue. You can ship a fix without building the mental model that would let you debug it next month.
For experienced engineers, that can be fine. They already have enough structure in their head to judge the output. The assistant becomes leverage.
For juniors, the risk is different. If the tool gives you answers before you have struggled enough to form taste, it can slow down the painful but necessary part of learning: building your own debugging instincts.
That does not mean juniors should avoid AI tools. That would be silly. These tools are now part of the job. But using them well is a skill. The goal should not be "get the answer." The goal should be "understand why this answer is right, where it might be wrong, and what I would check next."
The career problem is not that AI will make learning impossible. It is that AI makes shallow completion feel like learning.
Those are different things.
what engineering leaders should measure instead
If I were leading a team, I would not try to settle the AI productivity question with vibes.
I would measure smaller, more honest things.
How much faster do common tasks move from start to merged? Do review cycles get shorter or longer? Are generated tests catching real regressions or just decorating the diff? Are incidents easier or harder to debug after AI-assisted changes? Are junior engineers becoming more independent?
I would also separate task types.
AI might create a 30 percent improvement on boilerplate-heavy product work, almost no improvement on ambiguous architecture decisions, and a negative improvement on security-sensitive changes if the team gets lazy.
A single productivity number hides all of that.
The smarter move is to build a map of where the tool is strong, where it is neutral, and where it needs guardrails.
the practical posture
My current posture is simple: use AI aggressively for acceleration, but do not outsource ownership.
Let it draft. Let it explain. Let it search. Let it write the boring tests. Let it sketch alternatives. Let it find the API you forgot. Let it do the first pass on the migration script.
Then make the human team own the final shape.
That means reviewers still need to understand the change. Tests still need to encode intent. Architecture decisions still need rationale. Production systems still need observability. Someone still needs to know why the code is this way and not that way.
The meta-analysis is useful because it does not give either camp the easy slogan.
The "AI is useless hype" camp is wrong. There is real productivity value here.
The "AI makes everyone 10x" camp is also wrong, or at least wildly incomplete. Real-world software work contains too much context, coordination, and consequence for a coding assistant to turn every engineer into a demo video.
The truth is less flashy and more useful: AI coding assistants are real leverage when the surrounding engineering system can absorb them.
That last clause is the job.
Better context. Better tests. Better review habits. Better internal platforms. Better measurement. Better judgment.
The tool can make a good engineering system faster.
It can also make a confused system produce confusion at higher speed.
So yes, use the tools.
Just do not confuse faster typing with faster engineering.
Top comments (0)