After reading a book about large language models, I started looking at AI-generated code differently.
Every AI-generated function I had seen worked.
That was exactly why I became skeptical.
My experience with coding agents has been positive. The code looks correct, runs correctly, and helps me move faster.
The book made me think about what happens underneath.
Large language models generate code through prediction.
They are not deterministic systems in the simple sense that one plus one equals two.
That does not mean the generated code is automatically bad.
It means that successful output answers only one part of the question.
I still need to know whether the code matches the intent behind the request.
Why working code feels trustworthy
Software gives us strong feedback.
The compiler rejects some mistakes. Tests reject others.
A running application makes a generated change feel trustworthy.
That feeling is reasonable.
If the function compiles, passes its tests, and solves the immediate problem, what else should I check?
The problem is that tests cover only the behavior we wrote down.
They do not automatically reveal every unstated requirement, insecure assumption, or dependency that does not exist.
Evidence for usefulness and caution
AI coding systems can produce useful results.
GitHub has reported productivity improvements in controlled studies of Copilot users. Its research also discusses code-quality outcomes.
SWE-bench Verified evaluates coding agents against real GitHub issues and issue-specific tests. Passing those tests is meaningful evidence that an agent solved a task-level problem.
The evidence does not prove that the implementation matches every unstated intention. A test confirms the behavior it covers. It does not confirm that the code is secure, maintainable, or based on real dependencies.
A USENIX Security 2025 study examined 576,000 code samples from 16 code-generating LLMs. It found hallucinated package names, including packages that did not exist. The reported average was at least 5.2% for commercial models and 21.7% for open-source models.
Spracklen et al., “We Have a Package for You!”
A separate replication study examined security weaknesses in GitHub Copilot suggestions. Vulnerable suggestions decreased from 36.54% to 27.25% in the newer setup. The authors still concluded that the model suggested insecure code.
Tahir et al., “Assessing the Security of GitHub Copilot Generated Code.”
These findings do not cancel out the productivity evidence.
They describe different properties. AI-generated code can be useful and test-passing while still needing review.
The verification model I am looking for
The paper I found described a hybrid approach to formal verification.
When the AI makes an assumption, it should check that assumption and record where the supporting implementation exists.
Instead of reading a function name and inferring what it does, the system could point to the actual implementation.
It could leave a note or citation showing why the assumption is true.
That approach changes the question. I should not ask only whether the generated code looks correct. I should also ask what evidence supports its assumptions.
The useful output would be a patch with a trail of reasons that I can inspect.
What I think now
I still use AI coding agents.
My skepticism is not a conclusion that they are useless or unreliable.
I think prediction and verification are different jobs.
The model can propose code.
I still have to decide whether that code expresses the right intent.
That means running tests, reading the implementation, checking dependencies, reviewing security-sensitive operations, and comparing the result with the original requirement.
I have not personally experienced a coding agent hallucinating a function or producing an obviously broken implementation.
This concern grew from understanding how the system works and from published evidence, not from a dramatic failure in my own project.
For now, I will keep using AI-generated code. I will stop treating successful output as the end of verification.
Top comments (0)