AI in Software Engineering: Why Understanding Still Matters

Dietmar Schoder — Tue, 09 Jun 2026 17:53:25 +0000

The Limits of AI in Software Engineering: Why Understanding Still Matters

A software company delivers a product to its customers. Then a new government regulation appears, mandating changes to that software by a specific deadline. The company creates backlog items, breaks them into tasks, and developers start coding. The question is: how much AI should they use to write that code?

Three Levels of AI Adoption

Level one. The developer gives the AI access to the task description, the existing codebase, the version control system, and the development database. The developer then asks the AI to create a new branch, write the code change, and open a pull request.

Level two. The company also uses AI to read the new regulation, compare it with the existing system, write the backlog item, and break it down into smaller coding tasks. After the AI has handled the coding work from level one, the company asks it to perform the pull request review, merge to master, write test scenarios, run all tests, and deploy the change.

Level three. The company's own customers use AI to perform everything the software previously did for them.

How far is this a good idea? How realistic is this vision?

Comparing Human and Artificial Intelligence

To answer these questions, we must compare what human engineers do with what AI does. If we understand both, we can decide how much human work AI can realistically replace. At minimum, we can see what improves and what worsens as we replace more human work with AI.

So how do humans work? This question has generated fierce debate and endless scientific investigation. But for our purpose, a rough sketch of human cognition is enough. That sketch differs so fundamentally from how AI works that it alone reveals the answer.

The Human Mind: Building Models of Reality

A human being experiences the world. Whenever they want to understand something, they build a model of that domain. They construct this model as a system, meaning it has both structure and behaviour.

Take tennis. Anyone who knows what tennis means can easily create a mental model: two players with rackets, a ball, a net, a court with borders, and a barrier in the middle. These components represent the structure of the real-world system. Then the human adds serve, return, volley, fault, game, set, and match as behaviours or processes.

Every human has a model like this when they think of tennis, though not exactly the same model. And humans can do remarkably useful things with such models. They can observe a real tennis match and predict what happens next. They can refine the model when they learn new facts, such as that balls are yellow now instead of white. They can run mental simulations. What if there are four players? What if matches are too long and we introduce a tiebreak? They can even experiment with absurd hypotheticals. What if both players serve at the same time and must return the serves in parallel, meaning two balls are in play at once?

Some humans are better at this modelling than others. We usually call this skill "imagination," because we long believed that we hold images in our minds. But in fact, this skill is what human brains excel at, and it is what enables intelligence. Humans can invent mental models, explore them, use them for testing, adapt them, and constantly check how closely they match reality. Humans can write these models down, draw them, communicate them, and exchange enormously valuable knowledge through them.

The power of this ability cannot be overstated. Humans can listen to a fairy tale and, from listening alone with no further explanation, immediately establish what is valid in the fairy tale versus what is valid in real life. They can instantly recognize a fairy tale as such. They watch a film like Avatar, immerse themselves in a new world, and derive a new model of that world's rules. Humans are so good at this that they can detect flaws that break the rules in any fictional world, even when that world is complete fantasy.

Here is a certainty from daily life. Whenever someone has a highly realistic model of a particular domain in their head, we recognize that this person understands what they are talking about. Understanding is having a mental model of a real system, its structure and behaviour, that is close to reality. And it includes the ability to operate with this model in many useful ways, including learning.

How Large Language Models Work

Now, how does AI work? The AI the world is currently discussing consists of large language models. How do they work? During training, they read everything ever published on a given topic, such as tennis, on the internet. Then, when a user asks a question, the model searches through the vast mixture of words from its training and returns the most statistically likely words as an answer.

In other words, the AI selects human symbols based on probability, rearranges them, and returns them as an answer. Consider a simple example. Suppose we gain access to all written texts of an alien civilization, but we cannot translate them. The aliens send us questions. We do not understand anything they say. Nevertheless, we send back their own symbols, selected and arranged according to probabilities derived from their texts using a particular algorithm.

In this scenario, it is clear that we do not understand the aliens a little bit and gradually improve. No. We do not understand them at all, and we never will. And because we mirror their own texts back to them, the aliens will believe these are intelligent answers, as long as their texts are generally intelligent. This becomes even more impressive to them when a single user receives back the most matching answers from the entire alien knowledge base.

Even if this simple model of human intelligence and this simple model of LLMs are not remotely close to reality, one thing becomes crystal clear. Humans can understand portions of reality. LLMs cannot and never will.

The key difference is always what we call understanding. Understanding is the playful, joyful, realistic modelling of the world in the human mind. LLMs have none of this.

Implications for Software Engineering

Now we return to software engineering and AI. The full picture becomes visible immediately. If a software developer uses AI to gain a better understanding of their work, AI is certainly useful. But the more a developer or any user lets AI "do the work" as a black box, the more that user skips the understanding part. And AI will never replace that understanding.

Consequences follow directly. When the new code works in production, no one understands why it works. When it fails in production, no one understands why it fails either.

AI can then be used to fix the bug. But because the AI does not understand the software user's world, the new regulation, the existing code, or anything at all, it will likely fix some bugs while introducing new ones. AI will eventually encounter the same experience every software developer has when they do not truly understand what they are doing. The total number of bugs, vulnerabilities, and flaws in the software system will increase with every new attempt to fix it.

Conclusion

As long as humans use LLMs to gain more understanding, LLMs are helpful tools that make life easier for intelligent people. But the moment they are used to complete tasks faster by excluding humans, those tasks must not require the tiniest form of understanding. If they do, the work will be done with significantly lower quality. That will inevitably become costly and dangerous in the long run.

Leave a comment below. Let's talk.

DEV Community: Dietmar Schoder