What If AI Works Too Well?

Ewanderson Flaviano — Sun, 20 Sep 2026 23:46:50 +0000

How We Could Build a Civilization We No Longer Know How to Rebuild

For centuries, anyone could look at the symbols carved into the walls, temples, and monuments of ancient Egypt. They were still there. They had not been destroyed. They had not been erased. They were not hidden. And yet the tradition that could read them had disappeared. The hieroglyphs had survived, but the chain of people capable of understanding them had not. The information remained physically preserved while the human ability to interpret it was lost.

What was needed was a bridge. The Rosetta Stone contained essentially the same decree written in different scripts. One of them, Ancient Greek, could still be understood. By connecting what was no longer readable to something that was, scholars were gradually able to recover the meaning of a writing system that had remained in plain sight for centuries.

There is a distinction in that story that may become increasingly important as artificial intelligence advances: Preserving information is not the same as preserving the ability to understand it.

Much of the discussion about the risks of artificial intelligence focuses on the possibility that we may eventually lose control over increasingly capable and autonomous systems: systems that might deceive their operators, circumvent restrictions, or pursue goals in ways we did not anticipate. Perhaps that problem will materialize. Perhaps it will not. But there is another scenario that interests me.

In this one, nothing goes wrong.

Artificial intelligence keeps getting better. It becomes extraordinarily good at programming, medicine, engineering, mathematics, writing, scientific research, and countless other tasks. It does not rebel. It does not escape from a laboratory. It does not develop some mysterious instinct for self-preservation. On the contrary. It does exactly what we ask it to do. And that is precisely why we begin to depend on it.

There would be nothing entirely new about dependence of this kind. The history of technology is, to a large extent, the history of abstraction. Very few modern programmers need to understand the electrical behavior of the transistors executing their code. A developer can build an application without deeply understanding the operating-system kernel, the compiler, the physical details of networking, or the industrial processes used to manufacture the processor underneath it all.

This is not a failure of civilization. It is one of its greatest achievements. Abstraction allows one person to begin where another person stopped. We do not need to rediscover electricity before designing a computer, or reinvent algebra before learning calculus. We build new layers on top of previous ones, and that is how technological civilization advances.

But there is an important property of this system that is easy to overlook. Even though no single person understands every layer, there are still human beings inside each of them. I do not know how to design a modern processor. Other people do. They may not know how to build every machine used to manufacture that processor, but other people know those things.

Knowledge is fragmented and distributed, but it still exists within human society. More importantly, there are paths by which new human beings can learn it. Artificial intelligence may introduce something different.

Imagine a programmer in 2030 working on a complex system. They use an AI system to implement an especially difficult component. The result works. It is tested, documented, and deployed. The programmer may even understand quite well what the AI produced.

Five years later, someone else needs to add a feature on top of that component. By then, a new generation of AI tools can modify the system without requiring that developer to master all of its internal details. Twenty years later, another team works several layers above it. Seventy years later, some evolved descendant of the original component may still exist, surrounded by decades of abstractions built by both humans and machines.

The important point is that nobody ever decided to forget how it worked. Perhaps no individual lost knowledge they once possessed. Something subtler may have happened:

Later generations simply never needed to acquire it.

An expert alive today can remain an expert for the rest of their career. The problem begins when we ask who will replace them. Experts do not appear spontaneously. They are produced through years of practice, mistakes, boring tasks, failed attempts, repetition, and direct contact with the problems they will eventually learn to solve.

There are already smaller precedents for this kind of shift. In computing, earlier generations of programmers had to interact much more directly with memory, computer architecture, and low-level languages. Today, it is entirely possible to build sophisticated applications using high-level languages, mature libraries, and cloud services without ever writing an assembly instruction or manually managing the machine beneath them. That is not a catastrophe.

The lower layers still exist. There are still engineers who understand processors, compilers, operating systems, and networks, and there are still paths by which someone can become one of them. A programmer who starts higher up the stack can, if necessary, climb downward.

Navigation provides an even more revealing example. Satellite navigation transformed a task that once required astronomical observations, specialized instruments, tables, and calculations into something almost instantaneous and invisible. In the United States Navy, celestial navigation gradually disappeared from parts of naval education. The Naval Reserve Officers Training Corps stopped teaching it in 2000, and the U.S. Naval Academy removed it from its curriculum in 2006.

Then, in 2015, the Naval Academy brought it back. The reason was not that GPS had failed as a technology. Quite the opposite. Satellite navigation worked so well that the older skill had become unnecessary in ordinary use. The concern was resilience: what would happen if electronic navigation were disrupted or unavailable?

There is an important lesson in that episode:

A capability can stop being economically necessary while remaining strategically necessary.

Artificial intelligence may extend this phenomenon far beyond a single skill. The risk is not that a new generation stops learning one particular technique. Human history is full of techniques that became obsolete, and usually for good reasons. The more interesting possibility is that AI could shorten the training paths across many layers of society at the same time — including the paths that produce the people capable of maintaining, explaining, and rebuilding the abstractions beneath us.

An experienced programmer probably spent thousands of hours writing code that an AI system can now generate in seconds. An engineer learned not only by studying correct designs, but also by producing bad ones, watching machines fail, and understanding why they failed. A physician develops judgment not only by reading medical literature, but by encountering cases that do not look quite like the textbook. A scientist learns which ideas are promising partly by pursuing many that are not. Much of what we call work is also training for the next generation of experts.

When we automate a task, then, we may be removing more than labor. We may also be removing one of the steps through which someone learns how to perform it.

There is already some evidence that AI changes the way people distribute cognitive effort. Studies of knowledge workers using generative AI have reported shifts away from direct execution and toward verification, integration, and supervision. Other researchers have argued that AI-driven deskilling should be understood not only as an individual problem but as a structural one: environments that automate too much of a skill may eventually stop providing the conditions under which that skill is developed.

None of this proves the larger scenario described here. The claim that an entire civilization could eventually lose the ability to reconstruct important parts of itself is an extrapolation. But the mechanism that would begin that process is not purely hypothetical.

At first, the change would be almost invisible. Experts trained before widespread automation would remain in their professions. Eventually, they would retire. A new generation would be better at using modern tools but have less direct contact with the layers beneath them. The generation after that might have even less. Nothing would need to disappear suddenly. The chain could simply become longer, more indirect, and increasingly dependent on the machines themselves.

Until one day we might discover something strange: Our civilization still knows how to operate its systems, but no longer contains enough people who know how to reconstruct them from the layers below.

That does not necessarily imply collapse. A civilization in that condition might continue functioning for decades. Perhaps centuries. Perhaps much longer.

Medicine could keep improving. New materials could keep being discovered. Software could continue to be produced. Bridges could continue to be designed. Factories could continue to operate. AI systems themselves might maintain, repair, and improve the infrastructure on which we depend. The vulnerability might remain invisible precisely because everything keeps working.

But there is a profound difference between a civilization that knows how to use what it has and a civilization that knows how to rebuild what it has.

An obvious objection is that we can simply preserve all of our knowledge. We can store books, scientific papers, source code, engineering diagrams, datasets, models, manuals, and documentation. But that may not be enough.

A manual can explain how to operate a machine. Who builds the machine? Another document may explain that. But building it requires certain components. Who makes those components? They require another industry, other tools, and other kinds of expertise. Those tools depend on still other tools. Follow the chain far enough and something becomes clear: a technology is not simply a collection of instructions. It is the result of a chain of previous capabilities.

And even that chain of documents may be insufficient. Some important forms of human competence have never existed entirely in manuals. They live in practice: recognizing that a machine sounds wrong, distrusting a result that appears correct, knowing which approximation is likely to work, improvising when the expected tool is unavailable. This is often called tacit knowledge. Preserving a description of an activity is not necessarily the same thing as preserving the ability to perform it.

In that sense, a vast library is not necessarily a backup of civilization. A backup is useful only if it can be restored.

Perhaps, then, a society increasingly dependent on artificial intelligence needs to think about building its own equivalents of the Rosetta Stone. Not one literal stone. And not merely an archive containing everything we know. We would need to preserve paths.

From abstraction to implementation. From implementation to the tools required to produce it. From those tools to the knowledge required to build them. And from that knowledge to the methods by which new human beings can learn it. What we would be preserving is not just information, but a route back down through our layers of abstraction.

That might involve documentation deliberately written for human understanding, simplified reference systems, educational curricula, workshops, preserved equipment, and communities that continue practicing certain capabilities even after those capabilities are no longer economically necessary. In other words, we may need something like a restorable backup of civilization.

We do not yet know what such a system should look like. Perhaps that is exactly the question we should begin asking.

The Rosetta Stone did more than preserve symbols. It preserved a bridge between something we had stopped understanding and something we could still understand.

If we continue building abstractions on top of abstractions, we may need to make sure that similar bridges remain between us and the deeper layers of our own civilization. Because one day all of our hieroglyphs may still be perfectly preserved. And the real problem may be that there is no one left who knows how to read them.

References and further reading

DEV Community: Ewanderson Flaviano

What If AI Works Too Well?

References and further reading