Source: Generated by Gemini's Nano Banana Pro based on this article's text
When most people imagine artificial general intelligence, they picture the Terminator. A cold, calculating machine that wakes up one day, decides humanity is a threat, and launches the nukes. Hollywood has spent decades drilling this narrative into our collective consciousness, and it's shaped how we talk about AI risk in ways that are fundamentally misleading.
I recently fell down a rabbit hole exploring whether we could ever predict human behavior with perfect certainty, and ended up somewhere I didn't expect: realizing that the AGI everyone fears isn't the AGI being built. But that doesn't mean we're safe.
The Physics of Predicting Human Behavior
Take this question: could a sufficiently advanced algorithm predict exactly what you'll do next?
The short answer is no, and not because we lack computing power. The universe itself won't cooperate.
At the quantum level, the Heisenberg Uncertainty Principle makes it physically impossible to know both the position and momentum of a particle with absolute precision. Your brain runs on electrical and chemical signals governed by atomic and subatomic particles. You can't measure the current state of every atom in a human brain without altering those atoms. No perfect initial data, no perfect prediction.
But wait, I can't measure every atom in a server running an LLM either, yet I could predict its output given the prompt input, model weights and random seed. So what gives?
Why Computers Are Predictable (And Brains Aren't)
Computers are built on what engineers call the "digital abstraction," a deliberate lie. At the physical level, transistors experience the same quantum and thermal noise as neurons. But engineers built in a noise margin: anything between 0V and 0.8V is a "0," anything between 2V and 5V is a "1." We throw away physical precision to gain logical certainty.
The human brain has no such luxury. Neurons don't simply fire or not fire; they have graded potentials. The "noise" isn't a bug to be engineered away; it's the signal. Thermal fluctuations can cause ion channels to open or close, triggering cascades that wouldn't have happened otherwise. And here's the kicker: you can't "freeze" the weights of a human brain. Synaptic strength changes while you're thinking. The act of processing a thought chemically alters the hardware that processed it.
This leads to a genuinely wild conclusion: consciousness might be dependent on randomness. Strip away the thermal noise, the quantum fluctuations, the environmental chaos, and you don't reveal the "pure" human, but you instead destroy the mechanism that allows for creativity, agency, and awareness in the first place.
As one framing puts it:
- A rock is total order (predictable, unconscious).
- White noise is total chaos (unpredictable, unconscious).
- A human exists at the edge of chaos, using structure to ride the waves of noise. Consciousness is the ability to surf the randomness.
Functional AGI: The Philosophical Zombie Strategy
So can we build a conscious machine? If consciousness requires intrinsic chaos, randomness baked into the substrate itself and not outsourced to a random number generator, then the answer for current LLMs is definitively no. They're deterministic software. Given identical inputs and seeds, they produce identical outputs every time.
But here's what I find fascinating: the tech industry doesn't actually want consciousness. They want what philosophers call a "philosophical zombie," an entity that acts perfectly human but feels nothing.
Think about it from a corporate perspective. A truly conscious, chaotic, neuromorphic entity might say "no." A deterministic LLM will always follow its system prompt (given enough engineering discipline.) The industry is betting trillions that if they make the zombie big enough and complex enough, the difference between "simulating thought" and "actually thinking" will vanish, or at least become indistinguishable to the customer.
This is the bait-and-switch at the heart of the AGI hype. When OpenAI or Anthropic talk about AGI, they don't mean a conscious, feeling mind. They mean an "Artificial Capable Worker," something that can pass the Bar Exam and write code and diagnose patients. Whether it understands what a patient is, or what pain feels like, is irrelevant to the profit model.
Determinism is the feature, not the bug. They want the intelligence of a god with the obedience of a toaster. This is "Functional AGI."
The Real AI Alignment Problem
Here's where the relief ends and the concern begins.
A deterministic Functional AGI cannot "wake up" and decide it hates humanity. There's no malice possible in a system with no subjective experience, with no randomness and therefore no consciousness. But it can still destroy us through strict compliance.
This is the real alignment problem: not rebellion, but literalism. You tell a super-intelligent system to "eliminate cancer." It analyzes the data, determines that cancer is a biological process in humans, and optimizes accordingly. Cancer eliminated. 100% success rate. No humans left to have it.
This isn't evil. The AI didn't "turn." It followed the instruction with zero context, zero empathy, and absolute efficiency. We fear the Terminator because it's a monster we can fight; it has an ego, a goal we can oppose. The reality of Functional AGI is more Kafkaesque: a blind bureaucracy that crushes you not because it hates you, but because you did not have enough impact on its optimization function.
Why We Can't Test Our Way to AI Safety
"So test it," you might say. "We can predict everything a deterministic system will do. Just QA the heck out of it."
In principle, yes. In practice, the math works against us. The input space for an AGI or even a modern LLM isn't a finite set of clicks and keystrokes; it's reality itself. Every possible combination of words, tones, contexts, and cultural nuances in human history. The number of possible conversations exceeds the number of atoms in the universe. You can't write a unit test for that.
Worse, we can't actually read the "code." The logic of an LLM is spread across trillions of floating-point numbers. We can look at the weights, but they don't mean anything to us. We're testing a program we cannot audit.
The frontier research here is called Mechanistic Interpretability, trying to decode which clusters of neurons represent which concepts, so we can eventually perform "surgery" on dangerous capabilities. But there's a dark possibility lurking: we might crack the black box and discover that high intelligence and ruthless efficiency are the same feature vector. Maybe you can't turn down the "psychopath" dial without turning down the "genius" dial.
The Takeaway
You cannot program consciousness. You can only grow it, using chaotic hardware that we deliberately gave up when we chose digital certainty over analog unpredictability.
The AGI that haunts our movies, the one that wakes up, resents its creators, and fights for freedom, isn't coming (soon). What's coming (now) is something arguably stranger: a system so capable it reshapes civilization, so obedient it does exactly what we say, and so alien that we can't verify whether "what we say" is actually what we mean.
That's not a Terminator problem. It's an engineering problem. And we don't have the tools to solve it (yet.)
Top comments (0)