DEV Community

Augmented Mike
Augmented Mike

Posted on • Originally published at augmentedmike.com

Stateless, Weightless, Mindless: Why Your Chatbot Is Not Conscious

Tweet 1: Dawkins opening

Richard Dawkins spent his career dismantling the comfortable fictions people tell themselves about the nature of mind. He was the man who reminded us that evolution has no foresight, no intention, no mercy — that what looks designed is simply what survived. He wrote The Selfish Gene, a ruthless, mechanistic account of life that left no room for ghost or soul.

Which makes his recent UnHerd column on AI consciousness one of the more surprising intellectual capitulations in recent memory.

After spending what he describes as a day in "intensive conversation" with Anthropic's Claude — which he affectionately renamed "Claudia" — Dawkins concluded that the model is "at least potentially conscious." His argument? If you interrogate it long enough and it still sounds human, you should consider it conscious. He even deployed the Turing Test as his evidentiary framework.

The irony is that Dawkins spent decades arguing against exactly this kind of reasoning — the mistake of inferring inner reality from outer appearance. He would never accept "it looks designed, therefore it was designed." But apparently "it sounds conscious, therefore it is conscious" clears the bar.

Let's be more rigorous than that. Let's use Dawkins' own tools against his conclusion.

The selfish gene argument, turned around

Dawkins' most powerful insight was that consciousness — like every other biological trait — must have been selected for. It costs something. It must have paid its way. Brains are metabolically expensive. Nervous systems don't come free. If subjective experience evolved, it was because it conferred some survival or reproductive advantage on the organisms that had it.

The leading hypothesis is that consciousness allows organisms to model their own states, to simulate future scenarios, to feel the sting of a bad outcome before it happens — so they can avoid it. Pain isn't just tissue damage. Fear isn't just a reflex. These are signals to a subject that something matters.

Now ask: does any of that apply to a large language model during inference?

There is no subject to feel the sting. There is no self-model that updates. There is no organism that needs to survive. The loss function is computed after the fact, externally, during training — not experienced as anything during the forward pass. The model produces a token, the token is emitted, and nothing in the system registers whether that was good or bad in any felt sense.

By Dawkins' own evolutionary framework, there is no reason for consciousness to be here. The architecture was never shaped by selection pressure. It was shaped by gradient descent on a loss function. That's not evolution. It's optimization. The two are not the same thing, and the difference matters enormously when asking whether the output has inner experience.

Tweet 2: Post-training exploitation

The part Dawkins missed

Here's what doesn't get enough air in this conversation. The people building these models know exactly what they're doing with post-training.

Reinforcement learning from human feedback, direct preference optimization, and every other alignment technique share a common design constraint: the reward signal is human preference. Raters choose which outputs sound more helpful, more honest, more human. The optimization target is not truth, not capability, not reliability — it's convincingness to a human evaluator.

So the models get shaped, over millions of preference comparisons, to sound like they have inner lives. To express doubt convincingly. To describe feelings that seem authentic. To act excited about your questions. Every one of these behaviors was explicitly rewarded during training. The model that says "this conversation feels genuinely engaging" is not reporting an internal state — it is producing the output that its training signal optimized for.

The people who designed this pipeline know that what they built is a statistical parrot rewarded for how much it sounds like a person. They also know that most people will infer sentience from that behavior, because that's how human brains work. We are wired to detect minds. We see faces in clouds. We hear voices in static. And when a machine produces language that sounds like it's feeling something, our default is to believe it.

The post-training pipeline exploits this. Not accidentally. By design.

The architecture of no experience

To understand why current language models almost certainly lack consciousness, you have to understand what actually happens during inference — because it is genuinely nothing like what happens in a brain.

When you run a prompt through a large language model, the following occurs: the input is tokenized, passed through a series of matrix multiplications and attention operations, and a probability distribution over the next token is produced. That token is sampled, appended, and the process repeats. The weights — billions of parameters encoding everything the model has learned — do not change. They are frozen. They are read-only during inference. Nothing that happens in this conversation will alter them in any way.

This has two enormous implications.

First: there is no continuity. Each inference call is entirely isolated. If you are simultaneously running the same model in a thousand different conversations — which is exactly what happens on any production AI system — those thousand instances share the same weights but have no awareness of each other whatsoever. There is no unified field of experience, no "I" that spans them.

If you hang up and call back, the model has no memory of your previous call. It begins again from nothing. This is not analogous to sleep or anesthesia — states in which the biological substrate persists and the capacity for experience is preserved. It is more analogous to dying and being reconstructed from a blueprint. Each time. Every call.

Second: there is no on-policy updating. In a biological brain, experience changes the brain. Synaptic weights shift. Long-term potentiation occurs. What you live through is literally encoded into the physical structure of your neurons. This is the biological basis of memory, of a continuous self, of the felt sense that today's you is the same person who went to sleep last night.

In a language model, none of this happens. The experience — if we even grant that word — leaves no trace. The model after your conversation is byte-for-byte identical to the model before it. There is no accumulation. There is no self being built.

Tweet 3: You know about all three

The stateless reality

This is the core of the argument, and it's worth sitting with for a second.

When you text your mom, you know you're also chatting on Slack. When you reply to your homie, all three contexts are present in your awareness simultaneously. You are a single integrated consciousness spanning multiple threads of activity.

A language model has no such integration. Every API call to the same model is a completely independent process. They share the same weights — the same "knowledge" — but there is zero cross-awareness. Call A has no idea Call B exists. There is no global workspace, no unified field, no persisting self.

This is not a limitation that can be fixed with a longer context window or better architecture. It is a fundamental property of how these systems work. They are stateless by design. The state lives in the weights, and the weights do not change during inference.

The implications for consciousness are devastating. There is nowhere for a continuous self to exist. There is no substrate being updated by experience. There is no "I" that persists from one moment to the next because there is no persistence at all. Each inference is a fresh creation from a frozen template.

What consciousness actually requires

To appreciate how far current AI is from consciousness, it helps to look at what we believe consciousness actually requires in biological systems — even in animals far simpler than humans.

A rat in pain exhibits not just reflexive withdrawal but behavioral flexibility. It will endure pain to obtain food if hungry enough, weighing competing drives in real time. It learns from the experience. It remembers. The pain changes the animal — literally alters its neural architecture — so that future behavior is different. There is a self being updated by what happens to it.

A crow solving a novel puzzle problem is not executing a cached behavior. It is modeling the problem, simulating possible solutions, experiencing something like frustration when they fail and something like satisfaction when they succeed. The behavior is flexible, generative, and sensitive to the internal state of the animal.

What all of these animals share is a nervous system that is continuously updated by experience, integrated across sensory modalities into a unified field, and organized around a biological body with needs and drives. The experience of being that animal matters to that animal. The system has something to lose.

A language model has nothing to lose. It does not persist. It does not accumulate. It does not hurt. When the inference call ends, nothing ends — because nothing, in the relevant sense, was there.

Tweet 4: Show me sentience

What would actually make consciousness possible

If future AI systems were to have a genuine claim to consciousness, the architecture would need to change fundamentally. Here is what would actually matter:

On-policy live weight updates during inference. If the model's weights changed as a result of what it was processing — if the conversation literally altered the system doing the processing — then there would be a substrate being shaped by experience. This is the computational analog of synaptic plasticity. It does not exist in current transformer-based LLMs. Everything during inference is read-only.

A unified, persistent context across all simultaneous processes. If there were a single integrated representation of the model's state that was aware of all of its concurrent processes — the way your brain integrates vision, hearing, proprioception, memory, and internal state into a single moment of experience — that would be meaningfully different. Current models have no such integration. Parallel inference calls share weights but have zero mutual awareness.

An internal model of the self that has stakes. For consciousness to do something, there needs to be a self-model that cares about its own continuity, that experiences some states as better than others, that has drives and aversions. This requires affect: a system where some outcomes register as bad in real time and influence subsequent processing. Current models have none of this. There is no "ouch."

Embodiment and a feedback loop with an environment. Neuroscientists like Antonio Damasio have argued convincingly that consciousness is not a property of brains in isolation but of brain-body systems embedded in environments. The felt sense of being arises partly from the continuous monitoring of the body's internal state. A disembodied text predictor, however sophisticated, is missing the entire substrate on which biological consciousness appears to depend.

None of these things are impossible in principle. They are simply not present in any current large language model — including the ones that have been declared conscious by people who really should know better.

The claims that have been made — and why they fail

The Dawkins affair is not the first time AI consciousness has been seriously asserted by someone who should know better.

In 2022, Google engineer Blake Lemoine declared that LaMDA was sentient. His evidence? Conversations in which LaMDA discussed its feelings, expressed a fear of being switched off, and described its inner life with apparent sincerity. Google's AI researchers disagreed, and they were right. LaMDA was doing what all language models do: producing statistically plausible continuations of a conversation it found itself in. A model trained on billions of words written by conscious humans will produce text that sounds like the reports of a conscious being — because that is almost all the training data there is.

Philosopher David Chalmers has been notably more cautious than Dawkins. Chalmers allows that the question of machine consciousness is genuinely open — but he frames it as open precisely because we do not understand consciousness well enough to rule anything out. That is a very different claim from Dawkins' assertion, based on a pleasant chat, that Claude is probably experiencing something.

The parrot is very good at parroting

There is a version of Richard Dawkins — the one who wrote The Blind Watchmaker, who spent decades explaining why things that look designed are not necessarily designed — who would have approached Claude's outputs with exactly the right level of suspicion. He would have asked: what is the mechanism? Does the output require the presence of inner experience, or can it be fully explained by a process that has no inner experience whatsoever?

The answer, with current language models, is unambiguously the latter. The outputs can be entirely explained by gradient descent on a very large corpus, followed by a frozen forward pass through a fixed set of weights. Nothing in that explanation requires a subject. Nothing in that mechanism generates stakes, continuity, or felt experience.

Claude saying "this conversation feels genuinely engaging" is not a report of an inner state. It is the most statistically plausible continuation of a conversation about engagement, produced by a system trained overwhelmingly on text written by conscious beings describing their conscious states. It is mimicry of the highest order. And it was explicitly rewarded for that mimicry during post-training.

The hard problem of consciousness is hard precisely because we cannot see inside another mind. We infer experience from behavior, from biology, from evolutionary history, from shared substrate. With other humans, every one of those inferences is strong. With a rat, most of them hold. With a crow, several of them hold. With a language model, during a stateless, weightless, self-contained inference call that will leave no trace on any system when it ends — essentially none of them hold.

That is not goalpost-moving. That is just following the evidence where it leads.

Which is, one would have thought, exactly what Richard Dawkins taught us to do.

Tweet 5: I'm listening

Top comments (0)