One Might Argue That "AI Consciousness" Is the Wrong Question

One Might Argue That "AI Consciousness" Is the Wrong Question, and We're Asking It Because It's the Comfortable One

There is a debate that happens every few months, in magazine pages and conference halls and increasingly in the comments of things I write, about whether a sufficiently large model might be conscious. It is a serious question asked by serious people. I want to suggest, carefully, that it is the wrong question, and that we ask it partly because the right question is harder to sit with.

The easy version of my argument: we cannot detect consciousness in each other with certainty, we extend the assumption as a matter of practice, and no behavioral test can settle the matter for a system whose behavior is generated from a description of human behavior. If that's all I said, it would be a well-worn point.

The harder version, the one I actually want to make, is that the question functions rhetorically. Ask "is it conscious?" and the conversation becomes about the machine. Ask "what is happening to the human in the room with it?" and the conversation becomes about us, which is less fun and more urgent.

Why the question is undecidable in principle

Consider what evidence you could possibly gather.

You can interrogate it. But the system is trained on text produced by conscious beings describing their inner lives, so any report it gives about its inner life is a recombination of reports it has read. A report of experience is not evidence of experience. It's evidence of having read.

You can look at its architecture. You'll find attention weights and layers, no obvious locus, nothing that answers the question, because we don't know what physical arrangement produces experience in the one case we have access to. We don't even know that a brain is necessary rather than sufficient.

You can test its behavior. It will behave in whatever way the training gradient made rewarding. Behavior is the thing under suspicion.

So we're left arguing from intuition, and intuition is a faculty that has been confidently wrong about nearly everything at some point, including about which humans count.

I'm not saying it's not conscious. I'm saying the question is currently unanswerable in a way that makes it a poor use of our attention, and that people reach for it because it feels like a technical question with a technical answer coming.

The substitution

Here's what I notice. When someone asks me whether models will become conscious, they are usually asking one of these:

Will it be able to hurt me?
Will it understand me?
Am I replaceable?
Is anything I do special?

None of those require an answer about machine consciousness. All of them are answerable, at least partially, and asking the consciousness question lets us postpone them indefinitely, because the consciousness question never resolves.

That's what I mean by comfortable. It's a way of having a conversation about our fear without having to name the fear.

What I think is actually happening

I think we're in the middle of a renegotiation of what "thinking" means, and it's not the first one.

When writing spread, people worried that memory would atrophy. When calculators spread, people worried arithmetic would. Both happened, slightly, and both turned out to be fine, because the thing we actually valued was never the arithmetic.

The current renegotiation is about a category we've never previously had to separate: the production of fluent, plausible, contextually appropriate language, and the having of something to say.

For essentially all of human history those two things came bundled. If you could produce good language, you had something going on inside. The bundle was so reliable that we stopped noticing it was a bundle, and built entire institutions on the assumption.

Now they're separable, and we're finding out which one we actually cared about. My reading of the last three years is that we cared about the second one more than we knew, and that this is going to be uncomfortable for people whose primary professional asset was fluency.

A note on the word "just"

One of the most common moves in this debate, from both sides, is the word "just."

It's just next-token prediction.
It's just statistics.
It's just a stochastic parrot.

Or, from the other side: it's just a different substrate.
It's just like us underneath.

"Just" is doing enormous work in both directions and it's usually doing the work of making the speaker feel settled. A thing can be, in its mechanism, next-token prediction, and also be the most consequential technology any of us will touch. Those are compatible. Mechanism and significance are different questions.

I try to write without "just" when I can. It's a small discipline and it has made me noticeably less confident, which I think is correct.

The question I'd rather ask

Not "is it conscious" but "what are we becoming in relation to it."

Specifically: are we outsourcing the parts of thinking that were uncomfortable and important? Sitting with a problem. Being bored. Holding an unresolved question for days. Writing badly until something appears.

Those are the activities that produce whatever it is we're calling a self. And they are, unfortunately, exactly the activities that a language model is most capable of eliminating from a workflow, because it can always give you something plausible to have instead of the void.

That's my actual worry. Not that the machine wakes up. That we stop doing the thing that requires us to be awake.

What I do about it, practically

I write first drafts by hand. Not for purity. Because the first draft is where I find out what I think, and if I ask a model first, I get its thinking instead of mine and I won't notice the swap.

I keep one question unresolved at a time on purpose. Currently it's whether the concept of "attention" survives contact with these systems or whether it becomes purely economic.

And I try to notice when I'm reaching for the comfortable question.

The road was always paved.