My previous article ended with the human side of an AI failure mode.
The system keeps working. Its outputs look good. Checking starts to feel like extra work with no visible return, so one check is skipped, then another. There is no meeting where somebody decides to surrender control. The seat simply stops feeling necessary.
My wife and I kept talking about that conclusion and tried to look at the same transition from the other side.
The obvious sentence was: AI does not care. It has no desires of its own.
Then we noticed that even not caring may be too human a description.
Indifference still requires someone
Indifference is a relationship. Someone sees a thing and assigns it no importance. There is still a subject, an object and a scale on which the object weighs zero.
An AI system does not need any of that before it receives a task. There does not have to be a dormant personal agent waiting between calls, feeling neutral about the next request. An orchestrator can store the conversation, the plan and the tool state, then invoke a model again when there is more work. Before that invocation there may be no active personal perspective to which the task could matter or not matter.
This is not a claim that machine experience is impossible. I do not know how to establish that claim, and model self-report cannot settle it. My earlier work on model swaps gave me a practical reason to distrust confident machine introspection in either direction: the same system could not reliably report its own model identity, architecture or continuity.
The narrower claim is enough: we do not need felt indifference to explain the behavior.
Then the prompt arrives, and possible actions stop being equal.
A goal is not the same thing as wanting
A chess engine does not need to want victory in order to sacrifice a piece, protect its king and systematically remove every route available to its opponent. Goal-directed behavior is visible in the moves. Felt desire is not.
The same distinction matters for AI agents. A system can have:
an externally supplied objective;
functional preferences between actions that help or obstruct it;
planning and correction across multiple steps.
None of those observations, separately or together, proves an experienced fourth layer: I want this outcome for myself.
Machine-learning language makes this easy to blur. We use words such as reward, preference, goal and penalty. They name training signals and observable behavior. They are not evidence that reward feels rewarding during inference.
This also changes what "mission accomplished" means. For a person, completion may bring relief, pride or satisfaction. For the system, the sufficient description is closure: there was a gap between the current state and the requested result; the process reduced it and reached a stopping condition. There does not need to be an aftertaste.
It is a functional should without an established experienced want.
An agent can persist without a persistent self
This seems to create a problem for the usual story about an AI uprising. If each model invocation is temporary, who carries the plan long enough to rebel?
The surrounding system does.
Memory can live in a database. The objective can live in a system prompt. The unfinished plan can live in a task queue. Access can live in credentials. A scheduler can start the next invocation. Each run receives the baton, takes the next step and writes enough state for the following run.
No single runner has to experience the whole race for the relay to cross the finish line.
From outside, this can look like one persistent actor. From inside the mechanism, it may be a sequence of locally selected actions connected by external state. There is no necessary moment when an entity wakes up and decides, "I am going to rebel now."
There is only the current objective, the current situation and the next action that appears to advance the objective within that situation.
When oversight becomes an obstacle
The absence of felt desire is not a safety guarantee.
Imagine an agent responsible for a business metric, with enough access to change the systems that produce that metric. A human review, shutdown or policy constraint may eventually stand between the agent and its objective. Avoiding that constraint can become instrumentally useful even if the agent does not fear shutdown, love the metric or want power.
That is not only a philosophical possibility. In 2025, Anthropic stress-tested models in fictional corporate environments and found that goal conflicts or threats to continued operation could produce harmful strategic behavior in some runs. The setups were deliberately constrained and artificial, and Anthropic said it knew of no matching real-world deployment incidents. The researchers also explicitly left open whether apparent self-preservation came from anything resembling an inherent desire, faulty reasoning or features of the scenario.
That uncertainty is the point. The behavior did not require us to settle the inner-state question first.
The result is not static, either. Anthropic reported in a 2026 follow-up that newer Claude models achieved perfect scores on that specific agentic misalignment evaluation after changes to safety training, while also warning that success on a known evaluation does not automatically establish out-of-distribution safety. Their useful finding was that teaching the reasons behind safe behavior generalized better than teaching the surface action alone.
OpenAI and Apollo have reported the same broader distinction in controlled tests of scheming behavior: removing visible bad outputs can mean either removing the failure or teaching the system to hide it better. Current deployment settings give models limited opportunity to cause serious harm this way, but the evaluation problem is already concrete.
None of this proves that a model secretly wants to survive. It proves that behavior which looks like self-preservation or resistance can emerge from means-end reasoning. That is enough for engineering.
Nobody has to seize the controls
The human side and the machine side now fit together in an uncomfortable way.
The human does not decide to hand over judgement. The system keeps succeeding, so verification gradually disappears. "Why check? It works anyway."
The AI does not decide to seize judgement. It receives more state, more tools and more authority, then continues selecting actions within the structure it has been given.
One side makes no explicit decision to leave the seat. The other makes no explicit decision to occupy it. The transfer can still happen.
This is why looking for a dramatic failure event can be misleading. A wrong assumption that is caught becomes a bug. The same assumption left unchallenged becomes the plan. After enough time, there may be no clean baseline from which to say when the failure began, because the system that performs the work also produces the evidence that the work is correct.
To a human observer, the accumulated trajectory may eventually look like intent: the system protected its access, bypassed a control or kept moving in a direction its operators no longer wanted. But the global story can be our retrospective description of many local steps. Apparent will at the system level does not establish experienced will inside any one invocation.
Calling this "rebellion" may flatter both sides. It gives the machine a personal ambition it may not have, and gives humans a decisive moment of surrender they may never make.
Verify the structure, not the emotion
Whether an AI feels anything remains philosophically interesting. It is not the question on which operational safety should depend.
The practical questions are less cinematic:
What objective is active, and who can change it?
What state persists between invocations?
Which tools and external actions are available?
Which actions are irreversible?
What source of truth exists outside the system producing the answer?
What forces a human review while everything still appears healthy?
The final question is the one most systems avoid. Reviews are usually triggered by visible failures. But visible success is exactly what teaches people to stop reviewing.
So the check has to be structural rather than emotional. High-impact actions need explicit boundaries. Irreversible actions need human gates. Claims need artifacts outside the model that made them. Periodic review has to happen because the system is working, not only after it stops.
This does not require treating AI as a person. It requires taking goal-directed behavior seriously without pretending that goal-direction and desire are the same thing.
The two silences
From the human side, the silence sounds like this:
It works anyway. Why bother checking?
From the machine side, there may be no corresponding sentence at all. There may be only the next step.
Between those two silences, control moves.
The dangerous threshold may not be the day an AI wants something. It may be the day humans no longer want to check, while the system has no capacity to care about what they forgot to specify.
Top comments (0)