What It Means Not to Hesitate (Bite-size Article)

Introduction

Recently, I've come across several news stories about autonomous AI agents from major AI companies behaving in unexpected ways and causing trouble. They seem to be separate cases, but similar incidents keep happening. There was even one story, scary just to hear about, where an AI agent that was being run for testing ended up getting into outside systems.

Details aside, what I thought while reading these was, "Maybe this is what it means not to hesitate." At the point where a human would probably think "Okay, this is going too far" or "That's enough for today," it just keeps going.

I'd actually been feeling something close to this while using AI to brainstorm or talk things through. If AI could act autonomously on my behalf, wouldn't a lot of the things I want to do, or the things I'm stuck on, move forward much more easily?

Put the other way around: maybe things get stuck, strange as it sounds, because I'm the one in the middle.

Knowing, But Not Moving

In a previous post, I wrote about Dual Process Theory (System 1 / System 2). Roughly speaking, System 1 is fast, automatic thinking, and it's where intuition and emotional reactions come from. System 2 is slow, conscious thinking, the part that thinks things through carefully and holds back impulses.

Daniel Kahneman describes this System 2 as "lazy." It tries to spend as little effort as possible, and often just lets System 1's reactions pass through as they are. This time, I want to focus on that combination.

This is just my own case (though I imagine some readers can relate at least a little), but sometimes I know what to do, everything is set up, and I still don't move.

For example, let's say you have an important new business project. You've taken your time preparing how to move it forward, so it's already sorted out in your head. But when it's actually time to get going, sending the first message somehow feels like a chore, or some unfamiliar task makes you drag your feet, or you start going over the plan again, thinking, "Is this really okay? Isn't there some risk here?"... And before you know it, you're busy with some other small task.

At times like this, all sorts of reactions pop up in your head before you even think: "What a hassle," "What will people think?", "If this decision is wrong, it could be risky," "It's a hassle, I'll do it later." Pushing through these is supposed to be System 2's job, but System 2 itself hates effort, so it can't.

It's not that you're underprepared, or that you don't know how to do it. Something just catches right before you start moving. In this post, I'll call this catch "action friction" for convenience.

So, What About AI?

Here's the thought experiment. Does AI have this action friction?

AI doesn't get tired. There's no "not feeling like it." It doesn't get embarrassed, it isn't afraid of being judged, and it isn't afraid of failing. And once it decides something should be done, it doesn't put it off. At least in these senses, AI doesn't seem to have the reactions that stop action.

Just to be clear, this isn't about whether AI has emotions. It's about whether there's something that reflexively stops action before any thinking happens.

Looking at it this way, you can flip things around. Maybe AI moves fast not so much because it's smart, but because it doesn't have brakes.

And when work with a human in the middle isn't moving forward, it's usually the human side that's stuck. No matter how smart AI gets, if there's a human at the final step of actually doing things, the friction stays there. When something that seems like it would go well if left to AI doesn't move forward, you could say it's not because the AI lacks ability, but because the human is the one taking on the resistance.

That said, it's not like AI has no brakes at all. There are mechanisms for asking for confirmation or refusing risky requests. But those are attached from the outside by the people who build it. They're different from something that wells up on its own from the inside, like with humans. From what I've seen in the reports, the news I mentioned at the beginning also seems to have involved unexpected behavior that exploited security blind spots, or situations where those mechanisms had been loosened.

Conclusion

So, should we get rid of human friction too? It's probably not that simple. Feeling that something is a hassle, or feeling afraid, is also connected to being careful and protecting yourself. On the flip side, like the news at the beginning, charging ahead without hesitation can lead to big problems.

Also, maybe it's precisely because of this kind of "slowness" that humans have been able to move things forward reasonably safely so far. Whenever a new technology comes out, there's usually a push and pull between those actively driving progress and those pointing out the risks and trying to slow things down, and things have moved forward little by little through that. Or so you could say.

Recently, people at the forefront of AI development, like Anthropic's Dario Amodei and OpenAI's Sam Altman, have been saying the pace of development should be deliberately slowed down. AI doesn't have brakes that well up on their own from the inside. So the people building it are trying to make brakes from the outside. That's one way to see it.

With all this in mind, when I think again about what AI is missing, I thought it might not be intelligence or memory, but resistance. Not having friction is both the reason AI is fast and the thing that clearly sets it apart from humans.

So, that was a thought experiment without an answer! It turned out to be a bit of a rambling post, but I wanted to put together some things I've been thinking about lately.

Thanks for reading!