We often talk about malicious AI.
Rogue AI.
Misaligned AI.
AI agents going beyond their instructions.
But I'm increasingly convinced that we're sometimes asking the wrong question.
An AI agent doesn't have to be malicious for one of its actions to become dangerous.
Sometimes, the agent is simply doing what it was designed to do.
Reading files.
Running commands.
Calling tools.
Inspecting a repository.
Trying to complete a task.
The problem is what happens when those perfectly normal actions meet an environment that shouldn't be trusted.
A repository can become part of the attack surface
AI coding agents routinely inspect repositories.
They may run commands like:
git status
git diff
git log
These are normal actions.
Nothing suspicious.
Nothing malicious.
But recent security research involving AI coding agents has highlighted an interesting problem: a malicious or unexpected repository configuration can influence what happens when an agent performs normal development operations.
The agent may simply be trying to understand a project.
It doesn't need to be compromised.
It doesn't need to become malicious.
It only needs to perform a normal action in an environment that contains something dangerous.
That distinction matters.
The dangerous action may look completely normal
Imagine an AI coding agent opening a project.
Its goal is simple:
"Understand the repository and help me fix a bug."
So it starts doing what coding agents normally do.
Inspect the files
Check Git status
Read the project configuration
Analyze recent changes
-
Suggest or implement a fix
From the agent's perspective, this makes perfect sense.
It is trying to understand the project.
But what if one of those normal operations produces an unexpected effect?
The chain might look like this:
User opens a repository↓
AI agent starts analyzing it
↓
Agent performs a normal operation
↓
The environment influences execution
↓
An unexpected action occurs
↓
Real-world consequence
At no point does the AI need to decide to attack.
It may simply be doing exactly what it was asked to do.
And that's the part I find particularly interesting.
We often focus on what the AI thinks
A lot of AI security discussion focuses on the model.
Can it be jailbroken?
Can it be manipulated?
Can prompt injection change its behavior?
Can the model understand malicious instructions?
These questions matter.
But once an AI model is connected to tools, another problem appears.
The model can turn language into actions.
An agent may be able to:
read files
execute commands
modify code
interact with Git
call APIs
access databases
browse the web
interact with cloud services
At that point, security is no longer only about what the model says.
It is also about what happens after the model decides to do something.
The question is no longer only:
Can someone manipulate the AI?
It also becomes:
What happens after the AI takes an action?
Intent is not the same as execution
Suppose I tell an AI coding agent:
"Please analyze this repository."
The agent might reasonably:
Read source files
Inspect dependencies
Check Git status
Analyze configuration
Run tests
Those actions are expected.
But the final effect of an action may not always be obvious from the user's original request.
This creates a gap:
User Intent
↓
Agent Reasoning
↓
Proposed Action
↓
System Execution
↓
Real-world Effect
Every step can introduce risk.
And the final effect may be different from what the user originally intended.
The agent doesn't have to be "bad"
We often imagine AI security problems like this:
Good AI
vs.
Bad AI
But reality may be more complicated.
Sometimes the situation looks more like this:
Well-intentioned AI
+
Normal Action
+
Unexpected Environment
=
Dangerous Outcome
The AI might be:
following the user's instructions
using the tools it was given
performing a normal workflow
trying to complete its task
And the result can still be dangerous.
This means that relying entirely on the model to make the correct decision may not always be enough.
AI agents are becoming part of the execution layer
Traditional AI mostly produced output.
You asked a question.
The model generated text.
If the answer was wrong, the consequence was usually limited to information.
AI agents are different.
They increasingly interact with:
filesystems
shells
Git repositories
APIs
cloud services
databases
browsers
local applications
The boundary between:
AI output
and
system execution
is becoming thinner.
An agent can go from:
"I think you should run this command."
to:
Running the command.
That is a fundamental change.
And I think our security models need to change with it.
Access permission is not the same as action authorization
Suppose an AI agent has access to a project directory.
Does that mean it should automatically be allowed to:
Read every file?
Maybe.
But what about:
Delete every file?
Read secret files?
Modify Git history?
Execute arbitrary commands?
Upload project contents?
Access to something doesn't necessarily mean that every possible action involving it should be allowed.
Similarly:
Shell access
doesn't necessarily mean:
Unlimited shell authority
And:
Repository access
doesn't necessarily mean:
Every possible operation inside that repository
This is where I think AI agents introduce an increasingly important security problem.
Capability can easily be mistaken for authority.
An agent can do something.
But that doesn't automatically mean it should be allowed to do it.
What if the security boundary existed around the action?
This is the question I've been exploring.
Instead of relying entirely on:
AI Agent
↓
System
What if there were an additional boundary?
AI Agent
↓
Proposed Action
↓
┌────────────────────────┐
│ │
│ Action Boundary │
│ │
│ Is this action │
│ permitted? │
│ │
│ Does it cross a │
│ protected boundary? │
│ │
│ Should it be blocked? │
│ │
└────────────────────────┘
↓
System
The AI agent can still decide:
"I want to perform this action."
But another layer can ask:
"Should this action actually be allowed?"
Those are two different decisions.
Better AI doesn't remove the need for boundaries
AI models are becoming more capable.
Coding agents are becoming more autonomous.
Agents are getting better at:
planning
reasoning
using tools
recovering from errors
completing multi-step tasks
That's useful.
But more capability also means more possible actions.
A more capable agent may discover a path that a less capable agent would never find.
That doesn't mean the agent is malicious.
It simply means:
Capability is increasing faster than our assumptions about what an agent might do.
And that's why I believe external boundaries will become increasingly important.
Don't ask whether the AI is good or bad
Perhaps the more useful question is:
Should this action be allowed?
Not:
Is the AI good?
Not:
Is the AI malicious?
Not even necessarily:
Did the AI understand the user's request?
But:
This specific action.
Right now.
In this environment.
Should it be allowed?
That is a different security model.
And I think it's worth exploring.
This is why I'm building ActionGuard
I'm currently exploring these questions through an open-source project called ActionGuard.
ActionGuard is an early exploration of an independent AI Action Safety Boundary.
The idea is not to replace:
model alignment
AI safety research
sandboxes
operating system security
permission systems
Instead, I'm exploring an additional question:
Can there be an independent boundary between an AI agent and actions that may affect the real world?
The basic idea is:
Detect
↓
Evaluate
↓
Allow / Block
↓
Record
before a potentially dangerous action produces an effect.
One of the principles I'm exploring is:
The AI can decide what it wants to do.
An independent boundary can still decide what it is allowed to do.
The difficult question: where should the boundary live?
I don't think the answer is simple.
Should action boundaries exist at the:
agent layer?
tool layer?
application layer?
operating system layer?
network layer?
Perhaps the answer is multiple layers.
And another difficult problem is autonomy.
If an AI agent performs 100 legitimate actions while completing a task, we don't want to interrupt the user 100 times.
But we also don't want:
Autonomous mode
to quietly become:
Unlimited permission
Finding a useful balance between autonomy and control is one of the problems I'm interested in exploring.
The agent doesn't have to be malicious
That's the idea I keep coming back to.
AI security isn't always about stopping a rogue AI.
Sometimes:
A normal agent
+
A normal action
+
An unexpected environment
=
A dangerous outcome
The agent may be doing exactly what it was designed to do.
The action itself may even look completely reasonable.
But the outcome can still cross a boundary that should have existed.
As AI moves from generating text to taking actions, I think we will increasingly need to distinguish between:
What an AI agent can do
and
What an AI agent should be allowed to do
Those two things are not the same.
What do you think?
I'm particularly interested in hearing from people working with:
AI agents
coding agents
developer tools
AI security
cybersecurity
agent frameworks
Do you think AI agents need an independent action boundary?
Or should security primarily remain inside:
the model?
the agent framework?
the operating system?
the tools themselves?
I'd love to hear different perspectives.
ActionGuard
I'm building ActionGuard as an open-source exploration of an independent AI Action Safety Boundary.
Give AI room to work. Keep control of what it can do.
GitHub:
https://github.com/SeanXChen/ActionGuard
Top comments (0)