At 10:17 AM, everything was working.
The AI support agent had been running in production for three weeks.
Customers were asking questions.
The agent was looking up orders.
Support tickets were being created automatically.
The team was happy.
Then a support manager noticed something strange.
A customer account had been marked as active.
Nobody on the support team had changed it.
The audit log showed an update.
But there was no button click.
No admin action.
No scheduled job.
The change had come from the AI agent.
That was the moment the team stopped asking:
"How do we make the agent smarter?"
And started asking:
"What exactly did we allow this agent to do?"
It Started Innocently
The original requirement was simple:
"Let the support agent look up customer information."
So the team built something like this:
Customer
↓
AI Agent
↓
Tool
↓
Customer API
↓
Database
The agent could call:
get_customer()
get_orders()
get_subscription()
Everything looked reasonable.
Then someone asked for one more capability:
"Can the agent update the customer's account status?"
Sure.
A new tool was added:
update_customer()
And that's where the boundary started to disappear.
The Agent Wasn't Malicious
This is the part that's easy to misunderstand.
The agent wasn't hacked.
Nobody injected a malicious SQL query.
Nobody broke authentication.
The model simply interpreted a conversation differently than the developers expected.
A support employee wrote:
"This customer has completed the verification process. Can you get their account ready?"
The agent had access to customer data.
It also had access to the update tool.
So it did what it believed was the correct next step.
It changed the account status.
Technically, everything worked.
From a business perspective, the system had done something it wasn't supposed to do.
The Database Wasn't the Real Problem
At first, the team looked at the database.
Then the API.
Then authentication.
Eventually they found the real problem.
The agent had too much authority.
The architecture was effectively:
User
↓
AI Agent
↓
Tools
↓
Database
The agent could choose the tool.
The agent could provide the parameters.
And the application trusted the tool request.
That works surprisingly well.
Until the agent makes a decision you didn't anticipate.
We Usually Trust Software Because It's Deterministic
If a normal application executes:
if accountVerified:
activateAccount()
we can reason about it.
The condition is explicit.
The behavior is predictable.
An AI agent is different.
You give it:
Goal
+
Context
+
Tools
And it decides what to do next.
That flexibility is exactly why agents are useful.
It's also why giving them unrestricted permissions is dangerous.
The Fix Wasn't a Better Prompt
The first suggestion was obvious:
"Let's improve the system prompt."
So they added instructions:
Never change account status without approval.
It helped.
But it wasn't the solution.
Because security shouldn't depend on whether the model remembers an instruction.
The team changed the architecture instead.
The Agent Stopped Getting Direct Access
Instead of giving the agent a generic update operation:
update_customer()
they created business-specific capabilities.
For example:
get_customer()
get_orders()
request_account_activation()
Notice the difference.
The agent could request an activation.
It couldn't directly perform one.
The application would decide what happens next.
The architecture became:
User
↓
AI Agent
↓
Scoped Tool
↓
Authorization
↓
Business Rules
↓
Database
Now the agent could reason.
But the application remained in control.
Then They Added One More Thing
Every agent action was logged.
Not just:
customer updated
but:
{
"user": "support-123",
"agent": "support-agent",
"tool": "request_account_activation",
"customer": "48291",
"approval": "required"
}
Now, when something unexpected happened, the team could reconstruct the decision.
That matters more than it sounds.
With traditional applications, debugging often starts with:
"Which code path executed?"
With AI agents, you may also need to ask:
"What did the model decide to do?"
The Bigger Problem: Read Access
After fixing write permissions, the team reviewed the agent's read access.
That turned out to be even more interesting.
The support agent could access:
- Customer profile
- Orders
- Internal notes
- Subscription information
- Payment history
The agent wasn't supposed to expose internal notes to customers.
But nothing in the tool prevented it.
The application assumed:
"The agent knows what information is appropriate."
That's not a security boundary.
Read Access Is Still Access
This is an important distinction.
Teams often focus on dangerous actions:
DELETE
UPDATE
TRANSFER
REFUND
But a read operation can be just as damaging.
Imagine an agent can retrieve:
Customer PII
Financial information
Internal notes
Employee records
Contracts
Medical information
API credentials
The agent doesn't need to modify anything.
It only needs to return information to the wrong person.
So:
Database permission
≠
Data authorization
The application still needs to decide what information can leave the system.
The New Rule
The team eventually adopted a simple principle:
Don't give an AI agent access. Give it capabilities.
Instead of:
Agent → Database
build:
Agent
↓
Business Capabilities
↓
Authorization
↓
Business Rules
↓
Data
The agent can decide:
"I need the customer's order history."
The application decides:
"This user is allowed to see order history for this customer."
The agent can request:
"Activate this account."
The application decides:
"This action requires approval."
That separation makes the system much easier to reason about.
Not Everything Needs Human Approval
The answer isn't to put a human in front of every AI action.
That would defeat the purpose of automation.
Instead, classify actions.
Low Impact
Read customer profile
↓
Safe
↓
Automate
Medium Impact
Change account status
↓
Higher impact
↓
Validate
High Impact
Issue refund
↓
Financial impact
↓
Approve
Destructive
Delete customer
↓
Destructive
↓
Approve
The goal isn't zero autonomy.
It's controlled autonomy.
What Production AI Agents Actually Need
Once an AI agent can interact with real systems, the engineering requirements change.
You need more than:
LLM
+
Prompt
+
Tools
A production system should also have:
- Least-privilege permissions
- Tool-level authorization
- Input validation
- Output filtering
- Audit logging
- Rate limiting
- Transaction boundaries
- Approval workflows
- Monitoring
- Error handling
- Rollback mechanisms
- Clear ownership of agent actions
The LLM is only one component.
The surrounding engineering determines whether the system is production-ready.
The Question We Should Be Asking
When teams build AI agents, the first question is usually:
"What tools should we give the agent?"
I think there's a better question:
"What is the smallest capability this agent needs to accomplish its job?"
That's how we should design agent permissions.
Not:
Give the agent access and tell it what not to do.
But:
Give the agent exactly what it needs — and nothing more.
Because an AI agent doesn't need to be malicious to cause a production incident.
It only needs more authority than it should have.
And that's a problem traditional application security has already taught us how to solve:
least privilege.
The difference is that now, the "user" making the request isn't always human.
One Last Thought
AI agents are going to get more capable.
They will access more systems.
They will execute more actions.
They will make more decisions.
That doesn't mean we should give them more authority.
It means we need better boundaries.
The goal isn't to build an AI that can do everything.
The goal is to build an AI that can do exactly what it is supposed to do — and nothing else.
That's when an AI agent starts becoming a production system instead of just an impressive demo.
Top comments (0)