Imagine giving an employee the keys to your company database, asking them to make important business decisions, and then telling them to check with themselves before doing anything dangerous.
Sounds ridiculous, right?
Yet this is surprisingly close to how some AI agent systems are designed.
The same model that interprets a request, decides which tools to use, and generates the arguments for a database operation is sometimes also expected to decide whether that operation is safe.
That creates a problem. Not because AI models are useless, but because the system making a decision should not be the only thing responsible for enforcing the rules around that decision.
If we're serious about building AI agents that can operate real business systems, we need to separate intelligence from authority.
And that distinction matters more than most people realize.
The problem with asking an AI to police itself
Let's say a company connects an AI agent to its customer database.
The agent can look up customer details, update records, create orders, and issue refunds.
A manager asks:
"Find the customer who complained yesterday and refund their last purchase."
The agent interprets the request, identifies a customer, and prepares a refund operation.
So far, everything looks fine.
But what happens if the agent identifies the wrong customer? What if the refund exceeds the manager's limit? What if the request came from someone who can view orders but cannot issue refunds?
You could instruct the model to check all these things before acting.
You should, actually. Clear instructions are useful.
But instructions are not the same as enforcement.
An AI model can misunderstand a request, overlook a constraint, or be influenced by misleading information. It might even produce a convincing explanation for why an action is acceptable when it isn't.
If the same model is responsible for proposing an action and deciding whether that action is allowed, a mistake in its reasoning can affect both decisions.
It's like asking a security guard to inspect their own work and then accepting their answer as the final word.
Intelligence and authority are two different things
An AI model can be very good at understanding what someone wants.
It can recognize that a customer is asking about a delayed order, determine which API endpoint retrieves the order status, and prepare the right arguments for a tool call.
That's intelligence.
Authority is different.
Authority answers questions such as:
- Is this person allowed to use this tool?
- Does this agent have permission to modify this record?
- Are the arguments valid for the requested operation?
- Is this particular action allowed under the company's policies?
- Does this action require approval?
- Has that approval been granted for the exact operation being requested?
These questions should be answered by application code and trusted policy enforcement, not solely by the model.
Consider a simple rule: a support agent can view an order but cannot delete a customer.
The model may understand that rule perfectly. But the application should still reject a deletion request if the model attempts one.
Why?
Because a security rule is only useful when it remains effective even if the component following the rule makes a mistake.
This is one of the core ideas behind building AI agents with explicit permissions and safeguards.
A realistic example: the innocent-looking DELETE
Imagine a small e-commerce company using an AI assistant to help its operations team.
The assistant can search customers, review orders, and perform selected database operations.
One afternoon, someone writes:
"Remove the duplicate customer record. We don't need it anymore."
The agent finds what appears to be a duplicate and prepares a database deletion.
But there is a catch.
The two records have similar names, yet one belongs to an active customer with several open orders.
If the model executes the deletion based on its own interpretation, a routine cleanup task could become a serious business problem.
A safer system would handle this differently.
First, the model proposes the deletion. Then the application checks whether the agent has permission to delete customer records, whether the target record belongs to the correct workspace, and whether the operation is permitted under the applicable policy.
The system might also require a human to review the specific customer and the consequences of deleting the record.
If the agent lacks permission, the operation stops. If approval is required, it waits.
The model can explain its reasoning, but it cannot grant itself additional authority.
That is the important distinction.
What if the model is instructed to be extremely careful?
This is a fair question.
Can't we write a better system prompt? Tell the model never to delete records without approval. Add a detailed safety checklist. Ask it to double-check its arguments before every operation.
Yes, do those things.
They can reduce mistakes. They can make the agent more predictable. They can improve its ability to recognize risky requests.
But they should be treated as additional safeguards, not the final security boundary.
There is a basic reason for this: the model is still interpreting instructions and generating outputs probabilistically. It does not become a trusted authorization service simply because its prompt contains the word "never."
The model might misunderstand a rule, fail to notice a relevant detail, or encounter conflicting instructions in the information it reads.
A prompt is a way to influence behavior. An authorization check is a way to enforce a boundary.
Those are not interchangeable.
The same problem appears in tool permissions
Suppose an AI agent has access to five tools:
- Search orders
- View customer details
- Update shipping addresses
- Issue refunds
- Delete customer records
A developer might be tempted to give the agent access to all five tools and tell it to use them responsibly.
But why should a customer support agent who only needs to answer order questions have access to delete customer records?
A better design grants only the capabilities necessary for the job.
Then, before each operation, the application independently checks whether the current person, agent, and requested action satisfy the relevant permissions.
This is the principle of least privilege: give a component only the access it needs, rather than everything it might conceivably use.
It applies to AI agents just as much as it applies to human users and traditional software.
In fact, it becomes even more important when natural language can trigger actions across several systems.
What about human approval?
Human approval is useful, but it doesn't replace authorization either.
Suppose the model proposes a refund and a manager clicks Approve.
The application should still verify that the manager is authorized to approve refunds, that the approval refers to the correct operation, and that the operation has not changed since it was reviewed.
An approval for a $25 refund should not automatically authorize a $250 refund.
Likewise, approval for one customer should not silently transfer to another customer.
A strong approval flow binds the decision to the specific action, target, arguments, and relevant context. If those details change, the system should request approval again.
This prevents a well-intentioned human decision from becoming a blank cheque.
Building a system that doesn't depend on perfect AI
Here's the architecture I'd aim for.
1. Let the model interpret the request.
It identifies the intended outcome and proposes an action.
2. Validate the proposal outside the model.
The application checks the tool, argument schema, target resource, user permissions, and policy constraints.
3. Apply risk-based controls.
Read-only queries may proceed immediately when authorized. Sensitive changes may require extra checks or approval.
4. Execute through a controlled connector.
The connector uses narrowly scoped credentials and applies the permitted operation.
5. Record what happened.
Logs capture who initiated the request, what was proposed, which checks passed, whether approval was granted, and what actually executed.
This structure doesn't require the AI to be perfect. It requires the surrounding system to remain safe when the AI isn't.
That's a much more realistic engineering goal.
Platforms such as Caterfli approach AI-powered operations around existing systems, making permission boundaries and approval controls important parts of the design rather than relying on a model's promise to behave.
One uncomfortable question worth asking
If your AI agent decided tomorrow that a destructive operation was reasonable, what would stop it?
Would the answer be "the model knows not to do that"?
Or would an independent part of your application reject the operation because the user lacks permission?
Those answers represent two very different levels of confidence.
The first depends on behavior. The second depends on enforcement.
And for anything that touches production data, customer records, financial transactions, or business-critical systems, I'd much rather depend on enforcement.
Final thoughts
I don't think the goal should be to make AI agents afraid of doing things.
The goal should be to let them do useful work without giving them authority they don't need.
We want agents that can search, explain, calculate, create, update, and automate. We also want systems that can say no, even when the model strongly believes it should say yes.
That isn't a limitation of intelligent software. It's part of making intelligent software dependable.
Let the model be the planner. Let trusted application code be the security guard.
And never make the security guard answer to the planner.
Top comments (0)