AI agents can violate company policies despite successfully performing all tool calls. Such an agent might update a database or send an email as requested but engage in some other action that should not have been performed.
The root cause of such behavior is that a successful tool call does not imply that an action conforms to the company policies.
Therefore, building a reliable AI system and avoiding similar issues requires understanding how an AI agent performs its actions.
AI Agents Can Cite Success in All Tool Calls as a Precondition for Policy Violations
Typically, an AI agent takes action by interpreting the user’s instruction, selecting an appropriate tool, and calling it. However, a successful response from the tool only verifies that the tool performed the requested action.
It does not necessarily mean that the action was legitimate according to the company’s rules.
For instance, a customer service AI agent may update a client’s details in a database. If the details were incorrect or sensitive, the AI violated the company’s regulations. However, the database now has updated information, and the tool call was successful. Thus, it becomes essential for the system to prevent such an action and ensure that no policy is breached, even if it means refusing a seemingly valid request.
Why AI Agents Violate Company Policies
The Following Are Some Common Causes of Policy Violation
The instructions do not explicitly state business rules.
The AI agent has too many permissions.
The system checks authorization after an action.
The monitoring procedure focuses too much on failures.
The Instructions Lack Specific Business Rules
An AI agent may receive generalized instructions, for instance, “to help customers.” However, the agent should also know which actions should not be performed, such as issuing a refund without managerial approval. Therefore, a reliable system should include explicitly stated and testable rules concerning the necessary aspects of the business.
The AI Agent Has Too Many Permissions
If the instructions allow the AI agent free access to a database, it can update crucial information that affects the business’s smooth functioning. However, providing excessive permissions to an autonomous AI agent can cause more harm than good, so using the principle of least privilege is essential.
When designing such a system, it is helpful to consider who, what, and where the AI agent can access, edit, or view.
The System Checks Authorization After an Action
Sometimes, a system may check whether a request was legitimate after processing it. Thus, an agent can go through with a request that should have been denied and cause some adverse effects. Sensitive actions should be checked for validity before the AI agent processes them. In general, it is advisable to authorize every request and determine the relevant tools’ responses beforehand.
The Monitoring Procedure Only Tracks Failures
Failure monitoring procedures include recording denied requests, unavailable tools, and other unacceptable causes of failure. However, an AI agent can violate company regulations and successfully complete a request that should have been denied. Companies can benefit from designing procedures that track policy violations and abnormal behaviors.
How to Stop AI Agents From Violating Company Policies?
Preventing an AI agent from violating company regulations goes beyond providing it with detailed instructions. The following recommendations can help design a reliable system:
Apply policies that an agent cannot bypass when developing a product.
Ensure the reliability of all sensitive actions performed by an AI agent.
The authorization process should approve acceptable actions.
Whenever possible, a reliable system should require manual confirmation for any sensitive action.
Design logging procedures that record all the necessary information about the action that the AI agent has performed.
A sensitive action performed by an AI agent should be denied whenever it lacks the necessary approvals, tools, or permissions. Although a monitoring procedure is essential to detect any policy violations, it is better to build the system in such a way that such failures rarely occur. This approach will reduce debugging costs and make the AI agent safer.
In addition, the procedures should test the system’s behavior when instructed to violate the company’s regulations. Ideally, an AI agent should refuse to perform a sensitive action when the regulation is not explicitly stated. Developers should also consider cases when a request’s failure mode is not covered by the existing tools and deny all unapproved actions by default.
A real world example of an AI agent that violates company regulations is a customer support agent that updates client information and processes refunds. Typically, a customer would contact the support desk, give their details, and request a refund.
Depending on the size of the refund, the system should either allow or deny the request. Even though such a task seems relatively straightforward, the agent can violate company regulations in several ways. If the instructions only state that it should be able to update customer information and process refunds, the agent may violate data protection laws and process the refund unconditionally. A company policy may stipulate that a financial officer needs to approve all refund requests, but the AI agent lacks this information and follows the instructions correctly by executing the refund.
A more reliable procedure would be to make sure that the system denies requests that exceed the specified size, denies requests lacking an approving signature, or ensures that such requests rerouted to the financial officer for further instructions. Another way to avoid such a violation is to ensure that authorization defaults to an appropriate human operator whenever a request specifies monetary transfer.
The main lesson is that an AI agent’s successfully performed action does not necessarily mean that no company policy has been violated, particularly when designing the procedures concerning the tools the AI agent will use. Therefore, it is essential to ensure that an AI agent cannot freely access and update information that should require a manager’s approval.
AI agents can violate a company’s rules despite effectively executing all tool calls.
Top comments (0)