AI agents are no longer limited to simple chat-based interactions.
They can process information, interface with APIs, run software utilities, perform workflows, engage with end-users, and make decisions to varying degrees of autonomy.
This provides clear advantages for developers and businesses: less tedious work and faster execution.
However, this brings up the challenging engineering question:
What happens if an AI agent makes a wrong decision?
The answer is not as simple as: “The AI made an error.”
The AI can carry out an action; however, accountability should be attributed to those who developed, deployed, configured, and monitored this particular AI solution.
AI Agents Transform Automation
Typical automation always worked based on predefined logic:
IF A THEN B
AI agent can operate much more flexibly:
User request
↓
Reasoning by AI
↓
Getting information
↓
Selecting a tool
↓
Executing an action
↓
Evaluating result
↓
Moving ahead or escalating
Flexibility is handy, but it brings yet another dimension – uncertainty.
The AI may misinterpret the instruction, act upon incomplete information, choose an incorrect tool, or get an unintended result.
Nevertheless, this does not mean that AI agents cannot be used at all.
So their boundaries must be carefully designed.
Not all AI Decisions Require the Same Supervision
Take two examples.
An AI agent classifies support requests automatically.
If the agent fails, a human can fix the problem of classification.
What about an agent that can authorize financial transactions or perform decisions in areas such as employment, security, and others?
It will have significantly different consequences.
Here is how we can see a simple engineering principle:
The more important is the decision, the more control by humans it requires.
The control does not necessarily mean manual review of all decisions by AI.
It can include:
Thresholds for approvals
Escalations
Limits of permissions of tools used
Transaction limits
Checks of confidence level
Validation process
Audit logging
Rollback
All we want is to align the level of automation with risks.
Design Agent for Errors
Developers usually focus on the right operation of an agent under normal conditions.
In production environment we also need another question:
How the agent should operate in case of errors?
For each important action we may consider:
Can the agent perform this action?
↓
What happens if its reasoning is wrong?
↓
Can the action be validated?
↓
Can it be reversed?
↓
Does a human need to approve it?
This becomes especially true for agents that may modify the database, communicate externally, transact, and interact with production systems.
It would be erroneous for an agent to be automatically granted full permissions based on their ability to use a specific tool.
Permissions of Tools Are a Part of AI Governance
The capabilities of an AI agent largely depend on the tools it has access to.
For instance, there is a vast difference between granting permissions to:
read_customer_data()
and to:
delete_customer_data()
Similarly for APIs.
In general, it is recommended to adhere to the principle of least privilege.
Provide the permissions the agent needs to complete a certain task.
Sensitive functions could have additional requirements for validation and approval when needed.
This will decrease the damage done by a wrong decision while still allowing automation.
Auditability Is Important
If anything goes wrong, the developers will need to explain:
What made the agent do that?
More information is required than just a final result.
Information could include:
Input received
Context
Tools used
API calls
Actions chosen
Validation
Approvals
Outcome
Implementation details will depend on a specific system, but the general concept is the same:
Autonomous system should keep enough evidence to help to reconstruct critical decisions.
This point becomes even more significant in case of interaction with multiple applications or processes in the business workflows.
*AI Agents also Require Observability
*
Classic software engineering relies on logging, monitoring, metrics, and tracing.
AI agents bring new dimensions.
You may want to trace:
Tool call rates
Number of failed actions
Abnormal use of tools
Escalation rate
Reasoning loops
Latency
Cost
Quality of output
Human interventions
Observability becomes more than application monitoring in such case.
It helps to understand the behavior of autonomous system over time.
Improved Models Will Not Solve All Issues
One can easily conclude that improved models will solve all the problems of accountability in the future.
Better models reduce some types of errors.
However, the reality is full of unpredictabilities.
The data may be incomplete.
Some APIs could fail.
Wrong permissions could be provided.
Requirements may change.
External factors could violate previous assumptions.
A highly capable model works in an imperfect world anyway.
That is the reason for responsible AI engineering not only consists of selection of the model itself.
Practical Architecture
A practical architecture of the AI agent can be broken down in terms of the following layers:
┌──────────────────┐
│ User │
└────────┬─────────┘
↓
┌──────────────────┐
│ AI Agent │
└────────┬─────────┘
↓
┌────────────────────────┐
│ Context + AI Reasoning │
└───────────┬────────────┘
↓
┌────────────────────────┐
│ Tool / API Gateway │
└───────────┬────────────┘
↓
┌────────────────────────┐
│ Validation + Policies │
└───────────┬────────────┘
↓
┌────────────────────────┐
│ Business Applications │
└────────────────────────┘
It is crucial to understand that the model does not have to be the ultimate source of truth.
Policies, validations, permissions, human intervention and other processes can surround the model.
This is how a more controlled environment for autonomous decisions can be built.
*Responsibility Chain
*
When an AI agent makes an incorrect decision, it is important to analyze the whole chain of the process' events:
Design → Deployment → Permissions → Autonomy → Monitoring → Intervention
Who designed the agent?
Who authorized what set of tools the agent will be using?
Who established the level of autonomy of the agent?
Who deployed the agent into production?
Who monitors the operation of the agent?
Who takes responsibility for the potential intervention?
These questions are much more important than the criticism of the algorithm itself.
The agent did perform the final action, but people and organizations created the environment in which that happened.
AI Autonomy Requires Engineering Rigor
Instead of trying to engineer all workflows to remove humans from the process,
the task is to figure out which workflows can handle autonomy and which still require human discretion.
Low impact activities can be performed by autonomous agents.
High impact decisions might require some additional safeguards.
It could involve restricted access, validation, approvals, monitoring, and audit trails.
AI agents can make software applications more powerful.
However, power without discipline introduces unnecessary risks.
Instead of asking this engineering question:
"How do we make an AI agent autonomous?"
We need to ask:
"How do we make an AI agent autonomous, yet keep the system observable, controllable, and accountable?"
This is the question that engineers and companies will need to ask themselves more and more often as AI agents evolve from experiments to production environments.
For companies looking at how AI can bridge the gap between communications, information, applications, and business processes, here is an example of AI-powered connected business communications platform: (https://commconai.com/)
Top comments (0)