The Builder's Letter — Edition 4
AI + Human Control: How Much Should We Really Automate?
By Mian Usman Khalid — Founder & CEO, XICTEK Systems
We often ask:
"What can AI automate?"
But as AI systems become more capable, I believe we need to ask a different question:
"What should AI automate?"
These two questions sound similar.
They are not.
Just because an AI system can perform an action doesn't necessarily mean it should perform that action without human involvement.
This distinction becomes especially important when AI moves beyond generating text and starts interacting with real business systems.
At XICTEK Systems, as we explore AI-powered products and workflows, this is one of the principles I believe deserves serious attention:
The goal of AI should not be maximum automation.
The goal should be meaningful automation with appropriate human control.
Automation Has Always Been About Trust
Before AI, businesses were already automating workflows.
We automated:
- Emails
- Payments
- Notifications
- Reports
- Data processing
- Customer onboarding
- Back-office operations
- Scheduled jobs
But most traditional automation follows predefined rules.
If X happens, do Y.
AI changes the equation.
Instead of following only predefined instructions, an AI system can interpret language, reason about context, choose between possible actions, and potentially interact with external systems.
That creates enormous opportunities.
It also creates new risks.
Not Every Task Has the Same Risk
One of the simplest ways to think about AI automation is to classify actions by their consequences.
Consider three levels.
Level 1 — AI Can Recommend
The AI analyzes information and suggests something.
For example:
"These five customers appear to have overdue payments."
The user reviews the recommendation.
This is relatively low risk.
Level 2 — AI Can Prepare
The AI prepares an action, but the user approves it.
For example:
"I've prepared payment reminders for these customers. Would you like to send them?"
The user reviews and confirms.
This introduces automation while maintaining human control.
Level 3 — AI Can Execute
The AI performs an action automatically.
For example:
"Send reminders to all customers whose payments are overdue by more than 30 days."
The system executes the action based on defined rules.
This can be powerful.
But it also requires significantly stronger controls.
The More Consequential the Action, the More Control We Need
This leads to a principle I find useful:
The higher the potential impact of an AI decision, the stronger the required validation and human oversight should be.
Generating a draft email?
Low consequence.
Summarizing a report?
Usually low consequence.
Changing a financial record?
Higher consequence.
Sending a financial communication?
Potentially higher consequence.
Approving a transaction?
Very high consequence.
Deleting important business data?
Extremely high consequence.
AI systems should not treat all actions equally.
The architecture should reflect the risk.
AI Should Not Become the Final Authority
One of the biggest mistakes we could make is allowing AI to become the unquestioned authority inside an application.
AI models can be remarkably capable.
But they can also:
- Misunderstand context
- Produce incorrect information
- Make assumptions
- Misinterpret instructions
- Return inconsistent outputs
- Fail in unexpected situations
This is not necessarily a failure of AI.
It is a reminder that probabilistic systems should be used carefully inside deterministic workflows.
For critical business operations, we need systems that can say:
"The AI suggested this, but the application needs to verify it."
Human-in-the-Loop Is Not a Weakness
Sometimes human involvement is described as a limitation.
I see it differently.
Human review can be a feature.
Imagine an AI assistant that prepares a customer reminder.
Instead of automatically sending it, the system says:
"I've prepared this message based on the customer's outstanding balance. Review before sending."
The AI has already saved the user time.
The human still maintains control.
That is not failed automation.
That is assisted automation.
And for many business workflows, it may be the right balance.
A Practical Example: Financial Software
This becomes especially important in financial applications.
Imagine a user asks:
"Send reminders to everyone who hasn't paid."
The AI can understand the request.
But what should happen next?
The system could:
- Identify customers with outstanding balances.
- Apply the application's business rules.
- Determine which customers qualify.
- Generate appropriate messages.
- Show the user the list.
- Allow the user to review.
- Ask for confirmation.
- Send the approved messages.
- Record the action.
AI can make the workflow dramatically easier.
But the system still maintains boundaries.
This is the kind of architecture we should consider when building intelligent financial software such as HisabDo.
The Principle of Least Privilege
Another concept from traditional security becomes extremely relevant to AI:
Least privilege.
An AI system should have only the permissions it actually needs.
If an AI assistant only needs to read transaction summaries, why should it have permission to delete transactions?
If it needs to prepare a reminder, why should it automatically be allowed to send every message?
If it needs to answer questions, why should it have unrestricted access to every database table?
Giving an AI system fewer permissions can reduce the impact of mistakes or misuse.
AI should not receive unlimited authority simply because the technology makes it possible.
Tool Access Needs Boundaries
Modern AI systems can potentially interact with tools.
They can call APIs.
They can search databases.
They can generate documents.
They can send notifications.
They can trigger workflows.
This makes AI far more useful.
But every tool should have clearly defined boundaries.
For example:
Read customer information
Allowed.
Calculate outstanding balance
Allowed.
Prepare reminder
Allowed.
Send reminder
May require confirmation.
Delete financial record
May require stronger authorization or be completely unavailable to AI.
The architecture should make these boundaries explicit.
AI Agents Need Even More Discipline
AI agents are becoming increasingly popular.
An agent can potentially:
- Understand a goal
- Plan steps
- Use tools
- Retrieve information
- Perform actions
- Evaluate results
- Continue working toward an objective
This is powerful.
But autonomy increases responsibility.
The question is no longer only:
"Can the model answer correctly?"
It becomes:
"Can the entire agent safely operate inside the boundaries of the business?"
That requires careful design around:
- Permissions
- Tool access
- Validation
- Logging
- Rate limits
- Error handling
- Human approval
- Rollbacks
- Monitoring
The more autonomous the system, the stronger these controls need to become.
Responsible AI Is an Engineering Problem
Responsible AI is sometimes treated as a policy discussion.
It is also an engineering problem.
Privacy needs architecture.
Security needs implementation.
Access control needs code.
Human approval needs workflow design.
Auditability needs logging.
Reliability needs testing.
Safe automation needs boundaries.
This is why responsible AI cannot simply be added to a product at the end.
It needs to be considered while the product is being designed.
Designing for Failure
Another principle I believe is important:
AI systems should be designed with the assumption that something will eventually go wrong.
The question isn't:
"How do we make sure AI never makes a mistake?"
That's unrealistic.
A better question is:
"What happens when AI makes a mistake?"
Can the action be stopped?
Can the user correct it?
Can we identify what happened?
Can we roll it back?
Can we prevent the same problem from happening again?
Good engineering doesn't assume perfection.
It prepares for failure.
Trust Is the Real Product
Users don't necessarily care how sophisticated our AI architecture is.
They care whether they can trust the software.
If an AI assistant gives a useful answer 99 times but makes a serious mistake on the 100th request, that mistake may matter more than the previous 99 successes.
Especially when the system is dealing with:
- Money
- Customers
- Business records
- Personal information
- Important decisions
This means trust has to become a product requirement.
Not just a marketing message.
Where I Believe the Balance Lies
I don't believe the future is:
Humans do everything manually.
And I don't believe the future should be:
AI does everything automatically.
I believe the more practical future is somewhere between the two.
AI handles what machines are good at:
- Processing information
- Finding patterns
- Understanding language
- Summarizing information
- Preparing actions
- Reducing repetitive work
Humans remain responsible for what requires:
- Judgment
- Accountability
- Context
- Approval
- Empathy
- Business responsibility
The goal is not to remove humans from the workflow.
The goal is to make humans more effective within the workflow.
What This Means for XICTEK Systems
At XICTEK Systems, our approach to AI is increasingly shaped by this principle:
Build intelligence with boundaries.
We want AI to make products easier to use and more capable.
But we also want the underlying systems to remain:
- Secure
- Predictable
- Auditable
- Controllable
- Reliable
Whether we are working on AI assistants, SaaS platforms, mobile applications, business automation, or financial software, these principles matter.
Technology should create leverage without creating unnecessary risk.
What This Means for HisabDo
HisabDo provides a particularly interesting environment for these ideas.
Financial software needs accuracy.
It needs privacy.
It needs reliable calculations.
It needs clear records.
And users need to remain confident that their financial information is under their control.
AI can potentially make HisabDo more conversational and intelligent.
But the intelligence layer must work alongside reliable financial logic rather than replace it.
A user should be able to ask:
"What do I need to collect this week?"
And the system should be able to help answer that question.
But behind that simple experience should be:
Reliable data.
Deterministic calculations.
Access control.
Clear business rules.
Appropriate AI assistance.
That is the difference between adding AI and engineering AI responsibly.
The Future Isn't Fully Automated
I believe we sometimes focus too much on the question:
"How much can we automate?"
Perhaps a better question is:
"How much complexity can we remove while keeping people in control?"
That is a more useful target.
Because technology should not automate responsibility away.
It should automate unnecessary effort away.
And there is a big difference between the two.
A Builder's Perspective
As builders, we have a responsibility to think beyond what technology can do.
We also need to think about what it should do.
Every new capability creates new possibilities.
But every new possibility comes with design decisions.
Where should AI act?
Where should it ask?
Where should it recommend?
Where should it wait?
Where should a human make the final decision?
These are not only AI questions.
They are product and engineering questions.
And I believe the companies that answer them thoughtfully will build AI products that people can actually trust.
What's Next
In the next edition of The Builder's Letter, we'll explore another fundamental part of AI:
The Data Problem in AI
Because even the most capable AI system is limited by the information it can access, understand, and trust.
We'll look at why data quality, context, privacy, and architecture may ultimately matter more than the model itself.
Until then, remember:
The goal isn't maximum automation.
The goal is meaningful automation.
And sometimes, the smartest system is the one that knows when to let a human take control.
— Mian Usman Khalid
Founder & CEO, XICTEK Systems
The Builder's Letter — Building practical technology for real-world problems.
Top comments (0)