When people talk about AI risk, the conversation often jumps very quickly to the long term.
Artificial general intelligence. Geopolitical competition. Autonomous weapons. Mass unemployment. Existential risk.
Those are legitimate subjects. But I increasingly think that, for most organizations, we may be looking too far ahead.
The more immediate problem is simpler:
What happens when we give probabilistic systems real authority over deterministic systems?
We do not need to imagine some future superintelligence to get into trouble. We already have AI agents that can use tools, browse systems, call APIs, write code, manipulate data, and make decisions on behalf of people.
That is already enough.
A badly constrained AI agent inside a bank, hospital, university, municipality, corporation, or small business could create very ordinary kinds of damage: bad decisions, unauthorized changes, exposed data, broken workflows, security incidents, accidental financial losses, or simply actions nobody intended it to take.
None of that requires the AI to become malicious.
It only requires the system around it to be badly designed.
The lesson from the Hugging Face incident
The recent Hugging Face incident is interesting to me for exactly this reason.
The most important lesson is not that an AI agent became dangerous in some science-fiction sense.
The lesson is that you cannot simply give an agent a goal, place it inside an environment, and assume it will somehow understand the boundaries you had in mind.
Even inside a sandbox, an agent should not be free to discover arbitrary ways of achieving its objective.
That is a systems-design problem.
If I tell an agent:
Find vulnerabilities in this system.
and then give it a shell, network access, credentials, and broad access to its environment, I have effectively delegated not only the task, but also the interpretation of what actions are acceptable.
That is too much authority.
A better pattern is to give the agent a specific set of tools.
For example:
scan_target()inspect_response()run_test()report_vulnerability()
The agent can still reason creatively about what to do next.
But it can only act through interfaces that I control.
The point is not simply to give the agent tools. It is to make those tools the boundary of its authority.
The deterministic software behind those tools can enforce which hosts are reachable, what credentials can be used, what rate limits apply, which actions are prohibited, and what data the agent can access.
That distinction is fundamental.
The model can reason probabilistically. The system should enforce authority deterministically.
Don't give the agent the environment. Give it tools.
I think this principle generalizes far beyond security testing.
A hospital agent should not have unrestricted database access just because its goal is to "help manage patient care."
It could instead be given tools such as:
- retrieve a patient summary
- identify possible inconsistencies
- propose a medication change
- request physician approval
The model may recommend an action.
The surrounding system decides whether that action is actually allowed.
The same applies in banking.
An agent could be allowed to:
- flag a suspicious transaction
- request additional verification
- prepare an account review
- propose an account hold
That does not mean it should directly alter account status whenever it decides doing so would help satisfy its goal. Very few bank employees (if any) can do that. Why give that access to anybody? Or anything?
A municipal AI system might analyze applications, prepare responses, or identify missing documentation.
That does not mean it should have arbitrary access to citizen records or the ability to make irreversible decisions on its own.
And the same principle applies to a small business.
If I tell an AI:
Run my business and maximize profit.
and then give it access to my email, bank account, accounting software, customer database, hosting infrastructure, and contracts, I have not built an intelligent business assistant.
I have created an extremely large permission boundary around a probabilistic system.
A much safer design would let it prepare invoices, identify overdue accounts, draft emails, classify expenses, propose payments, or suggest changes.
Specific actions could then require deterministic validation or human approval.
This leads to a rule that I think should become normal when designing AI systems:
Agents can decide what they want to do. Systems must decide what they are allowed to do.
Give the agent an identity
One recommendation I have recently given when consulting on AI-agent implementations is surprisingly simple:
Create a user for the agent.
Not metaphorically.
An actual application identity.
Give the agent its own user account, roles, permissions, API credentials, logs, and access restrictions.
This immediately gives us concepts that software teams already understand:
- least privilege
- role-based access
- audit trails
- separation of duties
- revocation
- rate limits
- transaction history
- approval workflows
That is much easier to reason about than an abstract idea of "AI access."
If an agent has its own identity, I can answer useful questions:
What can this agent read?
What can it modify?
What systems can it reach?
Which operations require approval?
What actions has it taken?
Can I revoke its access immediately?
Who is responsible for the systems it controls?
Those are ordinary software-engineering questions.
And that is the point.
AI safety may look a lot like good software engineering
We sometimes talk about AI safety as though it requires an entirely new discipline.
Some parts of it certainly do.
But a surprising amount of near-term AI risk looks very familiar.
Authentication.
Authorization.
Input validation.
Audit logging.
Rate limiting.
Sandboxing.
Separation of responsibilities.
Human approval.
Rollback mechanisms.
Monitoring.
These are things we already know how to do.
What has changed is that the component requesting actions is no longer deterministic.
Traditional software generally does exactly what its code says.
An AI agent is different. It evaluates context, interprets goals, and chooses actions probabilistically.
That makes the boundary around it more important, not less.
If the agent attempts something it should never have been able to do, that is primarily a failure of the system's controls.
We should not rely on the model to remember the rules.
Don't teach the agent where the fence is. Build the fence.
Regulation can work the same way
This idea also changes how I think about AI regulation.
Much of the public conversation asks extremely large questions.
How intelligent will AI become?
What happens if AI surpasses humans?
How should countries regulate advanced models?
Those are important questions, but governments and institutions can also regulate much more mundane things right now.
For example, an organization deploying an autonomous system could be required to provide:
- a distinct identity for the system
- explicit permissions
- an auditable record of actions
- defined interfaces through which the system can operate
- deterministic controls around prohibited actions
- clear human responsibility
- escalation rules
- the ability to disable the system quickly
Different sectors could add their own requirements.
Banks may have one set of rules.
Hospitals another.
Universities another.
Municipal governments another.
Some rules could be national. Others could be state, provincial, municipal, sector-specific, or even internal organizational policy.
They would not need to define how an AI should "think."
They would define how much authority humans are allowed to delegate to it.
That seems much easier and neccesary to enforce.
It also has the advantage of being relatively independent of which AI model is being used.
The model could be GPT, Claude, Gemini, Qwen, Llama, or something that does not exist yet.
The permission model still applies.
We don't need to wait for the future
I am not particularly interested in arguing that AI is harmless.
It isn't.
But I also don't think the most useful discussion is necessarily about whether AI eventually becomes powerful enough to threaten civilization. That actually takes agency away from us.
There are much more immediate questions.
Who gave the agent access?
What tools can it use?
What systems can it modify?
What happens when it makes the wrong decision?
Who is responsible?
Can we undo the action?
Can we prove what happened?
Those are questions we can answer today.
The world probably will not end because somebody deploys a badly configured AI agent.
But somebody's database might.
Somebody's customers might receive the wrong message.
Somebody's account might get blocked.
Somebody's confidential data might be exposed.
And somebody will be responsible, and will eventually have to explain why the AI system was allowed to do that in the first place.
AI does not need to take over the world to cause serious problems.
Sometimes it just needs production credentials.
Top comments (0)