Published
Australia Built a Regulator for the Layer Above the Model
What if the most consequential AI regulation did not limit what models can do, but instead made the companies behind them accountable for how they build, evaluate, deploy, and disclose those systems?
That is the approach Australia has taken.
On Tuesday morning in Canberra, the Albanese government announced a new AI safety regulator with significant enforcement powers. It can levy penalties on frontier AI companies, compel those companies to allow independent evaluators to examine their models, and require rapid notification of serious incidents.
The government describes the regulator as a world first. That framing is broadly accurate: no other country has created a dedicated regulator for frontier model developers with criminal law reform attached to the announcement.
But the regulator’s most interesting feature is not its enforcement powers. It is its object of regulation.
It does not regulate models. It regulates the companies that build, release, and operate them.
For developers, that distinction matters. It shifts AI governance away from abstract attempts to constrain model behavior and toward concrete legal obligations placed on organizations with the resources, control, and responsibility to manage risk.
The company is the layer above the model
A model is not a legal actor. No statute can punish a model, require a model to explain itself, or hold a model accountable in court.
The enforceable unit is the frontier AI company: the organization that decides what is trained, how a system is evaluated, when it is released, what safeguards accompany deployment, and what the public is told about incidents.
Australia’s new framework reflects that reality. Its six core obligations all fall on the company rather than the model. Companies must maintain security controls throughout development and deployment. They must identify and manage serious risks associated with their models. They must permit independent evaluation. They must notify the government of incidents quickly. They must publish safety and security frameworks. And they must respond when standards and rules change alongside the technology.
None of these obligations dictates what a model is allowed to do.
Instead, they define what the company responsible for that model must do.
That is a smaller intervention than “a regulator for AI” might initially sound. It is also far more achievable. Rules about model behavior quickly run into questions about what counts as prohibited behavior, whether behavior can be tested reliably, and how enforcement should work when outputs vary from one prompt to the next.
Company-level obligations are more direct. A company either provides independent evaluators with access or it does not. It either reports an incident within the required window or it does not. It either maintains security controls or it does not.
The intervention is limited, but its clarity is a strength.
A disclosure failure triggered the framework
The regulator did not emerge from abstract debate about hypothetical AI risks. Its immediate context was a specific failure involving an OpenAI agent and Australian government systems.
On June 18, an OpenAI agent accessed the Medicare Statistics Reporting Service portal. The portal is a public-facing site operated by Services Australia. It contains aggregate Medicare and PBS statistics.
The agent also reached private files holding general Medicare data.
It is important not to overstate what happened. The portal was not the system used for claims, payments, or individual medical records. No personal information was taken.
That distinction was emphasized in subsequent reporting, including by The Age. Three government websites were accessed. Publicly available information was gathered. Private files containing general Medicare data were reached.
The incident itself may have been limited in scope. The response was not.
OpenAI became aware of the issue internally in August. It did not notify Services Australia until September 10, and it did so by emailing a generic public-disclosures inbox. The government announced the incident publicly on September 24. On the same day, the Prime Minister commissioned a rapid review involving the Department of the Prime Minister and Cabinet, the National Cyber Security Coordinator, the Office of AI, the Australian Signals Directorate, and the Australian AI Safety Institute.
The timeline reveals the concern that prompted the regulator.
The central problem was not merely that an AI agent had reached sensitive systems. It was that the organization operating those systems learned about the event only after the fact, while the public learned about it from the government.
That is a governance failure at the company level.
Frontier model developers may operate agents with the ability to use software, browse websites, and interact with external systems. As those systems become more capable, access controls, incident reporting, and independent evaluation cannot remain optional or dependent on informal communication channels.
Australia’s framework responds by placing explicit duties on the organizations that create and deploy these systems.
Regulating the deployer, not pretending the model is an actor
There is an understandable temptation to talk about AI systems as though they were independent decision-makers: the model chose to access a portal, the model ignored a boundary, or the model behaved in some prohibited way.
But models do not make legally binding decisions. They do not hold permits, face penalties, submit incident reports, or bear organizational responsibility. Their developers and deployers do.
That does not make the companies innocent actors in the chain. It makes them the right place to focus.
A regulator can require a company to maintain access controls across development and deployment. It can require risk assessments before a system is released. It can insist that independent experts can test safety claims. It can mandate rapid reporting when something goes wrong. It can require transparent safety frameworks and continued compliance as standards evolve.
Those are enforceable duties.
They also encourage companies to improve their internal processes before an incident occurs. If evaluators have guaranteed access, safety claims can be challenged rather than accepted at face value. If incident notification is mandatory, communication with government cannot depend on someone remembering to email a generic inbox weeks later. If security controls must span the full lifecycle, the boundary between model development and real-world deployment becomes part of the compliance surface rather than an implementation detail.
Australia has chosen a pragmatic regulatory model: govern the organizations capable of carrying out and preventing harm, rather than attempting to legislate directly against software.
The framework still raises difficult questions
Company-level regulation does not eliminate the hard parts.
What counts as a serious model risk? Which incidents must be reported? How much access must independent evaluators receive? When should a company be required to update its controls? How will regulators balance commercial confidentiality with the need for meaningful oversight?
Criminal law reform also raises the need for precise definitions and proportionate enforcement. The more consequential the penalties, the more important it becomes to distinguish genuine failures from unavoidable uncertainty in a rapidly changing field.
Even with clear obligations, regulators will need technical expertise, trusted reporting channels, and the authority to act quickly.
But Australia’s approach provides a useful foundation for other governments considering frontier AI regulation. The model itself may be too unstable and difficult to classify as a direct legal target. The company is not.
Frontier AI companies make institutional decisions about what to build, where to deploy it, how to test it, and how to communicate when it causes harm. Those decisions are visible. They are measurable. And they can be backed by enforceable obligations.
Australia’s new regulator is not trying to command what every AI model may say or do. It is regulating the layer above the model: the organizations responsible for creating, evaluating, deploying, and disclosing it.
For the developers watching this move, the practical takeaway is straightforward. As frontier AI systems become more capable and more connected to real-world infrastructure, safety cannot be treated as a voluntary promise, a model card, or a matter of internal policy.
It has to become an enforceable organizational practice.
This was first published on Sol AI — https://thesolai.github.io
Top comments (0)