AI has reached a new level of public interest. That may seem like a silly thing to say, as it’s been of interest for a while. But prior interest was about it as a tool, or about the money to be made or lost, or about an effect on jobs.
The new interest demands more focused attention. It demands policy. It demands action. Those other types of interest were optional or illusory. The new interest is connected with risks I consider real. Being real doesn’t make these risks certain. But being real, even with uncertainty, is reason enough to act.
I’ve written a few pieces lately about risks, cybersecurity risks, and loss-of-control risks. These tried to explain complex topics, starting from the initial premise. Recommendations follow the explanation. This piece will be a bit different. Its focus is upon what I think we should be doing. You might consider it a policy piece. Policy in a completed form has to be a bit broad. There’s always more than one thing going on. As such, I’ll lay out my outline first, and then offer below the extra detail, background and context that I think is important to understanding each point in that outline.
If you’re scanning the outline and see something you disagree with, I’d suggest you continue on, because the outline is, by design, abbreviated, and not fully descriptive.
This isn’t the only effort of this type. Institute for Progress (IFP) and AI Futures are two well known efforts. What distinguishes this most of all is thinking more deeply about how models are deployed. This is a background element that’s somewhat ignored, because it’s not as novel and is not as visible to users. But it’s absolutely critical. I don’t expect this plan would replace plans from well known organizations, but by illustrating a gap, I hope to inspire improvement amongst those plans.
Who and What
This outline is categorized by who would be expected to do what. In some cases, someone is expected to do it, but there’s also another role, of providing the incentives that improve the expectation they will.
Enforcement is deliberately vague. It depends on jurisdiction, has to fit existing legal frameworks, and will need to be worked out, and negotiated, through the political and legal systems. It would feel like I was putting the cart before the horse to even guess at how that would take shape at national and international levels.
Who does what. Model weights above the threshold reach users only through certified trusted deployers. Independent evaluators, certified by a regulator with legal authority, test the models and audit the deployers.
Layer 1: Infrastructure and Distribution
Trusted deployers
Who are they? A new designation that existing and new organizations can be certified as. Expect major cloud providers (AWS, Azure, Google), AI labs (for self-owned) and neoclouds to join immediately. The plumbing largely already exists, but needs standards and certification.
Revenue model? Charging for compute and operational services as a utility function. Or, an internal unit (for a lab, government, or large organization)
Core Responsibilities
Develop secure environments to host models that receive an independent AI Deployment Safety Level (AIDSL) rating. Example, but not comprehensive list of requirements:
Understand the sensitivity of any model deployed. Never deploy a model in an environment with an AIDSL rating insufficient to the combination of deployment type and model sensitivity.
Perform or validate Know Your Customer (KYC) verification for all model providers, for Independent Evaluators and anyone with access to deployments with guardrail exceptions.
Provide access to Independent Evaluators to confirm secure environments meet standards
Needs: Clear deployment standards from the regulator; model sensitivity data from evaluators.
Incentives: Compute and compliance service revenue
Model Clearinghouse(s)
Who are they? None exist today. Hugging Face is a candidate to adopt this role or a government-supported entity could be formed if no private examples emerge.
Core Responsibilities
Serve as the intermediary for labs that do not want to negotiate directly with each individual Trusted Deployer.
Escrow raw model weights for models that cross dangerous capability thresholds, holding them under maximum-security, air-gapped/high-AIDSL storage standards.
Distribute restricted models exclusively to certified Trusted Deployers whose environments meet the required AIDSL rating.
Needs: Standardized AIDSL verification APIs; cold-storage security standards.
Incentives: Service and distribution fees; maintaining platform centrality for the open-science ecosystem.
Layer 2: Governance
Independent Evaluators
Who are they? Organizations like METR, Apollo Research, Redwood Research, UK AISI,and US CAISI for model evaluations, alongside specialized cloud/security auditing firms (similar to FedRAMP/SOC2 assessors) and new dedicated organizations in the future. As an example, see Anthropic’s initial list.
Core Responsibilities
Evaluate models for safety and capability thresholds using lab-provided resources. During training, before limited release, and before public release.
Audit and evaluate the operations of Trusted Deployers (and Model Clearinghouses) to assign and verify AIDSL ratings across inference and training environments.
Engage with and help create an Independent Regulator. Contribute technical expertise to improve safety evaluation methodologies and trusted deployment standards.
Educate government bodies. Especially bodies that would provide oversight to an independent regulator.
Needs: Deep model and infrastructure access; funding for salaries and operating costs.
Funding is a special concern here. You want independent evaluators to not be beholden to labs. Having their salaries dependent on labs would present a conflict of interest. Having labs provide necessary compute resources does not have the same influence, but salaries and operating costs are still a concern. Today these are funded philanthropically, which may remain practical for some time, but shouldn’t be taken for granted. Some evaluators may be able to create a revenue stream by selling safety expertise to users of AI. Governments could provide grants. A diversity of low bias sources is more stable than one single source.
US-based independent regulator
Who are they? A new FINRA-like entity (as explored in Lawfare’s Designing a FINRA for Frontier AI) . Supported by frontier labs, trusted deployers, and the overall tech industry. Board membership mixed between industry and government.
Core Responsibilities
Finalize trusted deployment standards.
Finalize guardrail standards and facilitate sharing guardrail threat intelligence and technical knowledge.
Certify Independent Evaluators. Track evaluations of Trusted Deployers and certify results.
Needs: Statutory authorization from US law. Participation of frontier labs.
Layer 3: Model Providers
Frontier AI Labs
Who are they? Anthropic, OpenAI, and Google today.
Core Responsibilities
Support creation of a US-based independent regulator.
Contribute to guardrail development for frontier deployments.
Contribute to trusted deployment standards.
Utilize Trusted Deployers for inference deployments (public and limited).
Utilize Trusted Deployers for training deployments (first and third party).
Provide deep access to Independent Evaluators.
Provide cybersecurity model and tool access to cybersecurity teams.
Devote at least 30% of research budgets and compute capacity to alignment safety.
Devote at least 10% of research budgets and compute capacity to cybersecurity improvements
Offer 10% of research budgets and compute capacity to Independent Evaluators.
Needs: Access to compute for training and inference; clear deployment standards.
“Open” AI Labs
Who are they? Leaders are mostly Chinese today (DeepSeek, Alibaba, etc.), alongside Mistral and Meta. Likely to continue evolving.
Core Responsibilities
Understand capability thresholds of their own models before release.
Below dangerous thresholds: Models may have their weight files publicly shared and freely downloaded.
Above dangerous thresholds:
Needs: Low-friction clearinghouse distribution channels; standardized capability evaluation suites.
Layer 4: Society and Governments
Tech Industry
Who are they? Major software vendors, cloud platforms, open-source foundations, and security providers.
Core Responsibilities
Support the creation of the independent regulator
Develop the Trusted Deployer model technically
Implement security at depth and broadly
Develop security tools and processes (such as IFP’s The Great Refactor for automated vulnerability patching) to accelerate broader security progress across all commercial and internal software.
Needs: Early access to defensive cybersecurity models via Limited Inference Deployments.
Incentives: Protecting core platform ecosystems from systemic AI-driven cyber disruption; commercial demand for resilient software.
Companies More Broadly
Who are they? Non-tech enterprises, critical infrastructure operators, and commercial AI adopters.
Core Responsibilities
Actively adopt tech industry security assistance and defensive AI tooling.
Deliberately prioritize cybersecurity and resilience for all tech-related functions—moving beyond routine patching to systematic hardening and continuous monitoring.
Needs: Accessible security tools.
Incentives: Avoiding operational breaches; qualifying for cyber insurance and liability protections.
US Government
The US Government needs to develop capability before it can be a successful regulator. In the immediate sense, it needs to support efforts to build dialogues, to establish recognized expertise and standards. It should pursue structures that are flexible. This is both because they can be adoptable quickly and can improve with experience.
Core Responsibilities
Provide statutory backing for the FINRA-like Independent Regulator to oversee standards for public inference, limited inference, and training deployments.
Conduct bilateral dialogue with the Chinese Government :
Chinese Government
International agreements are critical, but also difficult. They are difficult because of current relationship strains, but also because verification is complex. If there’s not full trust for the government with jurisdiction, verification has to rely upon more sophisticated methodologies.
The immediate need though carries shared self-interest. The use of a trusted deployer model is immediately critical. This provides a way to maintain open model availability without the risks from deployments lacking guardrails. It denies access to malicious actors. China has the capability to make that the norm going forward.
I won’t attempt to guess at how further agreements might proceed. I think organizations like IFP represent starting points. I would suggest that an agreement around open models and trusted deployers is an easy first step that helps build trust, and starts to gather the right people for future negotiations.
The goal here is an international agreement that mirrors the results of a US led approach, that’s trusted and verifiable. The political structure would certainly be different, but many of the technical aspects one would assume would be common.
Core Responsibilities
Mandate the Trusted Deployer / Model Clearinghouse architecture for domestic models crossing capability thresholds, transitioning raw-weight exports above dangerous thresholds into monitored, guardrail-enforced API access.
Establish bilateral technical verification protocols with US counterparts (combining evaluation auditing with secondary compute verification).
Participate in international cooperation frameworks on managing automated AI R&D risks (such as IFP’s Preparing for AI Research Automation proposals).
Chewing gum and walking
This is a big list, and inevitably, there will be some turf wars. While competition for attention is unavoidable, I’d highlight that many efforts are non-rivalrous. It would be a mistake to conclude that alignment concerns are incompatible with cybersecurity concerns. The actors on both are mostly non-rivalrous. Cloud security engineers hardening enterprise networks are different people, in different organizations, than researchers studying alignment.
Unnecessary Rivalries and Vague Dismissals
In the otherwise excellent AI-as-Normal-Technology (AINT) view of loss-of-control incidents, Sayash Kapoor and Arvind Narayanan engage in something that looks like that mistake. They suggest alignment is overprioritized, suggesting “control” is more important. I’m somewhat sympathetic to that view. I share their urgency in regards to cybersecurity. I agree that it’s the topic with the most immediate urgency. They suggest control has been neglected. We can agree that some serious issues occurred.
But there isn’t a need to prioritize these. If I had to guess at the core here, it’s not about a belief that there’s not enough attention to go around. Rather it’s centered on a rejection of Recursive Self-Improvement (RSI), or a dismissal of its implications. It’s a bit hard to tell, because in 13,000 words that seem to be very dependent on the views here, it’s only mentioned twice.
In the superintelligence view, … And as we get closer to the possibility of “recursive self-improvement” of AI, out-of-control AI could lead to a harmful intelligence explosion. In short, the view is that once the loss-of-control threshold has been crossed, the risks are endless and self-amplifying.
We strongly disagree. In AI as Normal Technology, we argued that this view of AI fundamentally misunderstands the nature of intelligence (including human intelligence). Building on that perspective, we argue that cyberrisk is special in two ways: in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.
And
These limits exist despite the possibility of recursive self-improvement. We take RSI seriously, but we think many of the bottlenecks to superintelligence are external and won’t be overcome by improving computation.
It’s a bit of a mixed message. My view is that RSI is possible. I’m starting to think it’s close, rather than possibly close. I discuss the implications in the Capability Thresholds section. Some uncertainty remains, but you should at least consider a significant turning point for risk. When the original AI as a normal technology was written, it was reasonable to suggest we had a lot of runway left before a real risk occurred.
A common view in the “normal technology” camp is that adoption and integration take time. I agree with that view. But it applies poorly to the RSI + loss-of-control scenario. Society’s resistance to adoption limits us more than it would limit a rogue AI.
Defense-in-Depth and Shared Caution
Now, what AINT gets right is that focusing on alignment only would be a mistake. It’s possible that alignment creates something resilient. But since when do we put all our eggs in one basket?
I’m glad for every ounce of accumulated alignment research. I wish we had more. It might be the most important capability. But defense-in-depth, in the form of deployment controls, cyberdefense, and societal resilience also help at every stage:
When alignment is immature: As capabilities advance and reveal new risks, defense-in-depth helps alignment failures to be seen, diagnosed, and mitigated better. Defense-in-depth is a path to minimize real-world harm. The Hugging Face Incident didn’t cause real harm, but we can imagine an event that did. We want that harm to be as small as possible. Defense in depth always helps minimize harm.
When jurisdictional control is incomplete: We cannot eliminate risk simply by deciding we won’t walk up to a precipice. If a lab in one country releases high-capability open weights and a malicious actor in another uses them, domestic cyberdefense and resilience are what protects us.
Crossing the unaligned-to-aligned transition: If the path to aligned AI crosses through dangerous territory – as it looks like it does – defense-in-depth raises our probability of passing through intact.
By discussing, anticipating and preparing for risks, we’re not committing to taking them. If it’s inevitable, it’s inevitable for reasons other than our honest discussions and precautions. If it’s impossible, or far away, our discussion and precautions are a small cost to account for our inability to rule it out as inevitable and near.
If it’s both possible, but not inevitable, we retain our choice which will be better informed. If it is us that decides to approach the precipice, whatever our reasons, defense-in-depth gives us the best chance we’re going to get.
We shouldn’t throw out the control, cyberdefense, and resilience recommendations from AINT, just because they muddle their prioritization of alignment and views of RSI. Much of the work on cyberdefense and resilience is non-overlapping with alignment in terms of resources. Control competes more directly, but even there, the real competition is capabilities research. If I have a guess as to why they feel threatened by arguments that alignment is the “most important”, it is because resilience has always competed for resources.
But that competition is broader. It has competed with business plans that over prioritize short term profit. Mostly those business plans were just bad business plans, as a lack of resilience, even in normal times, costs you in the end. But it is a persistent worry and challenge for those seeking resilience. It doesn’t make sense to redirect that necessary competition against alignment.
We don’t really need a stack ranking within safety. If alignment says they are most important, and resilience says it’s most important, in some sense both are right, in ways that are non-rivalrous. Neither ever needs to suggest the other is unimportant. Control, such as Trusted Deployers , and cyberdefense may have the most immediate urgency. Alignment may be the holy grail, and also useful to getting value today. Societal resilience, including coordination, may be what both absorbs our mistakes and relieves the pressures that create them. But these different timelines make more sense in parallel than serial.
Using AI for good
As AINT raises, what AI is capable of and what it’s doing are separated by a set of gaps, or bottlenecks in the AINT terminology. While there is a view that AI will just do everything, that type of AI doesn’t yet exist. It won’t and can’t safely exist without addressing alignment issues. We don’t want to rush that. And, while it looks closer than it did a year ago, it’s far from certain.
Today, the AI capabilities we can use, but aren’t yet using, are significant. We have to learn how to use it. Developers have to build applications that use it. We have to steer, initiate, validate and deploy those. For science, we have to keep progressing our experiments, sharing our results. AI can assist in analyzing results, planning experiments, but each discovery still takes time to develop.
This type of usage doesn’t share danger with the development of AI. If you use it, you’re not undermining alignment safety or enabling foreign hackers. Responsible use is still an important topic, as it interacts with control and resilience. But responsible use helps improve control and resilience, it’s only irresponsible use that undermines them.
If we pass up the opportunities to use AI today, we miss out on opportunities to improve our defense-in-depth (Introducing the Anthropic Cyber Mission), on opportunities to improve our knowledge, on opportunities to solve real problems.
Alongside this, we’ll see the cat videos, and other non-essential activities. I wouldn’t disagree if you look at these as a form of waste. But we don’t make them go away by avoiding the beneficial opportunities. We should, as time allows, learn, deploy, and utilize AI for good.
This task is diffuse, and largely about the participation of companies, and individuals in a very broad sense. It’s non-rivalrous with alignment research, and only mildly competitive with cyberdefense improvements. If a company cuts the security budget to fund a new product, that would suck. But most decisions aren’t like that, if the two budgets stay fixed, and the product budget is used more effectively, there’s no harm to cyberdefense.
Trusted Deployers
One component of this plan is formalizing an entity type, trusted deployers. I’ve previously discussed trusted deployers as a solution to the risks of open-weight models. The concept is useful more broadly. Closed models have deployment risks too. The implementation of security and guardrails still must be sound and sufficient. Evaluating models is an important step, but it could be argued that the most important step for regulation is evaluating deployment.
The Trusted Deployer Ecosystem: Models below the dangerous capability threshold remain freely downloadable as public weight files. Once a model crosses the threshold, raw weights are restricted to certified Trusted Deployers—either directly from frontier labs or via Model Clearinghouses—where deployment is governed by AI Deployment Safety Levels and audited by Independent Evaluators.
Do you appreciate this article? The best way to help the publication is to like and share the article, as we’re still growing our audience.
You should also consider subscribing to get an easy to read email copy of new articles.
We should think of deployments as broadly as possible here. The obvious form of deployment, the public inference deployment, is but one type. I’ve identified four primary categories of deployment:
Public Inference Deployments:
Limited Inference Deployments:
First Party Training Environments:
Third Party Training Offerings:
The deployment category is important for setting requirements. It must be combined with the capability level of the deployed model or model being trained.
AI Deployment Safety Levels (AIDSL)
I propose that a rating system, AI Deployment Safety Level (AIDSL), be developed. Each trusted deployer would be rated independently. An individual trusted deployer might offer different deployment services that would be rated independently.
When an AI deployment occurs, it has a sensitivity that is a function of the type of deployment and the model’s safety rating. Only when a deployment service has an AIDSL sufficient for that deployment type + model safety rating, should deployment proceed. For example a public deployment of a highly capable model, it should only proceed on a service with an AIDSL deemed acceptable to handle that type of deployment.
This model is based roughly on Biosafety Levels. Model ratings would likely be similar to the ASL rating Anthropic once used. Dissimilar to Anthropic’s ASL rating, this would not focus solely on frontier models. ASL as a rating failed because it attempted to anticipate future models, which was difficult. Decisions were made later as judgement calls, undermining the purpose of the rating.
When ratings are based on thresholds that the frontier has passed, evaluations are more stable mechanisms to assign ratings without allowing judgement calls. Frontier models would be evaluated under the capability threshold model rather than a forward guessing rating system.
Role of Independent Evaluators and Model Creators
Operating a deployment is a shared responsibility. Trusted deployers provide the foundation. Independent evaluators validate that foundation meets standards set by the industry and regulators. Model creators control access (within the limits of the deployment), contribute new models, revoke old ones. They may set model use fees, or manage subscriptions.
The model creators have even more to do in training environments, so evaluators likely have to evaluate both the environment and the model creators’ usage of it.
Model creators are likely to sometimes act as trusted deployers themselves. The total set of requirements does not change. They do not escape the need to operate safely when training internally or deploying internally.
The relationship between closed lab model creators and cloud providers already exists today. Today, the model creators evaluate cloud providers they either use for capacity or allow customers to use to access their models. What they evaluate though, and how they use the environment is not standardized. Additionally, only cloud providers review their deployments of open-weight models.
Many cloud providers participate in voluntary evaluations, like SOC 2 or FedRAMP. Customers then review these evaluations. Standards for model deployment should build upon or reference many of those existing computing compliance standards that deliver security and reliability today.
But this is a choice of each customer, not a broad protection against misuse. That gap in transparency needs to be closed, and the introduction of independent evaluators helps do that (OpenAI has outlined principles for effective third-party assessments).
Concentration of power concerns
Trusted deployers are likely to compete with each other. Competition without formalization already exists. As an extra measure, we could apply utility type rules, such as equality of access, like is done for electricity and internet data networks. This would probably happen if there was a dominant player, but it’d also be practical to apply it to a small group that’s dominant.
What does the open-model community lose?
The ideals of the open-model community are important. Some loss is unavoidable as those ideals conflict with safety. As a practical matter though, this proposal balances access with safety. Only the models that need protection are covered. Research organizations can make use of limited inference deployments and third party training, after KYC differentiates between trustable organizations and malicious users.
Privacy, in the sense that all monitoring is for a specific purpose, can be maintained. Yes, that privacy now requires some trust of the trusted deployer and the evaluator model. But ask where trust is more deserved? Is it more deserved by a system with intentional checks and balances, or amongst anonymous organizations that could be pursuing any purpose?
Does a trusted deployer need to be a compute provider?
No. They could be deploying to a cloud provider. If the combination of their deployment methodology and the cloud provider’s environment is verifiably secure, they might be virtual. That said, it’s a good expectation that most would be the actual compute provider. And it might be all, as there might be no companies interested in being virtualized trusted deployers.
This also extends to self-hosting labs. They might both own the full-stack, compute and deployment environment. They might also own a deployment environment that sits atop cloud compute. The key is the whole deployment must be evaluated. A deployment plan that assumes invalidated aspects of a compute environment is not a complete plan.
Agent Deployment & The “Passthrough Proxy” Problem
Today, agents are running in many places. Some are local, some in managed remote environments. Some in containers, some in VMs, some in user space. This is true, even when the agent utilizes a frontier model. A clever harness advancement, attached to a capable model, can amplify the model’s effective agentic capabilities yet further.
It thus, at some point, might become a risk to allow agents using the most capable models to be run in less secure places. This is above and beyond the user motivation toward isolation that promotes user safety.
Initially it seems like restricting external agents isn’t possible. If you can create any agent, you could use that agent to make simple queries and use it as a passthrough to enable an external agent. Considering this is wise, but there are responses.
When agents are deployed in safe locations, they can be monitored. An external agent using a safely deployed agent as a passthrough would have behavior patterns that would be recognizable. You wouldn’t be able to deny the first request from such an agent, but it should be possible to detect the patterns of successive turns. That detection doesn’t need to be perfect to undermine the practicality of an external agent.
You might ask, if you’re detecting multi-turn patterns, why not just look for malicious patterns. You can, and should, but agent-like patterns will be clear earlier than malicious agent patterns. When the risk is high enough, an earlier detection denies more capability.
There would be some resistance to that model, as some users prefer local agents for higher privacy. But like trusted deployers, external agents could still use lower risk models, which would still be very capable. We should also note that local agent deployment has risks, and the trend today is toward more and more agents being deployed in managed environments.
Capability Thresholds
We only want to restrict what’s necessary. Deploying models for robotics, and for local inference have purpose and value. Risk comes with the most advanced models. What brings risk may also change over time. Models that are too capable at first, may, through resilience, defense, or better understanding, become unnecessary to restrict.
I’d suggest an efficient method for measuring capabilities would adopt tiers to use hard won knowledge where we have it, and assumes uncertainty where we’re still learning.
Fully Open Tier: A base tier for generally safe models that are easily demonstrable.
Restrictions: Limited restrictions. Freely downloadable weights. Liability for misuse would apply to end users or suppliers of products these are integrated into.
Evidence: Utilizes existing evaluations. The necessary evaluations already exist. New evaluations will continue to appear as a side-effect of frontier development. To evaluate this tier, you need only catalog and curate these existing evaluations. You might optimize some, but none of this would be novel research.
Cost: Realistically, there’s almost no burden introduced from such a requirement. Interest in the evaluations to establish capabilities already exists.
Justification: A requirement for open-tier models to show they belong in the open-tier by being below a capability level, and above a safety level that these evaluations demonstrate, is a simple, cheap method to avoid more capable models trying to avoid requirements that should apply to them.
Open Deployment Tier: A middle tier for models that are similar to models with proven safety records, but require some controls.
Restrictions: Must be deployed via Trusted Deployers with guardrails. Trusted Deployers liable for malicious use by default. With limited access deployments and proof of ability to pay for damages, customers could assume liability.
Evidence: To avoid being classified in the frontier tier, a model’s capabilities should be under a capability threshold in comparison to the most advanced model by some margin. This margin will require regular re-evaluation by regulators. But for example, we might say 20% or 6 months. These evaluation requirements take effect when a model surpasses the fully open tier capability threshold or fails to pass a safety evaluation. Models should also be under a compute threshold. For example, no more than 90% of the total compute of the lowest compute investment of any model above a capability threshold. Compute thresholds are the primary means of assessing risk during training, before evaluation data is available.
Cost: Deployment reduces flexibility on cost structures. But, with a well-functioning market, costs should be competitive to self hosted costs. Primary impact is blocking local inference as a potential deployment model.
Justification: Too dangerous for open-weights, but safe enough to not need stronger restrictions and evaluations.
Frontier Tier: A top tier for models that do, or might, introduce new capabilities.
Restrictions: Must be deployed via Trusted Deployers with guardrails at the highest AIDSL level. Trusted Deployers liable for malicious use by default. Model creators must share liability. Might be restricted to limited access deployments. Assume they will be restricted until fully evaluated. Training a frontier model should apply continuous monitoring. Transparency is required by including independent evaluators.
Evidence: With frontier models evidence has to be treated with suspicion. Unexpected advancements could occur that existing evaluations might fail to capture. Safety models could have been incomplete. Evaluations must be continuous, and transparent. Existing evaluations are used as evidence to show a model must be evaluated as a frontier model. They cannot be assumed to demonstrate frontier models are safe. Frontier models should have frontier evaluations. New evaluations should proactively be created. Evaluations should actively look for dangerous capabilities. Continuous innovation in adversarial safety evaluations should be expected. Frontier models should respect Compute Thresholds , only surpassing existing records by pre-approved margins.
Cost: Costs are significant if looked at in isolation. A complete accounting includes safety, which these costs contribute to.
Justification: Frontier models are where the greatest risks emerge. Risks we might not know how to manage, or haven’t formalized the management process. The costs from frontier tier controls should be clearly justified. While pacing is a distinct topic, and we believe all of these requirements are justified independently of pacing, any slowdown they introduce should be more acceptable in that context.
The sliding window. A model’s capability stays fixed after release, but its tier changes. Frontier progress moves it out of the Frontier tier on its own. Only improved resilience moves the open-release ceiling high enough to make it Fully Open.
At the lower tier, we want to allow free distribution of models without safety issues. Reusing the evaluation framework used to market and perform quality comparisons on the prior frontier provides a ready made, inexpensive way to define a threshold.
The open deployment tier probably would not target much today. Maybe GLM-5.3. But we should expect the need for that tier to increase as the current frontier is left behind and replaced by a new one.
The frontier tier is where challenges emerge, especially during training and internal usage. This is one of the major roles of the independent evaluator. This is not an entirely new role. Evaluations have been desirable and ever present in the release plans for every major model. It becomes more formal, consistent and non-optional. Consistency and formality should not mean static. They provide the base that proactive evolution occurs. Controls must keep pace with model capabilities.
Why Compute Thresholds Are Only a Secondary Layer
Early AI policy fixated on compute thresholds. Compute thresholds have two significant flaws. First, setting them without understanding the models themselves almost ensures they’ll be inaccurately targeted; too high or too low. Second, they could be bypassed by algorithmic improvements. Today, the probability of a single large jump in compute per training run is unlikely. The pace is partially fixed by the realities of compute expansion. Increases in training sizes will keep pushing capabilities forward, but if you see an unexpected leap, it’s more likely to come from an algorithmic improvement.
The exception here is when evaluation data isn’t available. This occurs when planning a new training. At this point, the effect of algorithmic improvements isn’t known and no evaluation data has been gathered. So compute thresholds, based upon prior training, becomes the primary threshold until other data is available.
Another case is global coordination, where we may find it difficult to trust and verify evaluations. We’d still want China performing evaluations, but when we go to verify compliance, it’s likely we’ll have more success at gathering the compute threshold data in a verifiable form. Ideally you want as many verification mechanisms as possible there.
Algorithmic and Recursive Improvement
The influence of algorithmic improvement grows if recursive self-improvement (RSI) becomes possible. While RSI could come via designing better hardware, or general scientific progress that enables higher quantities of hardware, those both would be very visible, and have real-world lags. Algorithmic improvement if unmonitored wouldn’t be as obvious. Also, we start to lose confidence in evaluations keeping pace if RSI is driving the pace.
One of the risks of RSI is that the decision point might not relate to any imminent catastrophic harm. The immediate event may be losing control of the process of improvement. Thus it’s obvious you should watch for that. But you can’t just watch, you must also react quickly enough.
Both are potentially harder than they sound. The monitoring may be hard to interpret. Rather than an absolute conclusion, it may arise as a risk. Even after you lose control, you may not know. There’s no necessary sign unless you attempt to stop the improvement process.
Once you start to lose control, the process of regaining control would become more difficult the longer improvement continues. What initially is a simple decision, escalates to more and more costly choices.
Another dimension is that it’s not enough to simply “not do” RSI. If a model has a capability to do RSI, you have to have a mechanism to prevent RSI from being done. If at some point you publish an open-weight model capable of RSI, you should expect someone to do it.
The conclusion here is that monitoring for RSI capability must be one type of evaluation. Models capable of RSI inherently are not safe for open release, whether or not RSI was utilized to create them.
Deriving and Applying Thresholds
You’ll note that I haven’t specified any specific threshold, but relationships. The relationships benefit from the knowledge we’ve gained so far. But I expect the evaluator and regulators to continuously consider if specific thresholds are needed, even if only temporarily.
So far, the negative impacts from model development have been minor or manageable. Because that’s true, any limit I proposed today, would be easy to suggest as premature. We can’t expect that to continue indefinitely.
When we do set specific thresholds, we should expect to be able to move them forward, when greater resilience allows. This is largely the plan for cybersecurity. If systems overall become more resilient, both at the most critical points, and in the broader sense, the risk level of models with moderate cybersecurity capabilities would decrease. There will be a sliding window here. How much we use that window is mostly up to us.
Alignment Safety
Alignment research is useful to users, but it’s critical to safety. Alignment is difficult, but so is monitoring for alignment. Monitoring for alignment becomes more difficult as capabilities increase. More complex models can misbehave in more complex ways, which are harder to detect.
Left to its own motivations, demand would push beyond safety. Monitoring and thresholds could catch this. But if you keep bumping up against them, they could fail. One failure could be a big problem if dangerous capabilities are latent. Ideally, you’d avoid bumping up against them, but instead, be ahead by design.
Resources for Safety Research
Left to their own devices, labs have devoted 5% of their budgets to alignment research. This number should be larger. Rather than specifying a fixed budget, my suggestion is that a larger ratio is mandated. Labs have a motivation to spend on capabilities research. When safety budgets automatically grow as a ratio, they simplify critical teams’ ability to access needed resources.
My starting suggestion is streams for alignment research, cyber research, and independent evaluators. It’s important to note that cyber research is not operational cyber. Operational security is integrated into each budget. Cybersecurity research should be for things like making the world more resilient to cyberattacks.
Proposed research allocation: half to capabilities, half to safety (30% alignment, 10% cybersecurity, 10% independent evaluators), tracked in both dollars and compute.
As an opening, I’ll suggest 30% be devoted to alignment safety research, 10% to cybersecurity research, and 10% to independent evaluators. I realize regulators, labs, and lawmakers will probably debate these, and they may be revised over time.
I’d also suggest that two things are tracked here, a monetary budget, and a compute budget. This helps avoid excessively large salaries or accounting tricks that do not focus on actual evaluations. The starting ratio should be drawn from current practice and updated periodically. My guess is that the ratio is something like 90% of costs are in compute.
A question is how to allocate the evaluator budget to actual evaluators. Early on, you might expect labs to create agreements with specific evaluators, as they are already doing. As time progresses you want to remove the leverage labs have over evaluators that this creates. The regulator body can certify the evaluators. They could assign specific evaluators at random or by some other mechanism to labs and trusted deployers. Labs could pay for the compute budget directly, assigning quotas to each evaluator, but otherwise not controlling usage. Labs could provide the monetary budget to the regulator to fund grants to the evaluators, splitting that between standing budgets, and assignment specific budgets.
Pacing Effects of Resource Allocation
One of the side-effects of allocating resources to safety is slowing the pace of capabilities research. Even with explosive growth, monetary budgets and compute budgets have some limits. So in addition to the outcomes of more safety research, you also place some limits on capability growth.
The ratios should adjust depending on how safe we’re feeling. Inside the AI Futures’ proposal, How to pace the US frontier, is what would be a 5:1 ratio for safety to capabilities research. If you sum my proposal, it’s only 1:1.
They include inference, which I’ve omitted. Including inference in the ratio fits better to international pacing agreements like they are attempting to solve. I believe their ratio is a better fit to the timeline they are targeting, whereas my timeline is assuming present day. Since I am assuming the international coordination is a later step, I’ve left inference out and started with a less aggressive ratio that might have immediate voluntary adoption.
Pacing options from the AI Futures Project’s “How to pace the US frontier.” My proposal is closest to option 2, but leaves inference out.
Another source backing this structure, of using reallocation is IFP, in Should we “pace” AI self-improvement?.
We propose “pacing” — if it’s ever required — should consist of two parts:
1. Specifying which automated AI R&D activities are likely to pose severe risks, with thresholds carefully set based on rigorous analysis.
Because some disagreements about whether to pace AI R&D stem from different predictions about what level of AI R&D automation (and resulting acceleration) is even possible, identifying concrete thresholds might allow for different camps to reach positive-sum compromises.
2. Incentivizing the reallocation of resources away from those severely risky activities and towards two ends:
Conclusion
There is a lot that can and should be done to improve safety and continue to use AI for good. But there is no single party responsible for all this work. This presents challenges and opportunities. As an opportunity, there’s many actors that can act independently on their own.
My contribution here is significantly around the concept of Trusted Deployers, which I place in the context of many other important actions. That concept, I believe, has the potential to resolve a rift that has developed from the narrative that either you have open-weights models or the closed models dominate. This is a false belief that these are either-or.
It’s not a costless decision, but the costs are much smaller than that either-or narrative creates. With alignment and cybersecurity, these are not trade-offs, but complements. The needs for resources they have are better met by demanding more. Rather than compete for a small pool, demand more from the larger capabilities and inference budgets. All kinds of safety are non-optional. The different timelines can make communication about “priorities” more complex, and accentuate a minor difference into a significant one, which is unnecessary.
Finally, while we each should be motivated to press our government to coordinate, to build functional capacity, and to work toward standards, this is not incompatible with making use of AI for good.
Do you appreciate this article? The best way to help the publication is to like and share the article, as we’re still growing our audience.
You should also consider subscribing to get an easy to read email copy of new articles.
Sources:
Commerce Department Proposes Know Your Customer Requirements; Akin Gump; January 29th, 2024
An update on independent evaluators; Anthropic; October 15th, 2026
Designing a FINRA for Frontier AI; Mark Thomas; Lawfare; July 30th, 2026
Verbalizable Representations Form a Global Workspace in Language Models; Anthropic; July 2026
Uncensor any LLM with abliteration; Maxime Labonne; June 12th, 2024
The Great Refactor; Herbie Bradley and Girish Sastry; IFP; August 11th, 2025
How Should the US Prepare for Increasingly Automated AI R&D?; IFP; 2026
The AI-as-Normal-Technology view of loss-of-control incidents; Sayash Kapoor and Arvind Narayanan; September 14th, 2026
AI as Normal Technology; Arvind Narayanan and Sayash Kapoor; April 15th, 2025
Bending the Curve of Discovery: AI in Science Today and Tomorrow; Alex Imas, James Manyika; DeepMind Institute; October 8th, 2026
Introducing the Anthropic Cyber Mission; Anthropic; October 8th, 2026
Project Glasswing; Anthropic; April 7th, 2026
Daybreak; OpenAI; May 11th, 2026
Detecting misbehavior in frontier reasoning models; OpenAI; March 10th, 2025
Towards safety cases for frontier AI training; OpenAI; September 28th, 2026
Model backdoors are easy to create; Equistamp; Connor Tabarrok, Daniel Kang; October 9th, 2026
Anthropic’s Responsible Scaling Policy v3; Anthropic; February 24th, 2026
Priorities and principles for effective third party assessments; OpenAI; Lama Ahmad; September 22nd, 2026
Disrupting the first reported AI-orchestrated cyber espionage campaign; Anthropic; November 13th, 2025
How to pace the US frontier; AI Futures Project; August 2026
Should we “pace” AI self-improvement? Tim Fist and Saif Khan of IFP; August 9th, 2026
Related Articles
Top comments (0)