Key Takeaways
- Governor Gavin Newsom’s Executive Order N-9-26 requires state agencies to recommend emergency shutoff mechanisms for frontier AI models by November 16, 2026, filling a gap left by federal inaction.
- The order tasks state bodies with developing recommendations for embedding independent verification organisations inside AI labs, a response to concerns about developer self-attestation under California’s existing SB 53 framework.
- California became the first state to certify independent AI verification organisations and establish an auditor registry through SB 813 and AB 1405, signed in early September 2026. Newsom’s Executive Order N-9-26, signed September 18, 2026, asks state agencies to determine whether California law should require frontier AI developers to build independently testable emergency shutoff mechanisms into their models, a requirement that has no equivalent anywhere in US law. The Government Operations Agency and the Governor’s Office of Emergency Services have until November 16 to report back, on top of a legislative stack that already makes California the most active AI regulator among US states.
Four Areas, One Deadline
The order tasks two state bodies with developing recommendations across four areas. The first is embedding independent verification organisations directly inside frontier AI laboratories to conduct periodic audits, rather than relying on external reports filed by developers themselves. The second is requiring that the safety frameworks, transparency reports and risk assessments already mandated under state law be independently verified against standards set by a certified body, a direct response to concerns about self-attestation.
The third area is drawing the most attention: requiring frontier AI developers to build an independently testable emergency shutoff into their models. The fourth expands the definition of “critical safety incidents” that companies must report to the state, adding a broader category of “loss-of-control” events. Together, the four areas represent a significant escalation of the hands-on oversight California expects from AI developers operating in the state.
The Legislative Stack Behind the Order
The September 18 order arrived with infrastructure already in place. Newsom’s earlier Executive Order N-12-23, signed September 6, 2023, directed a state-wide assessment of generative AI risks to critical infrastructure and set procurement guidelines for state agencies.
The Transparency in Frontier Artificial Intelligence Act (SB 53), effective January 1, 2026, required frontier AI developers to publish safety frameworks, conduct catastrophic risk assessments and report specified critical safety incidents to the state. It also included whistleblower protections. Then, in early September 2026, Newsom signed SB 813 and AB 1405: the former established a certification framework for independent verification organisations, making California the first state to create such a system; the latter created a state registry for AI auditors, with standards governing their independence and integrity. Executive Order N-9-26 asks what additional mechanisms should be written into law on top of that base.
The Kill Switch Problem
A reliable, independently verifiable shutoff for a distributed frontier AI model would require control mechanisms sitting outside the model’s own architecture, hardware-level interventions or network isolation protocols that remain functional even if the system is behaving unexpectedly. Defining which models qualify is its own challenge: thresholds based on compute or parameter count do not cleanly map to risk, and dangerous emergent behaviours can appear in ways that evade those proxies.
Engineers have described what they call the graceful shutdown problem. Cutting power or connectivity to a complex AI system abruptly risks data corruption, residual processes and new vulnerabilities rather than a clean halt. A meaningful kill switch would need to bring the system to a safe, inert state while preserving diagnostic data, a level of internal control that is difficult to achieve in models whose internal states are opaque even to their developers. The November 16 recommendations will need to address whether a single universal mechanism is feasible, or whether a spectrum of interventions is more realistic. The logistics of independent organisations testing these mechanisms on an ongoing basis, with access to proprietary model internals, adds a further layer of complexity for both developers and regulators.
Where Industry Stands
Anthropic and OpenAI have both publicly acknowledged the need for safety measures and agreed to some voluntary White House commitments, including information sharing and pre-deployment testing. Anthropic CEO Dario Amodei has publicly called for the industry to pace development more cautiously on safety grounds, though he has also indicated that kill switches alone are not a sufficient answer to the risks posed by advanced models.
Industry representatives have raised concerns that on-site embedding of independent auditors creates tension with intellectual property protections for companies with sensitive model architectures, and that continuous external review can slow iterative development cycles. How “loss-of-control” incidents are defined will also matter operationally: a broad definition generates reporting volume that can obscure genuinely serious events, while a narrow one misses emerging risk patterns. The November 16 recommendations are where those definitional choices get made, or at least proposed.
The Federal Gap
Newsom has been direct about his motivation. In statements accompanying the order, he described the federal government’s approach as an “abject failure” to create meaningful AI oversight. President Trump has dismissed AI safety concerns and resisted regulatory frameworks, prioritising US competitive positioning over near-term oversight obligations. With Congress unlikely to act on AI governance before the 2026 midterm elections, California has moved to occupy the space.
This is a familiar dynamic in US regulatory history. California has used its economic scale to effectively set national standards in areas from vehicle emissions to consumer privacy, the California Consumer Privacy Act produced compliance changes that extended well beyond state lines. Whether the same dynamic plays out in AI governance depends on how many frontier developers treat California compliance as a floor for their global operations, a question the industry has not yet answered uniformly.
What Other Regulators Are Watching
California hosts a disproportionate share of the companies building frontier AI systems, and the state’s regulatory choices carry weight outside its borders. The kill switch proposal adds a dimension that other jurisdictions have not yet formalised.
Given California’s track record of converting recommendations into statute, and the absence of a federal framework, what the November 16 report says about kill switches and on-site auditors will be read carefully by developers, other state legislatures and international regulators. The EU, the UK’s AI Safety Institute and others have each expressed interest in interoperability between oversight regimes. A California standard that diverges sharply from those frameworks could fragment compliance obligations for companies operating across jurisdictions.
Originally published at https://autonainews.com/california-governor-seeks-ai-kill-switch-recommendations-by-november-16/
Top comments (0)