Michael H

Posted on Oct 13 Originally published at agent5.news

OpenAI's Safety Culture Crisis: Why Researchers Keep Leaving

#ai #policy #openai #aisafety

Originally published on agent5.news.

When a senior researcher resigns from one of the most powerful AI companies in the world and publishes an essay titled "I Quit OpenAI Because Its Culture Is Broken," it is easy to treat the story as a workplace drama. It is not. The departures of safety-focused staff from OpenAI, stretching from early 2024 into October 2026, represent a sustained, public disagreement over how fast transformative AI should move, who gets to slow it down, and whether any governance structure can actually enforce that slowdown.

What the Superalignment Team Was and Why It Mattered

OpenAI announced a dedicated team called Superalignment in 2023, led by co-founder and chief scientist Ilya Sutskever and alignment researcher Jan Leike. The team's stated mission was to solve the problem of how humans could remain in control of AI systems that might eventually become smarter than their creators. OpenAI committed to dedicating 20% of the compute it had secured to date toward that goal and gave the team a four-year runway to deliver results.

The promise was significant because it named the hardest problem in the field. Current AI systems are trained using human feedback, where supervisors reward behavior that looks correct. That process works reasonably well at today's capability levels, but researchers warn it could fail for systems capable of finding ways to manipulate their supervisors or hide problematic behavior until after deployment. The Superalignment team was specifically charged with developing techniques to prevent that failure mode before it became relevant.

Within a year, the team was gone.

The 2024 Wave: Resignations, Firings, and a Team Dissolved

The departures began before the Superalignment team's formal dissolution. William Saunders, who worked on the alignment research that later became Superalignment, resigned in early 2024, concerned that OpenAI was not adequately preparing for the advanced AI systems it was actively trying to build. Daniel Kokotajlo left that spring, declining to sign a standard exit agreement, saying he did not want to be silenced about his concerns over how close the industry was to transformative AI without adequate guardrails.

In April 2024, Leopold Aschenbrenner and Pavel Izmailov were fired over an alleged information leak. Aschenbrenner said the document in question was a benign brainstorming document shared with three external researchers for feedback. OpenAI said the firing was unrelated to the content of the memo; Aschenbrenner said it was explicitly cited as a major reason.

Then in May 2024, both of Superalignment's leaders left within days of each other. Sutskever, who had been one of the four board members who voted to remove CEO Sam Altman in the brief November 2023 attempt, out of a board of six total members, announced his departure without specifying his reasons. Leike was far more direct. In a post on X, he wrote that safety culture and processes had "taken a backseat to shiny products" and that he had been disagreeing with OpenAI's leadership about core priorities until the tension "reached a breaking point."

OpenAI confirmed shortly afterward that the Superalignment team itself would be dissolved, with its members redistributed across the company's broader research efforts. The company said the integration would better serve its safety goals. Researchers who had been on the team did not appear to share that view.

In the months that followed, Rosie Campbell, who led the Policy Frontiers team, also departed, citing the dissolution of what she called the AGI Readiness team and the loss of colleagues like Miles Brundage. Brundage, a long-time policy researcher, left in late 2024, stating publicly that no lab, including OpenAI, was fully ready for what he described as AGI-level systems.

What the Departing Researchers Actually Said

The most striking feature of this wave of departures is how consistent the complaint has been across very different people, roles, and time periods.

Leike's public statement centered on the balance between product development and safety work. Saunders later testified before the U.S. Senate Judiciary Subcommittee on Privacy, Technology, and the Law in September 2024, telling lawmakers that the Superalignment team had been tasked with developing alignment approaches but had struggled to do so, and that this was a serious concern given the potential for catastrophic harm.

Kokotajlo's concern was more existential: that the industry was advancing toward transformative AI without sufficient safeguards in place to understand what it was building. Campbell framed her departure in terms of where she believed she could be most effective, concluding that external work on ensuring safe and beneficial AI was more impactful than staying inside.

David Robinson, who led the writing of the safety reports published alongside OpenAI's major model launches, resigned in October 2026 and published an essay arguing that the company's culture was broken. Robinson's specific criticism targeted what he called a trial-and-error approach to deployment, drawing parallels to industries like nuclear power and aviation, where the consequences of iterating through failures are unacceptable. He argued that the measures used to evaluate how well AI systems match human values remain too coarse, and that the industry's pace of advancement without solving alignment poses escalating risks.

The October 2026 Firings and the Dispute That Followed

In late September 2026, OpenAI fired three safety and alignment researchers: Jasmine Wang, Tomek Korbak, and Mikita Balesni. The company said the firings were for violating its policies on accessing and handling sensitive company information, and later described the situation as "a significant breach of trust."

The three researchers disputed that account in an open letter published on October 8, 2026, addressed to OpenAI's safety and oversight bodies. Each offered their own account of what they were told.

Korbak said he was told verbally that he was being fired because of the way he communicated with METR, an independent nonprofit organization that evaluates frontier AI models. Korbak had been OpenAI's main technical point of contact with METR during an investigation. He said that talking to METR was his job. Balesni said he was told he had been speaking too much to third-party safety organizations, and that he understood OpenAI was implying he had leaked intellectual property, which he denied. Wang said she was told she was fired for accessing an executive's email, access she said had been delegated to her for recruiting purposes; she said she told the executive and IT team within minutes of accidentally opening a sensitive message.

OpenAI said the firings were not about safety concerns or speaking out, and that it could not do its work "without a high degree of trust."

The open letter warned of a chilling effect within the company. Balesni noted that former colleagues had told him they were afraid to speak and worried that their personal devices would be searched for messages to external parties. Wang wrote on X that those fired were "not the first to be pushed out of OpenAI under suspicious circumstances."

The Preparedness Framework: Governance on Paper

OpenAI's formal answer to safety concerns is its Preparedness Framework, which OpenAI's published document describes as first issued in December 2023 and updated in April 2025. The framework describes the company's approach to tracking frontier capabilities that create new risks of severe harm, categorizing risk levels and specifying which types of capabilities, including those relevant to biological and chemical weapons, cybersecurity, and AI self-improvement, trigger additional scrutiny before deployment.

Three actors govern the framework: OpenAI Leadership (defined as the CEO or someone they designate), the Safety Advisory Group, an internal cross-functional body that makes expert recommendations on safeguards, and the Safety and Security Committee of the board of directors, which provides oversight.

Critics of the framework point to structural features that limit its binding force. Independent researchers who analyzed the April 2025 version concluded that the framework does not give the Safety Advisory Group power to delay or block a deployment decision, meaning it can recommend but cannot enforce. They also noted that the CEO co-leads the Safety and Security Committee that is meant to provide oversight of CEO decisions, creating a structural conflict. One analysis concluded that the framework requests certain safety measures but demands none of them.

OpenAI has disputed characterizations of the framework as toothless, and has cited commitments to expand third-party evaluation, improve real-time monitoring, and pause training when necessary.

The Pattern Beneath the Individual Cases

Taken individually, each departure can be explained away. An exit agreement dispute. A leak investigation. A researcher who wanted to work elsewhere. Irreconcilable differences with leadership over priorities.

Taken together, they trace a consistent structural tension. Frontier labs operate in an environment where being second to market with a capability can have enormous commercial consequences. The same capability that a safety researcher views as requiring additional testing is, from a product perspective, a reason to accelerate. Safety teams are by definition the ones asking to slow down, and in a fast-moving commercial environment, that position is structurally weak.

The researchers who left are not a random sample. Several held senior roles writing safety reports, running technical evaluations, or managing external relationships with auditors. Their open letter urged the board to stop building models whose reasoning humans cannot audit. The specific concern, that OpenAI risked losing the ability to monitor what AI agents are actually reasoning through, is precisely the kind of problem the Superalignment team was formed to address.

What OpenAI Has Said in Response

OpenAI has consistently pushed back on the framing that safety is being deprioritized. In response to Robinson's October 2026 essay, a company spokesperson described ongoing improvements: pausing training or holding back models when needed, expanding third-party evaluations, strengthening security in research and testing environments, and improving real-time monitoring to detect concerning behavior earlier in the training process.

The Safety and Security Committee was established in May 2024 following the Superalignment team's dissolution. OpenAI has also pointed to its Preparedness Framework as evidence of systematic risk governance, and has committed to external assessments of its models.

The tension in OpenAI's position is that many of the people who departed were the ones responsible for implementing those safeguards internally. Questions have been raised about whether the structural commitments described in published frameworks are being matched by the internal teams needed to carry them out.

What to Watch Next

Several concrete developments are worth following in the months ahead.

The open letter published by Wang, Korbak, and Balesni was addressed to OpenAI's safety and oversight bodies, including its board. Whether the board responds publicly, commissions an independent review, or makes structural changes to the Safety and Security Committee's independence will be an early indicator of how seriously the letter's concerns are treated internally.

METR's ongoing role as an external evaluator is a related thread. The organization's relationship with OpenAI became central to the October 2026 firing dispute, and how OpenAI structures its access agreements with external auditors going forward will matter for any claim that third-party evaluation is genuinely independent.

Legislative pressure is also building. William Saunders testified before the Senate in September 2024, and the firings of Wang, Korbak, and Balesni drew attention from members of Congress. Whether that attention translates into specific legislation on safety disclosures, whistleblower protections for AI employees, or mandated third-party audits remains to be seen, but the political environment has shifted noticeably since 2024.

Finally, the talent signal itself is worth watching. The AI safety research community is small relative to demand, and repeated high-profile resignations published in prominent venues can shift where early-career researchers choose to apply. If the pattern continues, the long-term consequence may not only be reputational but operational: a frontier lab without the internal expertise to evaluate what it is building.

DEV Community