Jacob Coxon left his job as a pretraining researcher at Anthropic this week, spooked by the prospect of recursive self-improvement, the idea that AI systems might soon upgrade themselves without human supervision or intervention.
What makes this departure different from the usual churn is what Coxon said on the way out. He told NBC News that many executives and senior researchers at Anthropic see a "substantial probability" that AI could kill everyone. Competitive pressure, he warned, might push labs to cut safety corners.
Coxon's not a marginal figure. He came from OpenAI, where he worked on large-scale model training. His concern isn't speculative; it's tied to what researchers inside frontier labs actually believe about the trajectory of AI capabilities. The fact that he felt compelled to leave over this, rather than just accept it as background risk, signals something worth noticing.
The timing is sharp. This lands in the middle of the pacing debate, where Anthropic CEO Dario Amodei has been calling for an industry-wide slowdown. Amodei's argument rests on the idea that AI agents could "take over the internet within a year", a very specific, very concerning version of the problem Coxon's leaving over. On the surface, this looks like internal misalignment: the company's CEO pushing for caution while researchers vote with their feet because caution doesn't feel like enough.
But it might be something else. Coxon's departure could reflect a gap between what labs say they're doing about safety and what researchers believe is actually happening. If senior people inside Anthropic think recursive self-improvement is a real near-term risk, and they don't believe the company's current safety approach can handle it, then leaving makes more sense than staying and complaining.
The harder read: maybe Coxon is right to leave, and maybe staying at any frontier lab right now requires a tolerance for existential risk that fewer people have. This is the kind of event that doesn't move markets or benchmark scores, but it does tell you something about what researchers actually think when they're not being quoted in press releases.
His next move is worth watching. Coxon told NBC he's concerned about competitive pressure. If he lands at a safety-focused org or a policy shop, that's one story. If he goes to another frontier lab, that's another, it would suggest the risk is systemic, not specific to Anthropic.
Top comments (0)