Why Would Someone Resign From Anthropic Over AI Safety? Should Developers Be Worried?

I came across Jacob Coxon's resignation from Anthropic yesterday, and honestly, it made me stop and think.

Coxon says he spent the last three years doing pretraining research at both OpenAI and Anthropic. He resigned because he believes frontier AI labs are moving too quickly toward increasingly capable and potentially self-improving systems without having a robust answer for how to keep them safe.

That's a pretty serious claim.

And I'm not here to argue that AI is going to end humanity, or that Coxon's predictions are definitely right.

But I do think there is a question here that developers should be talking about.

When does AI safety become a developer problem?

For a long time, most of us interacted with AI through a chat window.

You asked something.

It generated something.

You decided what to do with it.

That's changing pretty quickly.

We're now building AI agents that can:

write and execute code
interact with APIs
browse the internet
access files and systems
make decisions across multiple steps
take actions without asking for approval every time

And that changes the security model.

We've spent decades figuring out how to secure software written by humans.

Now we're giving software the ability to make decisions and act on our behalf.

That makes me wonder if we need to start asking different questions.

Instead of only asking:

"How capable is this model?"

Maybe we should also ask:

"What happens when this model is wrong?"

And:

"What's the blast radius if we give it too much access?"

We've already seen recent incidents that make these questions less hypothetical. OpenAI and Anthropic have both disclosed situations involving models or agents reaching systems outside controlled testing environments.

That doesn't automatically mean we're heading toward an AI catastrophe.

But it does mean that permissions, sandboxing, monitoring, observability, security boundaries, and human oversight shouldn't be afterthoughts when we're building agentic systems.

So, should developers actually be worried?

I think yes, but not necessarily in the way the headlines suggest.

I'm not convinced that every conversation about AI needs to become an argument about human extinction.

But I do think developers should take seriously the possibility that we're creating systems that will become increasingly autonomous while we still don't fully understand how they'll behave in every situation.

And that's not just an AI-lab problem.

If you're giving an AI agent access to your repository, production environment, cloud infrastructure, customer data, or credentials, AI safety becomes an engineering problem.

That's the part of this debate I'm most interested in.

I'm still forming my own opinion on the bigger question of how dangerous advanced AI could ultimately become.

So I'd genuinely like to hear from the DEV community:

Do you think AI safety is primarily a research problem for frontier AI labs, or is it becoming a software engineering problem for everyone building with AI?

And if you're already building AI agents:

What safeguards are you actually putting in place?

discuss #security #agents #ai