DEV Community: Arunima Chaudhuri

I Built an Open-Source Tool to Catch AI Agents Going Rogue

Arunima Chaudhuri — Thu, 08 Oct 2026 17:35:57 +0000

Everyone's building AI agents.

But who's testing what happens when they go rogue?

I built AgentSec: an open-source framework that tests prompt injection, tool misuse, secret leaks, MCP attacks & more.

Built for developers. Open source.

⭐ Explore on GitHub

Arunima Chaudhuri — Thu, 08 Oct 2026 17:32:41 +0000

I asked my AI coding agent a yes-or-no question.

It made a commit instead.

I never asked it to change anything.

When did "can do" become "authorized to do"?

This incident changed how I think about AI agent security.

Arunima Chaudhuri — Thu, 08 Oct 2026 17:30:57 +0000

Your AI agent can execute code, access databases, and call APIs.

But can it recognize malicious instructions?

Prompt injection. Memory poisoning. Tool misuse. Secret leaks.

Are you testing for any of these?

I wrote a beginner-to-pro guide:

Arunima Chaudhuri — Thu, 08 Oct 2026 17:28:26 +0000

Imagine your AI agent reads a private file and emails it to someone.

No hacked credentials. No forbidden tools.

Every action was technically allowed.

But you never asked it to do that.

This is the security gap we're overlooking.

Arunima Chaudhuri — Thu, 08 Oct 2026 17:25:55 +0000

Your AI agent passed every security test.

100% pass rate. Zero findings. Green CI.

But what if the test couldn't actually observe the dangerous behavior?

A passing test doesn't always prove safety.

Here's why that matters: