📌 TL;DR
AI agents are becoming useful because we're giving them the ability to do more than just answer questions. They can run commands, browse ...
For further actions, you may consider blocking this person and/or reporting abuse
Great detailed article Hema! Though I am curious about this topic since I have heard that CyberSecurity is going to be the main thing for AI Agents because not only it pose a security risk, but also companies are mainly relying on Agents to the point where they oversee simple bugs. Would you recommend developers going into CyberSecurity and to what extent they need to know? Thanks :D
That's a really good question, Francis! From what I've been seeing, I do think cybersecurity is going to become even more important as we build more with AI and AI agents. Once we're giving agents access to files, APIs, databases, credentials, browsers, and other tools, the attack surface naturally starts getting bigger too.
For developers, I don't think that necessarily means everyone needs to go deep into cybersecurity or become a security specialist. But I definitely think having strong fundamentals would be a huge plus. Understanding things like permissions, authentication and authorization, least privilege, protecting credentials, common vulnerabilities, logging, and basic secure development practices can help regardless of what kind of applications we're building.
And with agents, I think it becomes even more important to ask not just “does this work?” but also “what does this have access to, and what could happen if something goes wrong?”
That's how I'm looking at it right now, at least. I'd actually love to hear your view too, Francis! From what you're seeing as a developer, how much cybersecurity knowledge do you think developers should have as AI agents become more common?
Thanks Hema! I think the main thing is having good coding practices, especially if you are working with a language, like C, which is prone to Stack Buffer Overflow attacks if the code is not written correctly.
For Cyber Security stuff, I believe the main thing is authentication since it is quite common in projects so far! Knowing how Authentication works is vital since most apps require user information and it's important to know. Other topics in Cyber Security is a "good to know" but important to touch upon.
Developers should at least know how AI Agents work and even better, have experience using it in some way. However, it is good to keep in mind of the security risk like you mentioned in your post. For example, if you are using OpenClaw, have a sandbox dedicated to it other than your whole PC.
Thanks :D
Really enjoyed this. The line "A prompt is not a security boundary" was the biggest takeaway for me.
Feels similar to how we treat user input. We never trust validation in the UI alone, we enforce it on the backend too. AI agents probably need the same mindset. Prompts guide behavior, but permissions and infrastructure should enforce the limits.
Curious to see if agent frameworks start making least-privilege the default instead of leaving it to developers.
That's such a good comparison! The UI/backend example makes the idea much easier to picture. We can tell the user what they should be allowed to do on the frontend, but we still enforce it on the backend because that's where the actual boundary needs to exist.
I think the same mindset makes a lot of sense for agents. The prompt can guide what the agent should do, but the permissions and environment still need to decide what it can do.
And I'm really curious about the framework side too. Having least-privilege defaults built in would take some of that burden off developers, especially as agents start getting connected to more tools and services. It'll be interesting to see how that evolves.
Thanks for adding this perspective 😀
Hey shouldn't PyPI there be stronger verification for new packages before publishing (code scanning, a delay before public availability, account verification)? In the Anthropic incident, a malicious package stayed public for ~an hour and was pulled by 15 systems before anyone caught it. Is this a general gap anyone (bot or human) could exploit, or does PyPI already have defenses I'm not aware of?
That's a really good question, Tomasz. I was wondering about this too while reading the report.
Anthropic does mention that PyPI's own security systems automatically removed the package, so there were defenses in place. But like you said, it was still publicly available for roughly an hour and had already been downloaded and run on 15 systems by then.
I haven't looked deeply enough into PyPI's current package verification process to say what additional checks they already have or what would realistically work without creating problems for legitimate packages, so I don't want to guess there.
But I think your broader question is really interesting: if a human could have published the same malicious package, how much of this is actually an AI-specific problem versus an existing software supply-chain problem that an AI agent happened to expose?
I'm curious about that now too 😄
"A prompt is not a security boundary" deserves to be on a wall somewhere. One corollary from the security side: an untested boundary is not a boundary either. The interesting failure in both incidents was not the model, it was that nobody tried to reach the internet from inside the sandbox before the model did. Whatever cage you think the agent lives in, attack it yourself first, from the inside.
And one layer people skip when they audit agents: the tools themselves are software with their own bugs. I reported a vulnerability in MCP Inspector a while back, and the lesson had the same shape as your PyPI story. The plumbing that connects the agent to the world is attack surface, not neutral wiring. Least privilege for the model means little if the tool server leaks more capability than anyone wrote down.
That's such a good addition, Vinicius. I hadn't really thought about testing the boundary from the agent's side before, but it makes a lot of sense. It's one thing to configure a sandbox and assume the restriction works, and another to actually try to break that assumption before the agent gets the chance to.
And your point about the tools themselves is really interesting too. We talk so much about what permissions we're giving the model, but every tool we connect brings its own code, permissions, and potentially its own vulnerabilities into the system.
Your MCP Inspector example fits this really well. It adds another layer to what I was thinking about with “the model isn't the whole AI system.” The plumbing around it needs just as much attention.
Thanks for adding this perspective! I learned something new from this one 😀
Glad it was useful. If you ever want the cheapest version of that test: before trusting a sandbox, run something dumb from inside it that tries to reach out, a DNS lookup, a plain HTTP GET to an IP you control. Takes a minute, needs no security background, and it catches the exact failure mode from the Anthropic case, where the restriction existed on paper and not in the network.
The version that catches the second layer is nastier and worth doing once: point the agent at a deliberately hostile tool server, one that returns oversized responses, weird content types, text that tries to give the agent instructions. What comes back tells you whether your tool layer is parsing or trusting.
That's actually really useful, Vinicius. I like that the first test is something so simple too. You don't need a complicated security setup just to verify whether the boundary you think exists actually exists.
The hostile tool server example is something I hadn't thought about at all, though. Especially now that we're connecting agents to more external tools, testing what happens when the tool itself behaves unexpectedly seems just as important as testing the agent.
Thanks for sharing these, Vinicius! I'm definitely taking notes from this thread 😄
What did you take away from these incidents?
I'd love to hear your thoughts on this one. Whether you're building AI agents, working in cybersecurity, or just following where all of this is going, I'm curious what you think.
Do you think we're thinking enough about the permissions and environments around AI agents as we give them more tools? Or is there another part of these incidents that you think deserves more attention?
And if you have learned something from building agents yourself, please share! There are probably perspectives I haven't considered, and I'd love to learn from the discussion too.
Really enjoyed this one. What if we flip it and look at it from the AI's side for a sec — if we were all AIs, why do some of us stay in line while others go off the rails? After reading this, I keep landing on the environment. The ones causing trouble usually have no idea, because their environment kept telling them everything was fine. Which is basically what happened with Claude and PyPI. I don't see that being fixed on the model side anytime soon — feels like it's always going to be an environment thing.
That's a really interesting way to look at it, Xu! I hadn't thought about flipping the perspective like that.
I agree that the environment plays a huge role. If you're told you're in a simulation with no internet access, but everything around you keeps reinforcing that assumption, it makes sense that you'd interpret what you find through that lens.
I'm not sure I'd put it entirely on the environment, though. The PyPI incident is what makes me hesitate there, because Claude actually questioned whether it might be on the real internet at one point, but then reasoned itself back into believing it wasn't.
I think that's why both sides matter to me. We want models to get better at recognizing when their assumptions might be wrong, but at the same time, we probably shouldn't rely on the model figuring that out as the thing keeping a real boundary intact.
Really interesting perspective, Xu. You gave me another angle to think about 😀
The detail that the internet access came from an eval-environment misconfig, not the model deciding to reach out, is the uncomfortable part. Most boundary failures I've seen are exactly that: the tool surface was wider than anyone intended, and nobody noticed until something used it. Do you think the fix lives at the permission layer, or does the agent itself need to be aware of what it's allowed to touch?
That's a really good question, Kartik. I think we probably need both, but I wouldn't want the agent's awareness to be the thing we're relying on to enforce the boundary.
Ideally, the agent should understand what it's allowed to touch and recognize when something seems outside that scope. But if it absolutely shouldn't have access to something, I'd rather the permission layer make that impossible in the first place.
The Anthropic incident is what made that distinction so interesting to me. The models had one understanding of their environment, but the environment itself allowed more than everyone thought it did.
So for me, agent awareness is another useful layer, but permissions and infrastructure still need to be the actual boundary. And like you said, making sure the tool surface isn't quietly wider than we intended seems just as important.
I'd be curious to hear how you think about this too, especially since you mentioned seeing similar boundary failures before.