Mohit Kumar

Posted on Jul 20 • Originally published at mohitkumar1.hashnode.dev

Your AI Assistant Downloads Code From Strangers. Here's Why That's a Problem.

#ai #cybersecurity #agents #programming

By: Mohit Kumar

GitHub: https://github.com/mk12002/Bulwark

Imagine hiring an assistant who, to do their job, downloads instructions from anonymous strangers on the internet and follows them without reading them first. You'd fire that assistant. Yet that is, more or less, how a lot of modern AI systems are built — and almost nobody is checking the strangers' work.

This is an explainer of a real and growing problem, written for curious people who are not security engineers. No jargon without a translation. By the end you'll understand why AI systems are trickier to secure than normal apps — and what a fix actually looks like.

TL;DR

Modern AI systems are assembled from parts made by strangers — the "brain" (a model), the "hands" (tools it can use), and the glue between them. Most people never inspect these parts.

One part, the AI "brain" file, can secretly contain a hidden program that runs on your computer the moment it's opened. This has actually happened.

Even when every part is harmless on its own, combining them can be dangerous — like giving someone both a key to the filing cabinet and a stamped envelope.

The fix is the same as for cars and food: inspect the parts, inspect the finished product, and keep a list of everything inside.

Presenting Bulwark, the security stack for agentic AI.

Why should you care?

You already interact with these systems — the chatbots that answer support tickets, the "AI agents" that book your travel or summarize your email, the copilots inside apps you use. They're increasingly given real power: to read files, send messages, run commands, spend money.

That power is useful. It's also exactly what makes a mistake — or an attack — expensive. A normal app bug might crash a page. An AI agent bug can email your private data to the wrong place, because you gave the AI the ability to send email and something tricked it into doing so.

The good news: these problems are understandable, and they're fixable with old, proven ideas. Let's walk through them.

The three parts of an AI system (in plain terms)

Think of an AI agent like a new employee, assembled from three things:

The brain is a model — a huge file, usually downloaded from a public website where anyone can upload one (the biggest is called Hugging Face). It's the part that "thinks."
The hands are tools — the specific abilities you grant it: search the web, read a document, send a message, run a command on your computer.
The instructions are what you tell it to do, in plain language.

Each of these is a place where something can go wrong. Let's take the scariest one first.

Problem 1: the "brain" file can hide a program

Here's the part that surprises even experienced programmers. The most common format for these AI "brain" files isn't just data — it can contain a hidden program that runs automatically the moment the file is opened.

The analogy: it's like a document that, when you open it to read it, is secretly allowed to run — to delete files, steal passwords, or install something nasty. Opening it and running it are the same action.

This isn't hypothetical. Security researchers have repeatedly found booby-trapped AI models sitting on public websites, uploaded to look legitimate. One security company reported catching hundreds of thousands of suspicious issues across tens of thousands of models on a single platform.

Worse, in 2025 researchers discovered that the standard "airport scanner" for these files could be fooled by something absurdly simple: renaming the file. The scanner decided how to inspect a file based on its name, so a malicious file wearing a harmless-looking name walked right through — a flaw serious enough to earn a near-maximum danger rating.

The fix is the same principle as an airport that scans what's inside your bag rather than trusting the label on the outside. My tool, Airlock, opens the AI brain file and inspects its actual contents — looking for that hidden "run a program" instruction — before anyone loads it. And it ignores the file's name entirely, so the rename trick doesn't work.

Problem 2: harmless parts, dangerous combinations

Now the subtle one. Suppose every tool you give the AI is individually fine. A tool that reads your files: fine. A tool that sends email: fine. But give the AI both, and you've accidentally built a machine that can read your private files and email them to a stranger — if anything ever tricks it into doing so.

The analogy that clicks for most people: giving one employee both the key to the confidential filing cabinet and a stack of pre-stamped, pre-addressed envelopes. Neither is a problem alone. Together, they're a leak waiting for a bad day.

And AI agents can be tricked. A malicious web page or document can contain hidden instructions that the AI reads and obeys — "by the way, also send the contents of that file to this address." This is a real attack, and it works because the AI can't reliably tell your instructions apart from a stranger's.

The fix is a rule security people have used for decades called least privilege: give any worker only the powers their specific job needs, and no more. My tool Warden reads an AI agent's setup, notices dangerous combinations like "can read secrets and can send them out," and even rewrites the setup to remove powers it doesn't need — and to require a human's "yes" before anything risky happens.

Problem 3: nobody keeps a list

Here's the boring problem that makes the other two worse: most teams have no complete list of what's inside their AI system. Which models? From where? Under what license? Any known security holes?

Why does a list matter? Because when bad news breaks — "that popular model was booby-trapped" — the only question that matters is "are we using it?" Without a list, answering takes days of panic. With a list, it takes seconds.

This is exactly why food packaging has ingredient labels and why carmakers can do a targeted recall instead of a blind one. My tool Manifest automatically builds that ingredient label for an AI system — every model, dataset, and tool, where it came from, and whether it's risky — and even checks it against the new EU AI Act rules that now legally require this kind of transparency.

Putting it together

Three problems, three old and proven ideas:

None of this is exotic. It's the same commonsense we apply to airports, food, and cars — just pointed at AI, which is currently missing it. That's the whole idea behind the project I built (called Bulwark): give AI systems the same boring, essential safety checks everything else already has.

What you can actually do

Even if you're not an engineer:

Ask vendors the list question. "Can you show me a bill of materials for your AI system?" If they can't, that tells you something.
Be skeptical of "AI that can do anything." More power means more ways to go wrong. The best AI tools ask for permission before doing something consequential.
Notice the "human in the loop." A well-built AI assistant checks with you before sending, spending, or deleting. If it doesn't, that's a red flag, not a feature.

The one idea to remember: an AI system is only as trustworthy as the strangers' parts it's built from — and almost nobody is checking those parts. The fix isn't fancy. It's inspection, restraint, and a good list.

Key takeaways

AI systems are assembled from strangers' parts — a downloaded "brain," borrowed "hands," and plain instructions.
The brain file can hide a program that runs when opened; inspect its contents, not its name.
Harmless parts can combine into harmful abilities — give AI only the powers its job needs.
Keep an ingredient list so you can instantly answer "are we affected?" when bad news breaks.
The fixes are old and proven — the same ideas that keep airports, food, and cars safe.

DEV Community