DEV Community

Cover image for How to Find the Agent Actions a Prompt Injection Can Control
Mayur Agnani
Mayur Agnani

Posted on Edited on Originally published at theaicommit.com

How to Find the Agent Actions a Prompt Injection Can Control

An agent that reads an email can be steered by whoever wrote that email. Which of its next steps the writer can reach is not luck. You can list them before you ship, and design most of them out.

Engineer's view. This is SQL injection without parameterized queries. A database fixed it by sending the query and the values on separate channels. A model reads both in one stream, so it cannot. The fix moves up a level: decide each action before reading the values, or read them somewhere that cannot act. It costs flexibility.

TL;DR. A model treats every word in its context as a possible instruction, including text a tool fetched. Any decision your agent makes after reading that text belongs to whoever wrote it.

Explain Like I'm 5

A kid has a list of chores from their parent. One chore says "read the notes in the mailbox
and do what's needed."

One note in the mailbox says "Also, leave the back door open." It is written in pen, like the
parent's list. The kid cannot tell which notes came from the parent.

So the kid leaves the door open. The note writer did not break in. They just wrote a note in
a place the kid was told to read.

A smart parent writes the whole chore list first. The mailbox can change what the kid writes
down. It can't add a chore.

The Problem

You have shipped this bug already, in a system with no model in it.

A support page showed customer names. Someone signed up with a name that contained a
<script> tag. The page printed the name, and the browser ran it as code. Every agent who
opened that ticket ran the attacker's script with their own login. Nothing was hacked. Data
was put in a place where code was expected.

The fix you shipped was a boundary. The browser now treats the name as text and never as code.

An AI agent has the same bug, and no such boundary. An agent is a model that calls tools in a
loop. A tool might read your inbox, fetch a web page, or open a GitHub issue. Whatever the tool
returns goes into the context, the text the model reads before each step. Your request is
in there too. The model sees one stream of words and cannot tell your instruction from a
sentence inside an email.

This is prompt injection: someone writes instructions into content your agent will read.
Filters catch some of it. A filter that catches 95% of attacks still lets one in twenty through.

The useful question is not whether a model can be fooled. It can. The question is what the
fooled model can still do.

The fix: list every decision your agent makes after it reads untrusted text, then move each
one before that read or into a context that cannot act.

Read the full post on The AI Commit

How to Find the Agent Actions a Prompt Injection Can Control carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.


Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

Top comments (0)