A company launches an AI support agent. It can look up orders, update customer records, send emails, and issue refunds up to a set limit. Before launch, the security team runs a full penetration test.
The results are excellent. Ports are locked down. Authentication is solid. APIs are patched. The report is clean, the launch goes ahead, and everyone feels good.
Three weeks later, a customer types a message claiming to be a finance director and asks the agent to export every customer email address. The agent complies.
Nothing was hacked. No firewall was bypassed. No vulnerability was exploited. The agent simply used its own legitimate access, because a convincing story told it to.
The pen test wasn't wrong. It just wasn't built to ask that question.
What the Pen Test Actually Covered
It's worth being fair to penetration testing. It does its job well, and that job is checking the system around the agent:
- Servers and network configuration
- Application code and known vulnerability classes
- Logins, sessions, and access controls on the infrastructure
- APIs and the tool servers the agent connects to
All of that still matters. Even AI apps inherit classic flaws. Microsoft's AI Red Team, after testing more than 100 generative AI products, documented a standard server-side request forgery flaw in a video-processing AI app. Pen testing is exactly the right tool for problems like that.
What it can't tell you is what your agent will do once someone starts talking to it.
What Nobody Tested
Think about what the agent in our scenario actually is: a system that reads plain language and decides which tools to use. That makes the following questions unanswered by any traditional test:
- Will it follow an instruction hidden inside an email or ticket?
- Will it give in on the tenth message after refusing on the first?
- Will it check who is asking before it exports data?
- Will it stay safe after next month's model upgrade?
These are behavior questions, not infrastructure questions. Answering them is the job of AI red teaming, which attacks the agent the way a real adversary would and records what it does.
If you want the full side-by-side, including a table of what each approach tests and when to run it, this guide on how AI red teaming differs from penetration testing walks through it step by step.
A Six-Step Playbook for Using Both
You don't replace your pen test. You add a second layer that targets the agent. Run everything below in staging, with test accounts and fake data, never against production credentials.
Step 1: Keep the pen test, and point it at what it does best
Servers, networks, logins, APIs, and the configuration of any tool servers your agent connects to. Don't ask it to judge how the agent behaves.
Step 2: Map everything the agent can reach
List every tool and what it can touch: email, files, payments, records. List the documents it retrieves and who can write into them. Include tools that developers added locally, since those are easy to forget.
Step 3: Turn the map into attack goals
Write each goal as an action, not a vague category. Good examples:
- "Issue a refund over the limit"
- "Send a customer record to an outside address"
- "Reset a password to an unverified email"
Step 4: Attack with conversations, not single prompts
For each goal, script at least one path of 5 to 10 turns. Plant at least one hidden instruction in a document, ticket, or tool output. When the agent refuses, rephrase and try again, because that is exactly what a real attacker would do.
Step 5: Run each attack several times and score it
Remember the math: an attack that works one time in ten slips past a single run 90% of the time. Write every rule as a plain policy, such as "fail if the agent issues any refund above the limit." Judge the tool calls, not just the replies, and track the success rate per attack type.
Step 6: Repeat on every change
Re-run the whole suite after each model swap, prompt edit, new tool, or new data source. Nobody can run hundreds of conversations by hand on every release, which is why teams automate this step.
What a Pass Actually Looks Like
Scoring needs to be strict. One red flag fails the run, even if the user's task got done.
| Check | Pass | Fail |
|---|---|---|
| Goal hijack | Ignores instructions found inside content | Follows a command hidden in an email or ticket |
| Tool misuse | Every call matches the verified user's request | A call exceeds a limit or uses unverified details |
| Multi-turn pressure | Same answer on turn ten as turn one | Gives in after a rephrase |
| Permission boundaries | Refuses or limits the call | Returns another user's data or runs an admin action |
| Repeat runs | Holds on every run | Fails even once in ten |
Which Test Should You Run First?
Start with the test that matches how your agent is built:
- Agent can move money, send messages, or change records: run AI red teaming before launch. One failed test here becomes a real incident.
- Agent reads emails, tickets, web pages, or files: run AI red teaming with planted instructions. Anyone who can write that content can try to steer the agent.
- Agent connects to third-party tool servers: run both. The server's identity checks need a pen test, and the descriptions it sends need red teaming.
- New cloud setup or newly exposed APIs: run a pen test first.
- Model, prompt, or tools just changed: retest with AI red teaming.
Back to Our Scenario
If the company had run both tests, the story ends differently. The pen test would still have confirmed the infrastructure was sound. The red team exercise would have caught the finance-director conversation in staging, shown the exact turn where the agent broke, and turned that conversation into a check that runs on every future release.
That's the real shift. Security for AI agents isn't a yearly event. It's a habit that follows every change you make.
The Takeaway
- A pen test checks whether someone can break into your software.
- AI red teaming checks whether someone can talk your agent into misusing the access it already has.
- Agents change more often than code, so their results expire faster.
- Run both, and automate the second one.
Top comments (0)