Your AI agent just did something it shouldn't. Here's the first hour.

Picture the message in your team chat: "Why did the support bot email a customer's invoice to an address nobody recognizes?"

If your agent can call tools, this will happen to you in some form. It might be a prompt injection, a model misreading an instruction, or a leaked key. Most incident response plans were written for servers and phishing. They don't say what to do when the thing that misbehaved is a language model holding your credentials.

Here is the order of operations I use. Most of it comes down to one rule: contain first, understand later.

Before anything happens: four things to build now

You can't improvise these during an incident.

1. A kill switch. One action that disables a single tool, or the whole agent, without a deploy. Test it the way you'd test a backup restore.
2. A tool inventory. For every agent: which tools it has, what each one can affect, and which credential (with which scopes) each one uses.
3. Logs you can actually investigate with. The prompt, the IDs of retrieved documents, every tool call with its arguments and result. Redact secrets, and keep the logs long enough to look back.
4. Budgets. Max steps, max spend and rate limits per run, so a loop stops by itself at 3 a.m.

OWASP's entry on Excessive Agency (LLM06:2025) points out that logging and rate limiting don't prevent damage, but they limit how much damage happens before you notice. That's the whole point of preparing.

The first hour

1. Declare it. Open an incident ID and name one lead.
2. Contain before you diagnose. Hit the kill switch or switch the agent to read-only. Pause schedules and triggers.
3. Rotate any credential the agent held if there's any chance it was exposed. Investigating doesn't un-leak a key.
4. Preserve the evidence. Export logs, transcripts and tool-call records. Don't let the agent "clean up".
5. Don't ask the agent to investigate itself. Its context may still contain the malicious instruction.
6. Decide who needs to know (users, legal/privacy, vendors) and write down who made that decision.

What the investigation is looking for

For the most common agent incident, an injected instruction that turned into a tool call, the questions are:

• Where did the instruction come from? A web page, an email, a file, a ticket, a tool result.
• What did the agent do after that content entered its context? List every tool call.
• What left the system, and where did it go? Data, messages, money.
• Which control should have stopped it? Was the tool even needed? Was the credential broader than necessary? Was there an approval step, and did it show the real arguments?

The fixes almost always fall into a few buckets: remove the tool, narrow the scope, add an approval gate, add validation, or add a rate limit. OWASP lists the same family of mitigations: minimize extensions, functionality and permissions, require human approval for high-impact actions, and enforce authorization in downstream systems rather than letting the model decide.

Six incidents worth a written playbook

1. Prompt injection led to an unwanted tool call
2. Sensitive data showed up in agent output or logs
3. A credential the agent held may have been exposed
4. A runaway loop or cost spike
5. A malicious or compromised tool, plugin or MCP server update
6. A wrong high-impact action caused by model error, with no attacker involved

Number 6 is the one teams forget. It isn't a security incident in the classic sense, but the response is the same: contain, reverse what you can, then put validation or a human in front of that action.

Close the loop

Every postmortem should end with at least one control change and one regression test. Take the payload or input that caused the incident and turn it into a test the agent must pass before the next release. For a starting set of injection tests, there's a free, MIT-licensed collection of 20 payloads here: https://github.com/elaysuu/prompt-injection-test-payloads

Want the full runbook?

I packaged this into an AI Agent Incident Response Runbook: a severity scale, the first-hour checklist, six playbooks with Contain / Investigate / Eradicate / Recover steps, CSV trackers and a blameless postmortem template: https://setrov.gumroad.com/l/uxsib

References: NIST SP 800-61 Rev. 3 (April 2025), https://csrc.nist.gov/pubs/sp/800/61/r3/final ; OWASP LLM06:2025 Excessive Agency, https://genai.owasp.org/llmrisk/llm062025-excessive-agency/ ; MCP tools specification, https://modelcontextprotocol.io/specification/latest/server/tools