Something went wrong with your AI system. Maybe it disclosed data it shouldn't have. Maybe an automated agent took an action nobody intended. Maybe a user found a way to manipulate it in ways you didn't anticipate. Whatever the specifics, the next few hours matter more than most teams expect.
Most organizations reach for their standard IT incident response procedure—and quickly discover it doesn't fit. AI systems fail differently than traditional software. A conventional bug reproduces reliably; an AI output is non-deterministic. Running the same prompt twice often yields different results, which makes verification awkward and confirms nothing. This isn't a theoretical concern. It shapes everything from how you collect evidence to how you write your post-incident review.
The First Hour: Triage With a Clear Head
Designate one person as incident lead before anything else. Multi-person decision-making without a single owner produces delays and conflicting actions at the worst possible moment. The lead's first job is documentation: when was this discovered, by whom, and through what mechanism. That timestamp is evidence.
Next, classify what actually happened. AI incidents generally fall into one of three categories: data exposure (the system shared something it shouldn't have), real-world action (an agent took a step with external consequences—sent an email, made an API call, modified a record), or output inaccuracy (the model produced something wrong in a way that caused harm). The classification drives your containment approach.
Also check whether external content entered the model's context—uploaded documents, scraped websites, email threads. If so, prompt injection is a possibility, and the scope of the incident may be wider than the initial report suggests.
Containing the Damage
The guiding principle is to use the smallest intervention that actually stops the problem. Revoking API credentials is often cleaner than disabling at the application layer, because application-level toggles can leave cached responses and queued jobs still executing. If an API key was exposed, that credential needs to be invalidated everywhere it's stored—not just in the affected service.
Before you consider anything contained, check for pending scheduled actions. An AI agent that was mid-workflow may have queued downstream steps that will execute regardless of whether you've switched off the front-end interface.
Preserving Evidence Before It Disappears
This is where many smaller teams make a costly mistake: they clean up or restart services before they've documented anything. AI systems generate evidence that vendors retain for limited and often short time windows, and those windows close fast.
Capture: full conversation transcripts with timestamps, session and trace IDs, the exact model version in use, the active system prompt at the time of the incident, any documents retrieved by the model, every tool call with its inputs and outputs, and the identity of the acting account. Once collected, disable automatic log deletion, instruct anyone involved not to remove related messages or communications, export everything you can access directly, and send a formal written log preservation request to your vendor.
Legal hold language matters here. A casual support ticket asking them to "keep the logs" is not the same as a formal written request tied to a potential legal proceeding—and the difference can be significant if this ends up in dispute.
Deciding Who to Notify and in What Order
Notification sequencing is as important as the content of what you say. Brief leadership before customers. Loop in your insurer before making any public statement. Business customers generally need to hear from you before affected individuals, and both before regulators.
Before you decide that disclosure isn't required, talk to legal counsel. The notification threshold under most data protection frameworks is lower than operators typically assume. Keeping communications factual and scoped to confirmed information is essential—speculation during an active investigation creates liability and erodes exactly the trust you're trying to protect.
What Comes After: Fixing the Gaps
The post-incident review, ideally completed within two weeks, should focus on systemic improvements rather than assigning individual blame. The most common finding is a logging deficiency: teams discover they captured only final model outputs, not the system prompts, retrieved context, tool calls, and user identifiers that would have answered their questions immediately.
Build the logging infrastructure you wish you'd had. Set explicit retention periods with intention. Reduce permissions to the minimum each component actually requires. Write at least one repeatable test case that would have caught this incident—so future deployments can be checked against it before going live.
AI incidents will happen. The difference between teams that handle them well and teams that don't usually comes down to one thing: whether they thought through the response before they needed it.
This guide originally appeared on agentpalisade.com. Agent Palisade helps small and mid-sized businesses put AI to work inside the tools they already use — practical automation, internal assistants, and AI security reviews. Book a free 30-minute call.
Top comments (0)