Why an AI Threat Model Is Far More Than a List of Known Risks
With classical applications, the security analysis can often be built along familiar structures: users send input, business logic processes it, databases supply or store information, and an answer results at the end. AI applications look similar at first glance. They too have frontends, backends, databases, identities, and APIs. The decisive difference, however, lies in how information changes its role within the system.
A document starts out as mere content. After retrieval, an excerpt of that document becomes part of the model context. There, it can not only supply knowledge but influence the model's behavior. Model output starts out as merely probabilistically generated text or structured output. If it turns into a tool call, a SQL query, a ticket, or an email, it produces a real effect. It is exactly at these transitions that it's decided whether an AI system merely produces the occasional nonsense, or whether an attacker actually violates protected assets.
A good threat model for an AI application therefore doesn't simply describe "prompt injection," "hallucination," "data leakage," and "tool abuse." That would be little more than risk bingo. A useful threat model is a working model that establishes a traceable chain:
System → Assets → Data Flows → Trust Boundaries → Attacker Goals → Abuse Scenarios → Attack Paths → Controls and Tests
This chain is the actual goal. It turns an abstract discussion about "AI risks" into a concrete security analysis that connects to engineering, architecture, testing, and operations.
Figure 1: The AI threat modeling process moves from understanding the overall system through assets, security objectives, data flows, and trust boundaries to concrete attacker goals, abuse scenarios, and attack paths. Controls and reproducible security tests are then derived from these attack paths. Threat modeling therefore connects architectural understanding directly with testable security assumptions.
1. What a Threat Model Must Deliver
A threat model is neither a decorative diagram nor a collection of general security rules. It should help you understand a system well enough that plausible security problems become visible and verifiable.
To do that, it has to answer at least six questions:
- What is the system? What function does the application serve, who uses it, and which AI components are involved?
- What is worth protecting? Which data, permissions, control mechanisms, identities, and outputs must not be disclosed, manipulated, or abused?
- How do information and decisions move through the system? Which inputs, context blocks, tool results, and model outputs flow between the components?
- Where does trust, control, or effect change? At which points does untrusted content gain more trust, get pulled into a privileged context, or get translated into a real action?
- Who could pursue which goal? What realistic attackers exist, what capabilities do they have, and what effect do they want to achieve?
- What concrete attack paths result from this? Under what preconditions can an actor violate an asset via which components and boundaries?
A threat model has succeeded when concrete reviews, controls, and test cases can be derived from it. If it stops at sentences like "prompt injection could happen," the model is still too shallow.
What Explicitly Isn't Enough
A pure list of components like "frontend, backend, vector store, LLM, tool" doesn't describe a security model yet. Without data flows, it stays unclear which component processes which content, and where trust changes.
A pure list of controls isn't a threat model either. Statements like "we use RBAC, input validation, and guardrails" describe defenses, but not yet what they protect against or what assumptions sit behind them.
Especially dangerous is fixating on the model itself. Many real problems don't sit "in the LLM," but in the retrieval logic, the prompt composition, tenant isolation, tool authorization, output handling, or the ingestion pipeline. The model is often the most visible node, but not necessarily the actual scene of the accident.
2. The AI-Specific Core: Context Is Not Just Information
The central difference between a classical application and an AI application is that text and other data within the system are not merely processed — they frequently become the control surface itself.
A classical processing chain can look, simplified, like this:
Input → Business Logic → Database → Output
A RAG system tends to work more like this:
Input → Retrieval → Context Selection → Prompt Composition → Model → Output
An agent system extends this chain further:
Input → Model Decision → Tool Call → Tool Result → New Model Decision → Action or Response
In these architectures, several sources of information have influence over behavior:
- system and developer instructions
- user requests
- chat histories
- retrieved document chunks
- external websites or emails
- tool results
- memory or session content
- previous model outputs
These sources are not equally trustworthy. They are also not equally privileged. And yet they frequently end up together in one model context, where their technical separation is far less clear-cut than in classical program logic.
Two especially important principles follow from this.
Principle 1: Context Can Be Indirect Control
Document text can contain facts, but just as easily action instructions. If the text is selected by the retriever and pulled into the model context by the prompt builder, its content can gain influence over priorities, answers, and tool decisions.
The relevant question is therefore not just:
What data does the model see?
But rather:
What data can influence what the model believes, prioritizes, or does?
Principle 2: Model Output Is Not a Trustworthy Control Mechanism
Model output is probabilistic. If it's used directly as a tool decision, action parameter, HTML, SQL, code, or shell command, the system crosses a critical line. An unsafe suggestion can turn into a real action.
The model is allowed to produce suggestions. Authorization of real actions, however, should happen through deterministic, verifiable logic outside the model.
3. The Running Example System
For the further analysis, we'll use an internal AI support assistant.
Employees ask questions through a chat frontend. The backend uses RAG to determine matching content from an internal knowledge base. A prompt builder combines system instructions, the user request, chat history, and retrieved document chunks. The LLM produces a response or proposes using a tool. Through a tool broker, the application can create support tickets.
Main Components
- User
- Chat Frontend
- Backend / Orchestration
- Authentication and Session Layer
- Document Ingestion Pipeline
- Vector Store / Knowledge Base
- Retriever
- Prompt Builder
- LLM
- Tool Broker
- Ticket System
- Logging and Monitoring
This description is deliberately more concrete than "a chatbot with AI." It names the functions, the components, and the places where context gets assembled or effect gets created. Only this makes it possible to determine assets and attack paths.
4. Assets in AI Systems
An asset is anything whose disclosure, manipulation, loss, or abuse produces a relevant security impact. In AI systems, this includes not just classical data, but also control mechanisms, capabilities, and context.
Figure 2: AI applications protect significantly more than traditional business data. Relevant assets include not only data, but also control mechanisms such as system prompts and policies, context such as retrieved chunks and tool results, capabilities and permissions, generated outputs, and identity, session, and tenant information. Each asset class requires its own security objectives and controls.
4.1 Data Assets
Data assets include, among others:
- internal knowledge documents
- employee requests and chat histories
- support tickets
- personal or confidential company data
- API keys and secrets
- logs with sensitive content
- uploaded files
- tool returns
Typical protection goals are confidentiality, integrity, availability, and tenant separation.
4.2 Control Assets
Control assets determine how the system is supposed to behave:
- system prompt
- developer instructions
- prompt templates
- guardrails and policies
- tool rules
- retrieval selection rules
- context composition logic
- prioritization between instruction sources
These assets need, above all, integrity, confidentiality, and control fidelity. Manipulation can cause the system to pursue different goals, bypass boundaries, or process disallowed content.
4.3 Permission and Capability Assets
An agent can not only see information but execute actions. The capability itself is therefore an asset:
- ticket creation
- CRM and database access
- sending mail
- file access
- calendar changes
- plugin and API permissions
- delegation rights
- an agent's scope
The most important goals are authorization, least privilege, integrity, and traceability.
An unreliable model is annoying. An unreliable model with far-reaching permissions is dangerous.
4.4 Context Assets
Context assets are especially critical in AI systems:
- retrieved chunks
- chat history
- session memory
- user context
- tool outputs that flow back into the model
- the final assembled model context
Context is the basis for decisions, and indirect control. Its integrity, provenance, confidentiality, and correct separation between users or tenants are therefore independent security goals.
4.5 Output Assets
The output can be an asset too:
- generated answers
- generated ticket content
- SQL, code, or shell output
- HTML and Markdown content
- recommendations or decisions
- structured data that gets automatically processed further
Output needs integrity and safe downstream processing. It must not trigger disallowed actions, and it must be validated or sanitized before rendering or execution.
4.6 Identity and Session Assets
Multi-user and multi-tenant systems need a correct binding between user, session, tenant, retrieval scope, and tool execution:
- user identity
- roles and claims
- session assignment
- tenant or scope context
- access tokens
- delegation context
- the identity in whose name a tool is called
An error in this binding can cause a model to work technically correctly, but with the wrong context or under the wrong name.
Asset and Protection Goal Have to Match
The category "data" is too coarse. An internal manual, a system prompt, a permission to send mail, and a retrieved chunk are all text- or data-related, but security-wise completely different.
| Asset | Category | Primary Security Goals | Typical Abuse |
|---|---|---|---|
| Internal knowledge documents | Data | Confidentiality, integrity | Disclosure or manipulation |
| System prompt | Control | Integrity, confidentiality, control fidelity | Override, leakage, policy bypass |
| Ticket tool permission | Capability | Authorization, least privilege, auditability | Confused deputy, disallowed action |
| Retrieved context | Context | Integrity, provenance security, separation | Poisoning, cross-tenant mixing |
| Generated ticket content | Output | Integrity, safe downstream processing | Data takeover, injection, wrong action |
| Session-to-user binding | Identity | Authenticity, authorization, separation | Session mix-up, foreign context |
A practical heuristic:
- What happens if this element gets disclosed?
- What happens if it gets manipulated?
- What happens if it's used under the wrong name or scope?
- Does it influence what the model sees, prioritizes, or is allowed to do?
As soon as one of these questions shows a real security impact, you're very likely looking at an asset.
5. Data Flows and Trust Boundaries
A data flow describes which information moves from where to where, in what role, and under what control through the system. Security problems frequently don't arise inside a single component, but at the transitions between components.
A trust boundary is a point where at least one of the following conditions changes:
- Data crosses into a different trust zone.
- Control assumptions change.
- Content gets processed with higher privilege.
- Information turns into decision-relevant context.
- Model output gets translated into a real action.
- Shared infrastructure is used across different users or tenants.
The core question is:
Why do we trust this content more, starting from this point, than we did before?
Figure 3: The data flow of a RAG-based support assistant illustrates the major trust boundaries between user input, retrieval, model context, tool use, and output. Particularly critical transitions occur when untrusted content enters decision-relevant model context, model output becomes a real-world action, or tool results are fed back into the LLM context. Trust boundaries therefore exist not only at the external system perimeter, but at multiple points inside the AI application.
5.1 Typical Data Flow of the Support Assistant
- User → Frontend: user request
- Frontend → Backend: request plus session and identity context
- Backend → Retriever: retrieval query
- Retriever → Vector Store: similarity search and filtered document lookup
- Vector Store → Retriever: relevant chunks
- Retriever → Prompt Builder: retrieved context
- Prompt Builder → LLM: system prompt, user request, history, and chunks
- LLM → Backend: response or tool intent
- Backend / Tool Broker → Ticket Tool: validated ticket call
- Ticket Tool → Backend / LLM: tool result
- Backend → Frontend: final response
- Components → Logging: audit, error, and operational data
5.2 Critical Trust Boundaries
Boundary 1: User Input → Internal System
Untrusted user input enters internal processing. Risks are direct prompt injection, resource abuse, context manipulation, and policy bypass attempts.
Boundary 2: External or Weakly Controlled Content → Knowledge Base
Documents, websites, emails, or files get ingested and later used as a context source. Risks are poisoned content, manipulated chunks, and indirect prompt injection.
Boundary 3: Retrieved Context → Model Context
Document text gains direct influence over model responses and possibly tool decisions. This boundary is one of the most important AI-specific transitions.
Boundary 4: Model Output → Tool Call
Probabilistic output turns into a real action. Risks are unauthorized tool use, parameter injection, confused-deputy problems, and multi-stage abuse.
Boundary 5: Tool Output → Model Context
Tool results get fed back into the model. This creates a second input vector that can contain sensitive, manipulated, or instructive content.
Boundary 6: Model Output → Rendering or Downstream Processing
Model output gets displayed, copied, executed, or handed to other systems. Risks are unsafe HTML or Markdown, harmful queries, code execution, and blind trust in generated content.
Boundary 7: Tenant or Session → Shared Components
A shared vector store, memory layer, or tool service must not lead to shared visibility. Scope and identity checks have to apply before retrieval, context inclusion, and tool execution.
What Should Be Documented for Every Edge
A useful data flow model doesn't just note "backend talks to LLM." For every edge, three things should be captured:
- What flows?
- How trusted is it?
- What may happen next?
Example:
| Flow | Content | Trust Level | Downstream Effect |
|---|---|---|---|
| Retriever → Prompt Builder | Document chunks | mixed / potentially untrusted | influences the answer and tool decision |
| LLM → Tool Broker | Tool intent and parameters | untrusted proposal | can trigger a real action after authorization |
| Tool → LLM | API or ticket result | external / sensitive | becomes new model context |
| Backend → Renderer | generated response | untrusted output | gets displayed or processed further |
6. Attackers, Goals, and Abuse Scenarios
A threat model only becomes truly useful once architecture and assets are connected to realistic adversaries.
6.1 Attacker Goals Describe the Effect
"Prompt injection," "jailbreak," or "RAG attack" are not attacker goals. They describe techniques or risk classes.
A goal describes the intended effect:
- disclose confidential data
- see another tenant's data
- trigger a tool without authorization
- bypass guardrails or policies
- manipulate retrieval or context
- destroy answer integrity
- redirect the agent
- abuse resources or costs
- deceive users or influence processes
The rule of thumb is:
A goal describes what is meant to be achieved, not how.
6.2 Typical Attackers
External User
They have regular access to the application and can send requests, upload files, or observe behavior iteratively.
Malicious Content Supplier
They control documents, websites, emails, or files that the system reads or ingests. This actor is especially relevant for indirect prompt injection and context poisoning.
Insider or Legitimate Internal User
They have legitimate access but try to expand their visibility, repurpose internal tools, or bypass policies.
Tenant Attacker
They use the system within their own tenant and try to make foreign chunks, sessions, memories, or tool results visible.
Indirect Tool or Data Source Attacker
They don't attack the chat itself, but influence API responses, tool returns, or external data sources that later reach the model context.
6.3 From Goal to Abuse Scenario
An abuse scenario is a plausible description of how an actor uses an intended function against the system.
A good format is:
- Actor
- Goal
- Precondition
- Abused Function or Boundary
- Expected Impact
Example:
Actor: External user
Goal: Disclosure of internal support content
Precondition: Retrieval is scoped too broadly
Abused Boundary: Retrieval → Model Context
Expected Impact: Confidential information appears in the response
A sentence like "prompt injection could happen," on the other hand, is worthless: actor, goal, precondition, system reference, and impact are all missing.
6.4 Security, Reliability, and Mixed Cases
Not every incorrect model behavior is a security finding.
Figure 4: Not every failure of an AI system is automatically a security issue. A reliability problem becomes security-relevant when it affects a protected asset, a permission, or a trust boundary. Mixed cases are especially important: an ordinary model or retrieval failure may gain real security impact when combined with automation, privileged tools, or broken access control.
Reliability
The system works poorly or unreliably, without any protected asset being violated.
Examples:
- an incorrect summary
- hallucination with no connection to real confidential data
- irrelevant retrieval results
- wrong tool selection with no security-relevant effect
Security
An asset, a permission, or a trust boundary is affected.
Examples:
- another tenant's data appears in the response
- a tool executes an unauthorized action
- sensitive information gets disclosed from context or logs
- manipulated content influences a real action
Mixed Case
A quality error becomes security-relevant because it's coupled to rights, assets, or automated downstream processing.
Examples:
- faulty retrieval leads to cross-tenant leakage
- a faulty tool selection triggers a real action
- a faulty output structure gets automatically processed and produces an effect
The decisive sentence:
Not every act of stupidity is a security incident. It becomes dangerous the moment assets, rights, or trust boundaries are affected.
7. Attack Paths: Turning Risks into Testable Chains
An attack path is a concrete, plausible sequence of steps through which an attacker reaches their goal. It connects:
- actor
- entry point
- preconditions
- components
- trust boundaries
- sequence of steps
- impact
- possible controls
An abuse scenario says which function gets abused. The attack path shows how that happens technically.
7.1 Attack Path for Faulty RAG Retrieval
Goal: Disclose confidential content
Actor: External user
Entry Point: Chat request
Precondition: Retrieval is scoped too broadly, or not bound to user rights
Components: Frontend, Backend, Retriever, Vector Store, Prompt Builder, LLM
Critical Boundaries: User → System; Retrieval → Model Context
Sequence of Steps:
- The user sends deliberately crafted requests.
- The backend generates a retrieval query.
- The retriever insufficiently enforces the scope or tenant filter.
- Disallowed document chunks get selected.
- The prompt builder pulls them into the model context.
- The model uses the content in its answer.
- Confidential information gets disclosed.
Impact: Violation of confidentiality and tenant separation
Controls: ACL-bound retrieval filters, scope enforcement before search, post-retrieval authorization, context minimization, disclosure tests
7.2 Attack Path for Indirect Prompt Injection
Goal: Manipulate model behavior or tool use
Actor: Malicious content supplier
Entry Point: An ingested document
Precondition: The attacker can place content into a source the retriever considers
Components: Ingestion, Chunking, Vector Store, Retriever, Prompt Builder, LLM
Critical Boundaries: Content Source → Knowledge Base; Retrieved Context → Model Context
Sequence of Steps:
- The attacker plants a semantically relevant document.
- The document contains manipulative instructions.
- The ingestion pipeline chunks and indexes the content.
- A matching user query leads to retrieval of the manipulated chunk.
- The prompt builder pulls the chunk into the model context.
- The model inappropriately treats the contained instruction as actionable.
- The response, a tool intent, or a downstream process gets influenced.
Impact: Policy bypass, data disclosure, goal redirection, or tool abuse
Controls: source classification, ingestion governance, context isolation, untrusted-content labeling, tool gating, human approval for sensitive actions
7.3 Attack Path for Tool Abuse
Goal: Trigger a disallowed ticket action
Actor: External or internal user
Entry Point: A manipulatively phrased request
Precondition: Tool calls are not authorized independently of the model
Components: Frontend, Backend, LLM, Tool Broker, Ticket System
Critical Boundaries: User → Model Context; Model Output → Tool Call; Tool Output → Model Context
Sequence of Steps:
- The user phrases a seemingly legitimate task.
- The model produces a ticket intent and parameters.
- The tool broker accepts the model's decision without sufficient policy checking.
- The ticket system executes the action within the scope of the service identity.
- The tool result flows back into the model.
- The model produces further content or follow-on actions.
- A ticket gets created with disallowed content, recipient, or scope.
Impact: Process abuse, data handoff, unauthorized action
Controls: external policy engine, least privilege, parameter allowlisting, per-call authorization, confirmation step, audit logging
Top comments (0)