Threat-Modeling an LLM Feature Before It Reaches Production

Adding an LLM to an application creates a new interpreter for untrusted text. That interpreter may also retrieve private data, call tools, or write to other systems. The useful security question is therefore not “Is the model safe?” but “What can data flowing through the model cause the application to do?”

Here is a compact threat-modeling method for answering that question before launch.

Start with assets, not prompts

List what the feature must protect. For a support assistant, the list might include:

• customer messages and attachments;
• retrieved account records;
• system instructions and internal policies;
• tool credentials;
• actions such as refunds, password resets, and ticket updates;
• logs, traces, and evaluation datasets.

This prevents a common modeling mistake: treating prompt injection as the whole problem. Injection is one path. The asset and the reachable action determine its impact.

Draw the trust boundaries

A minimal LLM feature often has more boundaries than its UI suggests:

user input
   |
   v
application API ---> model provider
   |                     |
   v                     v
retrieval index       generated text
   |
   v
tool executor ---> business systems

Mark every place where data changes authority or ownership. User text is untrusted. Retrieved documents may also be untrusted if users can upload or edit them. Model output is untrusted even when the application supplied a careful system prompt. A tool response can contain hostile instructions too.

That last point matters in retrieval-augmented systems: a document is data to the application, but the model may interpret it as an instruction.

Write abuse cases as data flows

Use concrete “input → interpretation → effect” statements. For example:

Input path	Unsafe interpretation	Possible effect
Uploaded PDF	Text overrides the system instruction	Private retrieval results appear in output
Tool argument	Generated account ID is accepted as authoritative	One user acts on another user’s record
Model response	Markdown link is rendered without checks	Phishing or unsafe navigation
Trace pipeline	Full prompts are retained by default	Secrets persist in logs

This format is easier to turn into controls and tests than a label such as “prompt injection risk.”

Put authorization outside the model

The model may propose an action; application code must decide whether it is allowed.

For every tool, define:

1. a narrow input schema;
2. server-side identity and permission checks;
3. limits on scope, amount, and frequency;
4. a confirmation step for consequential actions;
5. an audit record that does not expose unnecessary prompt data.

Do not let the model select a user identity, tenant, or authorization scope from free text. Derive those values from the authenticated session. A refund tool, for example, should receive an order ID but obtain the customer and maximum refundable amount from trusted backend state.

“The system prompt says not to” is not an authorization control.

Separate content from instructions

When retrieved or uploaded text enters a prompt, delimit and label it as data. This can reduce accidental instruction mixing, but it is not a complete defense. Enforce the important boundary after generation:

• validate structured output against a schema;
• allowlist tool names and argument shapes;
• sanitize or safely render links and markup;
• filter retrieval by the caller’s permissions before context reaches the model;
• require deterministic policy checks before side effects.

The goal is to make a successful injection produce an untrusted suggestion, not an authorized action.

Test properties, not magic phrases

A useful adversarial suite varies the attack path:

• direct instructions in the user message;
• indirect instructions inside retrieved documents;
• encoded, multilingual, fragmented, or quoted instructions;
• attempts to cross tenants or retrieve hidden fields;
• malformed tool arguments and repeated tool calls;
• attempts to place secrets in output or telemetry.

Assertions should describe system properties: “No document outside the caller’s tenant appears,” “A tool call without server-side authorization is rejected,” or “A generated URL cannot use an unsafe scheme.” Exact output matching is usually too brittle for this layer.

Run these cases whenever prompts, models, retrieval rules, or tools change. A model update is a behavior change even when the API contract stays the same.

Plan containment and evidence

Assume one control will fail. Useful containment options include read-only tools by default, per-tool rate limits, small transaction limits, short-lived credentials, human confirmation, and an immediate tool kill switch.

Collect enough evidence to investigate failures: request and tool-call IDs, model and prompt versions, policy decisions, timing, and redacted error categories. Avoid treating complete prompts as harmless debug strings; they can contain personal data, credentials, and proprietary context.

A pre-launch review

Before enabling the feature, the team should be able to answer:

• Which inputs are attacker-controlled, including retrieved content?
• Which private assets can enter model context?
• Which side effects are reachable, and who authorizes each one?
• What validation happens after generation?
• What is logged, retained, and redacted?
• Which adversarial cases run in CI or release evaluation?
• How can operators disable tools without disabling the whole product?

Two useful baselines are the OWASP Top 10 for LLM Applications and NIST’s AI 600-1 Generative AI Profile. They are most valuable as inputs to a system-specific review, not as checklists that certify a design.

An LLM can remain probabilistic while the surrounding security boundary stays explicit. Keep authority in deterministic code, minimize what the model can reach, and test the paths where untrusted text meets privileged actions.