LLM applications aren't just traditional web apps with an API call bolted on. When a model sits in the middle of your request-handling logic, it changes the attack surface in ways that catch engineering teams off-guard — particularly because models can't distinguish between legitimate system instructions and attacker-supplied content.
This checklist covers six risk categories that teams building LLM-backed features need to address before going to production.
Why the Threat Model Is Different
In a conventional web application, your code is what executes instructions — and code doesn't confuse user input for control flow unless you've already made a mistake like an injection vulnerability. With LLMs, every piece of text in the context window is processed as tokens. System prompts, user messages, retrieved documents, tool outputs — the model sees them all the same way.
This means untrusted input can reach parts of the system that would otherwise be protected. A document retrieved from a knowledge base might contain attacker-controlled text that instructs the model to behave differently. A user message might attempt to override system-level instructions. Because models are designed to be helpful and follow instructions, they're inherently susceptible to this category of abuse.
The foundational assumption: everything the model reads is potentially untrusted, and everything it writes must be validated before it goes anywhere.
Six Risk Domains and 16 Controls
Trust Boundaries
These three controls treat the model as an untrusted component at both ends of a transaction:
- Treat all model output as untrusted input for any downstream system — browsers, databases, interpreters
- Use message roles (system, user, assistant) to structurally separate instructions from user content
- Treat retrieved or third-party content as untrusted, even when you control the retrieval layer
Input and Output Validation
Models don't produce deterministic output, so you can't assume format or content will always be safe:
- Validate model output against strict schemas before acting on it
- Enforce allowlists for actions, URLs, and identifiers the model is permitted to reference
- Use JSON mode or structured outputs to constrain what the model can return
Data Exposure and Privacy
Whatever you send to a model provider leaves your infrastructure. This is a different kind of data-boundary concern than most applications face:
- Minimize sensitive data sent to providers; redact PII and secrets before they enter context
- Confirm your provider's data-retention and training-opt-out settings before processing any regulated data
- Avoid logging full prompt and response payloads when they contain personal or sensitive information
Access Control and Excessive Agency
Agentic applications that give models access to tools or APIs introduce what's sometimes called the "confused deputy" problem: the model acts with application-level privileges instead of user-level privileges. This has predictable consequences.
- Enforce user permissions on all data the model retrieves and all actions it can trigger
- Scope tool definitions narrowly — the model should only be able to do what the requesting user is authorized to do
- Require explicit human confirmation before high-impact or irreversible actions execute
Abuse and Cost Controls
AI endpoints are expensive to operate and attractive targets for abuse. Denial-of-wallet is a real attack class — someone else's AI budget becomes your liability:
- Authenticate and rate-limit all AI-facing endpoints the same way you would any sensitive API
- Set per-user token and output caps and enforce them server-side, not just in the client
- Add timeouts and circuit breakers to prevent runaway inference costs from cascading
Monitoring
Monitoring is often the last thing teams implement, but it's necessary for detecting attacks that don't trigger validation errors:
- Log tool calls and high-risk model actions in a tamper-resistant, queryable store
- Monitor for anomalies in model behavior, output patterns, and spending over time
Gaps That Commonly Get Missed
System prompt secrecy is not a security control. Prompts are frequently recoverable through direct instruction or indirect probing. Never place secrets, credentials, or security-critical logic inside a system prompt — authorization must live in your application code, not in the model's context.
Provider data governance depends on the tier. Enterprise offerings from major providers include data-retention controls that consumer or free-tier APIs typically don't. If you're processing personal data or information subject to compliance requirements, verify the data handling terms in your contract before building, not after.
The confused deputy problem compounds as agentic scope expands. Applications where the model can send messages, write to databases, call external APIs, or take other persistent actions carry substantially higher risk when access is scoped at the application level rather than the user level. Every tool you add to a model's toolkit deserves a straightforward question: what happens if an attacker manipulates the model into using this capability against the user's interest?
The principle connecting all sixteen controls: treat the model as a credulous, non-deterministic component — validate everything it reads and emits, enforce authorization and spending limits in your own code, and never grant it more access than the user it is acting on behalf of.
This guide originally appeared on agentpalisade.com. Agent Palisade helps small and mid-sized businesses put AI to work inside the tools they already use — practical automation, internal assistants, and AI security reviews. Book a free 30-minute call.
Top comments (0)