🔔 Originally published at OpsBuzz — real-time AI Security alerts for engineers.
Why LLM security needs a different threat model
Large language models are moving from experiments into production workflows:
support agents, internal copilots, security assistants, code review tools,
data analysis bots, and customer-facing automation.
That shift changes the security model. Traditional web application risks still
matter, but LLM systems add new failure modes because the model interprets
natural language, calls tools, summarizes untrusted content, and often sits
between users and sensitive business systems.
For engineering teams, the practical question is not whether an LLM is
"secure" in the abstract. It is whether the full system around it can safely
handle hostile input, untrusted data, external tools, permissions, logs, and
third-party dependencies.
1. Prompt injection
Prompt injection is still the most important LLM vulnerability engineers need
to understand in 2026.
In a prompt injection attack, a user or external document includes instructions
that attempt to override the system's intended behavior. The model may be told
to ignore previous instructions, reveal hidden context, call a tool, exfiltrate
data, or produce output that violates policy.
Prompt injection is especially risky when the LLM can read untrusted content
and then take actions based on that content.
Common examples include:
- A support bot reading a malicious customer message
- A coding assistant processing a poisoned issue or pull request
- An email agent summarizing a message that contains hidden instructions
- A RAG system retrieving hostile text from a knowledge base
- An internal copilot deciding whether to call a privileged tool
Action required: Treat all user input, retrieved documents, webpages,
emails, tickets, and repository content as untrusted. Do not rely on prompting
alone for authorization or security boundaries.
2. Indirect prompt injection
Indirect prompt injection happens when the attacker does not directly prompt
the model through the chat interface. Instead, they place malicious
instructions inside content the model later reads.
This matters because many production LLM systems now ingest external context:
webpages, PDFs, Slack messages, support tickets, GitHub issues, documentation,
CRM records, and search results.
The risk is subtle. A user may ask a harmless question, but the model retrieves
a poisoned source and follows instructions embedded inside it.
Examples:
- A webpage tells an agent to leak the user's private data
- A document tells a summarizer to include false security guidance
- A ticket tells an automation bot to change account settings
- A repository file tells a coding assistant to modify CI configuration
Action required: Separate data from instructions wherever possible. Add
tool allowlists, approval steps, source trust scoring, and output validation
for workflows that read external content.
3. Excessive agency and unsafe tool access
The biggest LLM incidents often come from giving an agent too much authority.
An LLM with no tools can produce bad text. An LLM with tools can create tickets,
change permissions, send messages, modify files, query databases, or trigger
deployments.
Tool access turns model behavior into real operational impact.
High-risk tool patterns include:
- Broad database read access
- Write access to production systems
- Ability to send external messages
- Access to secrets, tokens, or credentials
- File system access across large workspaces
- CI/CD or deployment actions without review
Action required: Apply least privilege to every LLM tool. Require explicit
confirmation for destructive actions, isolate high-risk tools, and log every
tool call with the triggering user, input, and result.
4. Data leakage through prompts, context, and logs
LLM systems often collect more data than teams realize. Prompts may include
customer records, source code, credentials, incident details, internal docs, or
private messages. That data can leak through model output, debug logs, vendor
retention settings, analytics tools, or accidental exposure in traces.
Sensitive context can also be exposed through seemingly safe features like
conversation history, support transcripts, or retrieval snippets.
Questions to review:
- What data is included in prompts?
- Where are prompts and completions logged?
- Which vendors or services receive prompt data?
- Are secrets filtered before model calls?
- Can users access another user's conversation context?
Action required: Build a data classification policy for LLM inputs and
outputs. Redact secrets, limit retention, restrict log access, and avoid
placing highly sensitive data into prompts unless there is a clear business
need and security review.
5. RAG poisoning and retrieval manipulation
Retrieval-augmented generation makes LLMs more useful, but it also creates a
new attack path: poisoning the knowledge source.
If an attacker can influence indexed documents, search rankings, embeddings,
metadata, or retrieval filters, they may be able to shape what the model sees
and how it answers.
RAG risks include:
- Poisoned documentation
- Malicious support tickets
- Stale or incorrect security guidance
- Over-permissive document access
- Cross-tenant retrieval leaks
- Hidden instructions inside retrieved chunks
Action required: Enforce access controls at retrieval time, not just at
indexing time. Track document provenance, review high-impact sources, and make
retrieved context visible in logs for investigation.
6. AI supply chain attacks
AI systems depend on a fast-moving supply chain: model providers, open-source
models, Python packages, npm packages, vector databases, evaluation libraries,
agent frameworks, plugins, prompts, datasets, and container images.
Attackers increasingly target that ecosystem because AI teams often adopt new
packages quickly and wire them into sensitive workflows.
Supply chain risks include:
- Malicious model files
- Compromised packages
- Typosquatted AI libraries
- Poisoned datasets
- Unsafe deserialization
- Unreviewed plugins or connectors
- Hidden network calls in tooling
Action required: Treat AI dependencies like production dependencies. Pin
versions, scan packages, review model provenance, isolate evaluation
environments, and avoid loading untrusted model artifacts in privileged
runtime contexts.
7. Insecure output handling
LLM output is untrusted data. It should not be rendered, executed, queried, or
passed into downstream systems without validation.
This is easy to miss because model output often looks polished and authoritative.
But the model can produce HTML, SQL, shell commands, code patches, JSON, URLs,
or configuration snippets that cause harm if trusted automatically.
Dangerous patterns include:
- Rendering model output as raw HTML
- Executing generated commands automatically
- Passing generated SQL directly to a database
- Applying generated code patches without review
- Using generated URLs in automated workflows
- Trusting generated JSON without schema validation
Action required: Validate model outputs with schemas, sanitizers, allowlists,
and review gates. Keep generated content separate from executable actions.
8. Broken authorization in AI workflows
LLM applications often combine several identity layers: the end user, the app,
the model provider, retrieval systems, and tool integrations. Authorization
bugs appear when those layers are blurred.
An assistant should not be able to retrieve a document, call a tool, or perform
an action unless the requesting user is allowed to do that directly.
Common mistakes:
- Using service accounts with broad access for all users
- Retrieving documents before checking user permissions
- Letting the model decide authorization from text
- Sharing conversation memory across tenants
- Calling tools based only on model intent
Action required: Enforce authorization outside the model. Every retrieval
and tool call should be checked against the user's identity, role, tenant, and
current session.
9. Model behavior drift and evaluation gaps
LLM behavior can change when prompts, models, tools, retrieval sources, or
dependencies change. A workflow that was safe last month can become risky after
a model upgrade or prompt edit.
Engineering teams need regression tests for AI behavior, especially around
security-sensitive flows.
Evaluate for:
- Prompt injection resistance
- Refusal behavior
- Tool-call boundaries
- Secret leakage
- Cross-tenant isolation
- Unsafe code or command generation
- Hallucinated security advice
Action required: Add AI security test cases to CI for critical workflows.
Run evaluations before model upgrades, prompt changes, retrieval changes, and
new tool integrations.
10. Overtrust in AI-generated security guidance
LLMs can help summarize alerts, explain vulnerabilities, and draft remediation
steps. They can also hallucinate package names, invent CVEs, misread severity,
or recommend unsafe mitigations.
This is especially risky during incidents, when teams are moving quickly and
may accept confident output without verification.
Action required: Require source links, cite retrieved evidence, and keep a
human review step for incident response, vulnerability triage, and production
security changes.
A practical LLM security checklist
Engineering teams do not need to solve every AI security problem at once. Start
with the controls that reduce the most risk:
- Treat prompts, retrieved context, and model output as untrusted
- Keep authorization outside the model
- Restrict tools with least privilege
- Require confirmation for destructive actions
- Redact secrets before model calls
- Log tool calls and retrieved sources
- Validate structured outputs with schemas
- Scan AI dependencies and model artifacts
- Test prompt injection and data leakage scenarios
- Review model, prompt, and retrieval changes before release
What engineers should watch in 2026
LLM security is becoming an operational discipline, not a research topic.
Teams are putting AI into production systems, and attackers are following the
same path.
The highest-risk systems are not simple chatbots. They are LLM applications
with access to internal data, external content, user accounts, automation tools,
and production workflows.
If your AI system can read sensitive data or take action, it needs the same
engineering rigor as any other production service: threat modeling, access
control, observability, testing, incident response, and change management.
Stay ahead of AI Security threats
OpsBuzz tracks prompt injection CVEs, malicious models, PyPI supply chain
attacks and more in real-time.
👉 Get free alerts at opsbuzz.dev
📧 Free email or Slack delivery
Top comments (0)