When you add retrieval-augmented generation to an LLM deployment, you are not just wiring up a smarter context window—you are adding a pipeline with multiple new attack surfaces. Every component in that pipeline, from document ingestion to vector storage to retrieval assembly, represents a place where security controls need to hold independently.
Here is what practitioners get wrong in each area, and what to do instead.
Your Knowledge Base Is an Untrusted Input
The instinct to treat internal documents as trustworthy is understandable but dangerous. In practice, most RAG knowledge bases pull from wikis, ticketing systems, shared drives, support email threads, and user uploads. Any of those sources can be influenced by someone trying to manipulate the system.
The mechanism is indirect prompt injection: an attacker embeds instructions inside a document knowing that if a legitimate user's query retrieves it, the model will process those instructions alongside real content. Studies have shown that crafted documents can be positioned to surface consistently in retrieval results. Since language models cannot reliably tell the difference between instructions and data, a malicious document can override system prompts, trigger unintended tool calls, or cause the model to leak other retrieved content.
The practical fix: treat every retrieved chunk as untrusted input. Separate instruction content from data content explicitly in your prompt templates. Constrain what tools or actions the model can take when executing queries. Verify document provenance during ingestion rather than assuming source equals trust.
Multi-Tenant Systems Need Structural Isolation, Not Just Filters
If your RAG system serves more than one customer—or even multiple teams with different data sensitivity levels—access control at query time is the critical failure point. Relying solely on metadata filters in application code is fragile. A single misconfiguration or injected parameter that alters the filter value can silently expose one tenant's documents inside another tenant's response, with no visible error.
Metadata filtering is not a security boundary. Hard partitions are: per-tenant namespaces, separate collections per tenant, or dedicated indexes for high-sensitivity workloads. Whatever approach you choose, re-verify document-level authorization on every query, not just at ingestion time. The assumption that an index only contains documents the requesting user is entitled to see is almost always wrong as schemas and permissions evolve.
Vector Stores Deserve the Same Controls as Your Primary Databases
Vector databases are often treated as infrastructure utilities rather than sensitive data stores, but that framing causes real problems. They hold semantically queryable representations of your most sensitive content.
Two issues compound this. First, similarity search allows anyone with read access to probe a corpus by meaning—not by knowing exact file names or keywords. Partial read access to a vector store is more useful to an attacker than it appears. Second, embeddings are not one-way hashes. Academic research has demonstrated that substantial portions of original text can be recovered from embedding vectors, and those reconstruction techniques have advanced considerably in recent years. Treat read access to your vector store as effective exposure of the underlying documents.
Controls that apply to your primary databases should apply here too: authentication required, no public endpoints, private network access, encryption at rest and in transit, narrowly scoped API keys, and query logging.
Deletion and Retention Break Across the Pipeline
RAG duplicates your data. A document lives in the source system, passes through an embedding pipeline, lands in the vector index, and may be cached in intermediate layers. Standard deletion processes usually touch only the source record.
This creates a serious compliance gap. Honoring a deletion request or enforcing a retention policy requires propagating that operation across every layer where a copy exists. Vectors derived from PII-containing content are not meaningfully anonymized—they carry recoverable information. If those vectors persist after the source record is deleted, the deletion never fully completed.
Before production, map the full data lifecycle: where does each chunk go, and what triggers its removal from every system that holds a copy? Automated lineage tracking—linking each vector chunk back to its source record—is the only reliable way to verify that deletion propagates completely.
A Practical Security Checklist
Ingestion: validate and sanitize document sources before indexing; scan ingested files for embedded secrets and executable content; treat all retrieved context as untrusted regardless of origin.
Access control: enforce tenant or user-level isolation at the index or namespace level, not just in filter parameters; re-check document-level authorization at retrieval time, not only when content was first indexed.
Vector database hardening: require authentication; restrict to private network access; encrypt at rest and in transit; rotate credentials; log all queries.
Prompt assembly: clearly delimit retrieved text from system instructions; cap retrieved volume and rank by relevance; prevent retrieved content from escalating the model's tool permissions.
Data governance: avoid embedding PII unless required and protected; honor deletion and retention policies across every layer; maintain lineage from each chunk back to its source record.
Monitoring: log retrieval queries and their sources; alert on anomalous retrieval patterns that might indicate systematic probing or data exfiltration.
Extending your threat model past the prompt layer to cover the full retrieval pipeline is the difference between a RAG deployment that is genuinely secure and one that appears secure until something goes wrong.
This guide originally appeared on agentpalisade.com. Agent Palisade helps small and mid-sized businesses put AI to work inside the tools they already use — practical automation, internal assistants, and AI security reviews. Book a free 30-minute call.
Top comments (0)