Prompt injection is a trust-boundary bug in the retrieval pipeline
A retrieval-augmented application concatenates retrieved text with instructions and sends both to a model. When the retrieved text was written by an attacker, the model receives attacker-provided instructions in the same channel as its operator-provided ones. From that point, no prompt wording is a boundary, because the attacker's text arrives with the same authority as the context it was retrieved alongside.
The boundary that is missing
In a conventional application, data from an untrusted source is parsed, validated and quoted. It does not become executable. SQL injection became manageable by separating the query from the data, which required a mechanism the database enforced rather than a policy the developer remembered.
A language model has no equivalent separation by default. Its input is a token stream, and instructions are not syntactically distinguishable from content. Any text in the context window can influence behaviour, and a document written to contain a directive is not different from a document that happens to mention one.
Where the exposure lives
Risk is proportional to what the application can do after the model produces output.
A retrieval system that answers questions over a corpus can leak content the user is entitled to, or return wrong information. Serious, and reversible.
A system with tool access is different. If the model can call an API, send a message, write to a database or execute code, then an injected instruction that is followed once is an action taken with the application's privileges and the user's session.
The gap between those two cases is a design decision, and it is often made implicitly when a tool is added for convenience.
Controls that reduce the exposure
Treat retrieved content as data. Wrap it in a structure the model is trained to treat as untrusted, and keep instructions out of that structure. This helps and does not solve; behaviour under adversarial content is probabilistic.
Constrain the tools. Each tool should have the narrowest permission that satisfies the use case, scoped to the current user rather than to the application's service identity. An injected instruction can only cause the damage the tool permits.
Require confirmation for consequential actions. An action that changes state, sends data or costs money should require a human decision or a second, independent check. That confirmation is a boundary the model cannot cross on its own.
Validate outputs against a schema before acting. A tool call that does not match the expected shape is rejected, which catches garbled output and some attempts to redirect the call.
Egress-restrict the runtime. A retrieval pipeline rarely needs arbitrary internet addresses. Restricting its network access limits exfiltration even when the model is manipulated.
Detection is weak, so design for containment
There is no reliable detector for an injected instruction, because the carrying surface and the attack are the same channel. Filters catch known patterns and fail against a rephrase.
A design that relies on detecting injection is therefore fragile, and one that relies on limiting the effect of a successful injection is durable. That is the same reasoning applied to memory corruption: you cannot prove an exploit never lands, so you bound the damage and detect the attempt.
Log what the model did with the tool, not only what it said. A record of a tool call, its arguments, the user session and the retrieved document that preceded it answers the incident question, and it is frequently the only evidence available.
The design review question
For every AI feature with tool access, ask: if the retrieved content were written by an attacker, what is the worst action the model could take that the user would not notice? The answer is the risk, and the tool permission model is the only place it can be reduced.
References
- NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
- NIST SP 800-53 Rev. 5, Security and Privacy Controls: https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final
- OWASP Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/
Top comments (2)
The design review question at the end is the useful one, and "the user would not notice" is the part teams skip. A confirmation step only works if the person approving sees what is actually about to happen, not a summary written by the same model. One cheap habit: log each tool call with its arguments and the document that preceded it, so the first incident review is a lookup instead of a reconstruction.
Agree that prompt wording can't be the boundary. In my own agent code, the send gate is a check in code, outside the model. A refusal is a path that exists, not a sentence the model is asked to respect. The model can read the injected text. It cannot reach the send.