TL;DR— Most prompt injection defenses treat it like spam detection: classify the bad input, block it, move on. That approach plateaus because natural language has no reliable boundary between instruction and data. The fix that actually scales is architectural— provenance tracking and capability-scoped execution, the same lesson SQL injection taught us decades ago with prepared statements.
Every few weeks a new jailbreak technique makes the rounds, a vendor ships a patch, and the cycle repeats. Look at the shape of that cycle closely and you'll notice it's identical to the spam filter arms race from the 2000s: classify the bad input, block it, wait for the next mutation. It never converged then either. The reason prompt injection defense keeps plateauing is that the industry is solving the wrong layer of the problem. Detecting malicious text is a losing game because natural language has no syntactic boundary between an instruction and a piece of data. SQL had this exact problem, and it wasn't solved by getting better at pattern-matching malicious strings. It was solved by making the ambiguity structurally impossible.
The category error
A prompt injection classifier— whether it's a fine-tuned model, a set of regex heuristics, or a hosted moderation API— is fundamentally trying to answer: does this text look like an attack? That's a content question. But the actual vulnerability isn't in the content. It's in the fact that your system concatenates untrusted data (a scraped webpage, a support ticket, a tool response, another agent's output) into the same context window as your trusted system instructions, with no mechanism for the model to reliably tell them apart.
This is the same category error as thinking SQL injection is a string-sanitization problem. You can strip quotes, escape special characters, and blocklist keywords all day, and someone will find the encoding you missed. The actual fix was prepared statements: separate the query structure from the data values at the protocol level, so the database engine never has the opportunity to misinterpret data as code. The vulnerability didn't get patched. It got architected out.
Why detection plateaus
Guardrail models and jailbreak classifiers are Bayesian filters operating on an adversarial, unbounded input space. Every defense you publish becomes a training signal for the next attack. Indirect prompt injection makes this worse, because the attacker doesn't even need access to your chat interface— they just need their payload to end up somewhere your agent will read it: a PDF, a calendar invite, a GitHub issue, a product review. The attack surface isn't your prompt box. It's every document your agent is allowed to ingest.
The deeper issue is that a transformer processes a token stream, not a labeled data structure. Instructions and data live in the same representational space. You can add delimiters, role tags, and system-prompt framing, but the model is still doing next-token prediction over a flat sequence, and a sufficiently well-crafted piece of "data" can shift its behavior exactly like an instruction would. This isn't a bug that better training eliminates. It's a structural property of how these models currently ingest context. Treating it as a bug you can patch away with a stronger classifier is why the arms race never ends.
What the SQL analogy actually teaches
The lesson from prepared statements isn't "add more escaping." It's "stop trusting the channel to self-report its own trust level, and instead track provenance outside the content itself." Applied to LLM systems, that means every piece of context entering a prompt should carry a provenance tag that lives in your orchestration layer, not in the text the model sees: system, user, retrieved_untrusted, tool_output_untrusted. The model doesn't need to understand this tag semantically. Your control plane does.
Once you have provenance, you can enforce policy outside the model: content tagged retrieved_untrusted is never allowed to trigger a tool call directly, is never concatenated into a position that resembles a system instruction, and is never permitted to modify the agent's plan without a re-authorization step that originates from a trusted source. This is taint tracking, borrowed straight from web security, applied to context assembly instead of variable assignment. It doesn't require the model to be smarter about detecting attacks. It requires your pipeline to stop giving untrusted text the same execution privileges as trusted instructions.
Defense in depth means capability boundaries, not more classifiers
If provenance tracking is the schema-level fix, capability scoping is the runtime fix. The actual damage from a successful injection almost never happens in the text output— it happens in the action the model was permitted to take afterward: sending an email, executing a database write, calling a payment API, exfiltrating data through a crafted URL. Defense in depth for agentic systems means treating every tool call as if the instruction behind it might be attacker-controlled, and asking: what's the blast radius if it is?
- Least privilege by default. An agent summarizing customer emails should not hold the same credential as one that can issue refunds. Split them, even if it's the same underlying model.
- No untrusted-to-action shortcuts. Content from retrieval or tool output should never be able to directly trigger a high-privilege action without passing through a policy check that doesn't rely on the model's own judgment.
- Output-side sanitization. If your agent's output can end up rendered in a browser, executed as code, or interpolated into another prompt downstream, treat that output the way you'd treat any untrusted string headed for a sink— encode it for its destination.
- Human confirmation at privilege boundaries. Not for every action— that just trains people to click through— but specifically at the moment an agent crosses from read to write, or from internal to external effect.
None of this stops an injection from happening. It stops an injection from mattering. That's the actual definition of defense in depth: layers that assume the outer layer will eventually fail, and are designed so failure there doesn't cascade into a compromised action.
What to actually build
If you're responsible for an LLM system today, the ROI ordering is not: better jailbreak classifier, then better system prompt, then better guardrail model. It's the reverse. First, map every place untrusted content enters your context and tag it at the orchestration layer. Second, enumerate every tool call your agent can make and assign it the minimum privilege required, with no tool holding blanket access because it was convenient during a demo. Third, put policy enforcement on actions, not on text— the model's opinion about whether an instruction is legitimate is not a security boundary. Only after those three are in place does a jailbreak classifier earn its keep, as a monitoring signal rather than a gate.
The model providers will keep improving instruction-following robustness, and that helps. But waiting for the model to become un-injectable is waiting for a property no current architecture guarantees. The systems that hold up under real adversarial pressure are the ones that assumed injection would eventually succeed, and built the blast radius small enough that it wouldn't matter when it did.
Top comments (0)