Cross-post. Original: stellarbytecapital.com/blog/ai-agent-security-threat-model
While a chatbot only produces text, an autonomous agent takes actions: it calls tools, runs code, moves data, and spends money. That shift changes the security problem entirely. "Is the prompt safe?" is no longer the question. The question is: what can this agent do, and what stops it when it goes wrong?
Treating agent security as prompt filtering is how teams end up with an impressive demo and a production incident. You need a threat model. Here's ours.
Three planes of attack surface
An agent's exposure lives on three distinct planes. Confusing them is why defenses miss:
- The model plane — what goes into and out of the LLM: prompts, retrieved documents, tool outputs fed back as context.
- The action plane — the tools the agent can invoke: shell, HTTP, database, file I/O, payments, and third-party tools / MCP servers.
- The runtime plane — where the agent's code and tools actually execute: the process, container, host, and network.
Prompt filtering only touches the first plane. Most real damage happens on the second and third.
The threat model
| Threat | Vector | Primary control |
|---|---|---|
| Prompt injection | Malicious instructions hidden in a page/doc/tool output the agent reads | Treat retrieved content as untrusted; confirm before consequential actions |
| Tool misuse / over-permission | Broad tools (shell, DB write, payments) the agent is steered into abusing | Least privilege; gate high-impact tools behind explicit approval |
| Code-execution escape | Agent runs generated code that breaks out of its sandbox | Disposable one-shot containers; gVisor/Firecracker for multi-tenant |
| Data exfiltration | Agent sends private data to an external endpoint | Default-deny egress; allowlist required hosts; log every call |
| Credential theft | Agent code reads API keys mounted into its environment | Keep secrets out of the runtime; proxy authed calls through a trusted layer |
| Supply-chain / malicious tools | A third-party tool or MCP server behaves adversarially | Vet and pin tools; isolate them like generated code |
| Unbounded cost | Runaway loop burns tokens, CPU, or spend | Per-user quotas; hard caps on iterations, tokens, wall-clock |
| Cross-tenant leakage | One user's run leaks state into another's | No reuse between runs; isolate state per execution |
The design principles that hold it together
- Least privilege everywhere. Every tool, mount, and network path defaults to off.
- Assume the code and the content are hostile. Generated code, retrieved docs, and tool outputs are all untrusted input.
- Human-in-the-loop for consequential actions. Reading is cheap; sending money or deleting data should require a confirmation gate, not the model's judgment alone.
- Isolate execution, deny egress. An escaped or hijacked agent with nowhere to send data is a contained one.
- Audit everything. You cannot secure what you cannot reconstruct.
Prompt filtering asks "will the model say something bad?" A threat model asks "when something goes wrong, what's the blast radius?" — and shrinks it to near zero.
We're Xingyao Byte — building secure AI-execution layers, quant trading systems, and payment platforms. Remote, async-first → stellarbytecapital.com
Top comments (1)
The three-plane split is useful but the runtime plane is the one teams collapse into the action plane in practice. A container is not a security boundary when the agent has a shell tool. The container is just the action plane with a different label. The real boundary is whether a tool can reach a resource the agent's process owner can access but the agent's task should not. That is a capability question, not a deployment topology question.
The action plane section is the strongest part of this model. Least privilege on tools is table stakes. What I would add: the approval gate on high-impact tools is itself a tool call. The same agent that is being gated can, in theory, craft a prompt that makes the gate approve itself. The gate needs to be a separate process with no shared context, or the model plane contaminates the action plane.