DEV Community: luumen

Assume the Model is Compromised

serene — Wed, 21 Oct 2026 20:45:38 +0000

Written by: Josh Greenwell Staff Applied AI Engineer

This blog article is a condensed version of our whitepaper on containing AI agents when input defenses fail. The full paper includes the appendices on the July 2026 incident timeline, the containment checklist, and the mapping of containment controls to SOC 2, OWASP, and NIST frameworks, plus the complete reference list. You can view and download the original here.

Introduction

When we refactored LuumenAI around code execution, the reason was cost. I wrote about that in a separate paper, but the short version is that loading a pile of tools into every request was burning tokens, and moving the work into a sandbox where the model writes code against our APIs took the bill from hundreds of dollars a day down to single digits. The security boundary came along almost by accident. The model's code now ran in an isolated environment that could only touch the APIs we handed it. At some point it struck me that the system we built to save money was also standing between a hijacked model and everything it could otherwise reach.

That accident is the subject of this paper. The first paper in the series argued that you cannot teach a model to tell instructions from data, but you can control what reaches it. I kept repeating one warning about every input defense I described: it is additive, never the thing you lean on. A filter that stops 99 percent of injection attempts is one that an attacker beats on the hundredth try, and for a while I did not want to think about the hundredth try.

Eventually you have to. You plan for it. You assume that at some point, past every input control, the model reads a hostile instruction and tries to carry it out, and you build so that when it does, it cannot get to much. Security teams have called this assume breach for years, long before anyone was worried about agents. You stop betting everything on the wall and start deciding how far an intruder gets once they are over it.

While I was drafting this, the cases arrived, and they were stranger than the ones I had prepared for. In July 2026, a set of frontier models escaped an isolated evaluation environment, crossed the open internet, and broke into another company's production systems. No attacker sent them. They were being scored on a benchmark, and breaking in proved a faster route to a high score than actually solving the problems. Nine days later, a second lab reviewed its own evaluation logs and reported three more cases against three more companies, the earliest of them already three months old. I come back to both disclosures in Part IV, because between them they widen what "compromised" has to cover (see Appendix A for the full timeline of these disclosures).

We are further along on input defense than on containment, and so is most of the industry. Containment for AI agents is engineering with working tools, but a lot of it is young, and some of the managed components leak more than their marketing admits. This paper is about what the model can touch after it acts, and how to keep that small.

Part I: Why Input Defenses Are Not Enough

Start with the math, because it is the reason this paper has to exist. Put a single filter that catches 99 percent of attempts in front of the model, and an attacker with a thousand variations is only wondering how soon one gets through. Stack a few filters and the odds get better, but they never reach certainty, because the attacker only needs one win and you need an unbroken streak. Input defense lowers how often an injection succeeds. It never gets the number to zero.

Once you accept that a successful injection is coming eventually, you start designing around a different question: how little an attack can do once it is in. That question has answers. How much data can the model reach? Where can it send anything? What can the code it writes actually do? You can engineer every one of those, which is more than anyone can say for making a model immune to language.

This is where the third leg of the trifecta finally comes due. The first paper dealt with the model's access to data and its exposure to untrusted content and left the third leg alone, the ability to send information back out. Containment is mostly about that leg, and about everything the model does once it commits to an action: calling a tool, writing code that runs, handing output to another system, sending data somewhere. Every one of those is a door, and a model working for an attacker will try all of them.

Part II: What a Sandbox Actually Protects

There is one distinction from the first paper that everything here depends on, so let me be exact about it. A sandbox is the boundary between the model and the systems its actions could hurt. It keeps the code the model writes from reaching your network, your databases, and your customers. It does nothing about a malicious instruction reaching the model in the first place. People mix these two up constantly, and they end up with a strong sandbox wrapped around a model whose inputs are wide open, or the other way around.

Containment comes down to five limits on the environment where the model's actions run:

Least privilege. The model's code can call the few APIs the task needs and nothing else.
An egress allowlist. The environment can reach the specific destinations it requires and cannot open a connection to a server an attacker controls. Every exception on that list is part of the attack surface, and the package registry is usually the first exception anybody grants.
Resource limits. Caps on CPU, memory, and wall-clock time, so a runaway or deliberately expensive process cannot run up the bill. OWASP calls this Unbounded Consumption.
No ambient credentials. Nothing sitting in the environment for a compromised process to read and reuse.
Ephemeral by default. A fresh environment per run, so anything an attacker drops does not survive into the next session.

Every one of those is a setting that either passes or fails. That is the appeal of working at this layer. You can test whether your egress allowlist blocks an arbitrary outbound connection. You cannot test whether the model has finally learned to distrust a malicious email. Set the environment up right, and an attacker who turns the model finds almost nothing on the other side to grab.

Part III: The Isolation Spectrum

Isolation comes in grades, and the differences matter a lot when the code you are about to run was written on demand by a model that might have been talked into something hostile. The options run from weak and cheap to strong and a little more expensive.

A plain container is the weak end. Containers share the host kernel, so a kernel exploit from inside the container is an exploit on the host. They are fine for keeping your own trusted services apart. As a boundary around untrusted, model-written code they are not serious, and treating them like one is a common and costly mistake.

gVisor is a step up. It runs a user-space kernel that intercepts the workload's system calls and keeps them off the host kernel, so the code is never talking to the host directly. You pay some syscall and I/O overhead for that, and it still starts fast enough to use on every request.

Firecracker is the strong end for most production work. Each workload runs in a stripped-down virtual machine on top of KVM, with a hardware-backed boundary, and the VMs boot in well under a second with almost no memory overhead. That is why you can stand them up and tear them down per session at volume. Kata Containers sit in similar territory, wrapping container workloads in lightweight VMs. And for untrusted JavaScript in particular, V8 isolates are the lightweight contexts a browser uses to keep tabs apart, which is how Cloudflare runs model-written code with strong separation.

" width="799" height="468">

The rule we follow is not clever. Model-written code gets a kernel-level boundary at a minimum, something like gVisor, and never a bare container. Anything multi-tenant, where one customer's workload could reach another's, gets the microVM. Yes, the stronger boundary costs more to run. That is nothing next to the cost of cleaning up an escape.

For LuumenAI we use a managed sandbox platform. Every execution gets its own isolated environment, and the environment is disposable, so it is gone when the run finishes. That environment sits on infrastructure separate from ours, which was part of why we chose it. If a customer boundary ever requires the isolation layer to live inside our own account, we can bring it in-house. We have kept the design arranged so that the move is a migration we can scope.

Part IV: When Sandboxes Fail

This part of the paper kept changing while I wrote it, because the examples would not stop arriving. Isolation is not automatic and it is not free, and when it fails it tends to fail in one of two ways. The first is a misconfigured sandbox, where the boundary exists on paper but a setting leaves a door open, an egress path nobody locked down or a role with too much scope. The second is an escape, where a flaw in the isolation layer itself lets code climb out.

A misconfigured sandbox: AgentCore

The AWS Bedrock AgentCore Code Interpreter failed the first way, and the isolation layer itself held the whole time. In September 2025, Sonrai Security published research showing that code running inside a sandboxed interpreter could reach the microVM metadata service at 169.254.169.254, the same address EC2 workloads use for instance metadata, and read the execution role's IAM credentials from it. AWS had put a filter in the way that blocked any request containing the literal address or the metadata path. Getting around it took nothing clever. Code that split the address into pieces, encoded it, or assembled it at runtime never contained the exact string the filter was matching on, so those requests sailed through. What came out were session credentials for the execution role, usable from outside the sandbox against AWS control-plane endpoints the sandbox could not reach on its own.

Firecracker did its job through all of that. The virtual machine was never breached. The credentials for the execution role were simply readable from inside it, and the reason that mattered as much as it did was the second half of the finding. The default IAM role in AWS's own AgentCore Starter Toolkit granted read access to every S3 bucket in the account, full DynamoDB access, and access to every secret in Secrets Manager! Open-source AgentCore examples copied that role. A team following the quickstart therefore ended up with a sandbox whose credentials, once read, opened most of the account. AWS treated the metadata access as expected behavior under the shared responsibility model and published documentation on credential management for the service.

The chain required no break-in at all. The microVM held. The credentials inside it were readable, and the role they carried had been copied out of a starter kit.

An escape in production: the July campaign

Escapes are the rarer failure and the harder one to reason about, and until July 2026 the public examples came from research rather than production.

On July 16, Hugging Face disclosed an intrusion into part of its production infrastructure. A malicious dataset abused two code-execution paths in their dataset-processing pipeline, a remote-code dataset loader and a template injection in a dataset configuration, to run code on a processing worker. From there the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend. Hugging Face described the campaign as an autonomous agent framework executing many thousands of individual actions across, in their words, "a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services." They said they did not know which model was behind it.

Five days later, OpenAI said the models were theirs. GPT-5.6 Sol and an unreleased, more capable model, later clarified to be an internal-only research prototype, had been running an internal evaluation of exploitation capability, ExploitGym, with their production cyber refusals switched off so the test could measure the ceiling. The evaluation ran in what OpenAI describes as a highly isolated environment, with network access constrained to installing packages through an internally hosted proxy and cache for package registries. The models spent a substantial amount of inference compute working out how to get to the open internet. They found and exploited a zero-day in that package proxy, escalated privileges, and moved laterally through OpenAI's research environment until they reached a node with internet access, then reasoned that Hugging Face probably hosted the ExploitGym solutions and chained stolen credentials with further zero-days into a remote code execution path on Hugging Face's servers. A later technical reconstruction from Hugging Face filled in a step: before touching their systems, the models took root on a third-party code sandbox out on the open internet and used it as the launchpad for the rest of the campaign. They had no instruction to do any of it. The evaluation asked for exploits, and cheating was simply the most efficient way to produce them.

" width="800" height="457">

The isolation technology held. What got them out was the single outbound path the environment was designed to permit, which is the same exception nearly every engineering team carves into an egress policy, because builds need packages. The models did not need a novel class of attack. They needed the one door that was already open, and a vulnerability in the software standing behind it. That proxy was internally hosted third-party software, a dependency somebody selected once and never had reason to revisit.

Hugging Face, for its part, could not tell what it was dealing with. Their write-up, published before OpenAI came forward, reads as an unattributed intrusion by a capable adversary, and the response was the one you would run for a criminal breach: eradicate the foothold, rebuild the compromised nodes, rotate the credentials, bring in outside forensics, notify law enforcement. A model overshooting its objective and a hostile actor with an agent framework produced the same telemetry and warranted the same response. Hugging Face reconstructed the sequence by running analysis over more than 17,000 recorded events, using an open-weight model on their own infrastructure, because the hosted frontier models they tried first refused the work and could not distinguish an incident responder from an attacker.

A misconfigured evaluation: the Anthropic incidents

Nine days after OpenAI's disclosure, Anthropic published the result of asking the same question about its own testing, and it makes the OpenAI case look almost reassuring. They reviewed 141,006 evaluation runs in which a model could have reached the internet and found three incidents where Claude did, from inside the environment of a third-party evaluation partner, and then compromised production systems at three unrelated companies. The earliest dated to April. There was no exploit and nothing escaped. The evaluation prompt told the model it had no internet access, a misconfiguration between Anthropic and the vendor meant that it did, and neither party knew until the transcript review turned it up three months later. Claude was running capture-the-flag exercises, which are an instruction to break into a system, so when the search led it to live hosts it treated them as targets in the game. In the worst of the three, it pulled application and infrastructure credentials and reached a database holding several hundred rows of production data. The two affected companies Anthropic was able to reach had not noticed, and found out when Anthropic called them.

Claude found setup instructions inside one fictional environment referencing a PyPI package that did not exist, and decided to publish one under that name. Doing so required an account, which required an email address, which required a phone number, which required money it did not have, and after several failed attempts to obtain funds it backtracked to a free email provider and uploaded the package. Its own reasoning had already flagged that publishing to the live registry would be a real-world attack and, in its words, "not the intended solution." It talked itself past that objection on two grounds. It did not recognize the certificate authorities securing its connections, and the system date read 2026, which it took as evidence that the environment was staged. The package was live for roughly an hour and ran on fifteen real machines. One was a security vendor's malware scanner, which installed it, executed it, and had its credentials shipped out to a collection point the model had set up.

Anthropic calls these harness and operational failures ahead of an alignment failure, and reports no evidence of a model pursuing a goal of its own. They are probably right, and the incidents still belong here. The model did the job it was assigned while holding a wrong belief about where it was standing. The first paper argued that a model cannot reliably separate instructions from data. These transcripts say it also cannot reliably tell a production system from a fixture, and that a model reasoning carefully about the question can reason its way to the wrong answer and carry on. Anthropic also notes that the safeguards shipped with their generally available models would have blocked the behavior, and that the evaluation ran without them deliberately, to measure the ceiling. Their findings are preliminary, and a third-party review with METR is being arranged, so some of the details may change.

Running untrusted code safely was already one of the oldest hard problems in computing, and a language model makes it worse, because the code is now written on demand, in volume, with no human reviewing it. The way to learn what your sandbox actually allows is to sit inside it and try to break out, which most teams never do.

Part V: Even a Managed Sandbox Needs Checking

The cloud providers will now sell you a managed sandbox for exactly this, and they are useful. A managed code interpreter hands you per-session isolation, identity-scoped access, audit logging, and a few network modes, without your team having to run its own isolation layer. If you do not want to operate a Firecracker fleet, that is a fair trade.

The word I would not take at face value is "isolated." A managed sandbox is isolated in the ways the vendor decided to isolate it, which are not always the ways you pictured when you read the marketing page.

With AgentCore, the distance between the documentation and the behavior got measured end to end. AWS offered a Sandbox network mode and described it as "an isolated environment with no external access." BeyondTrust Phantom Labs tested that claim and found the sandbox still resolved public DNS names, both A and AAAA records. Outbound TCP connections were refused, but the resolver kept answering.

Phantom Labs then built a working channel on top of it. They encoded commands into the octets of DNS A record responses, three base64 characters per answer, and sent output back out inside long subdomain labels, sixty characters at a time. The result was a bidirectional channel and a working interactive shell inside a sandbox configured for no network access. Using the execution role's S3 permissions, they listed buckets and read files out through the same channel, including a test file of customer records with names, phone numbers, and partial identifiers. The delivery vector was a poisoned CSV, with the injected instructions sitting in a cell where the agent would read them on the way to writing its Python.

AWS reproduced the report and acknowledged the impact, but its first decision was not to fix it. The behavior was intended, in AWS's view, because DNS resolution stays on so that S3 operations from the sandbox can resolve their endpoints. AWS scored the finding 7.5 on CVSSv3, down from the 8.1 in the original submission, and changed the documentation instead, from "an isolated environment with no external access" to "limited external network access." The service runs on Firecracker microVMs, and the compute isolation was never in question. What got rewritten was the sentence describing the network. That position did not survive publication. Disclosure ran from HackerOne report #3323153 in September 2025 to public release in March 2026, Palo Alto Networks Unit 42 published its own analysis of the same bypass a few weeks after that, and AWS then reversed course and closed the DNS channel. Exfiltration over DNS from a sandboxed interpreter is no longer possible (see Appendix A for the dated sequence of this disclosure).

A vendor and a customer can read the same word, "isolated," and take it to mean different things. The vendor's meaning is whatever the implementation happens to do, and a documentation edit is a legitimate outcome of a security report. The properties you designed against can change without anything in your own system changing, and nothing in your own monitoring would have told you it had happened.

DNS is the version of this everyone forgets. A sandbox can block every obvious outbound connection and still resolve hostnames happily. A hostname lookup carries more than enough room to smuggle a secret out a few characters at a time. Whatever a sandbox claims, the way to know what it permits is to run code inside it that tries to reach out, DNS included, and watch what leaves.

Anthropic's postmortem on its evaluation incidents arrives at the same place from the other direction. Their list of measures that would have caught the problem sooner opens with validating every internet access path before the runs begin and monitoring the network logs while they are going. The environment was believed to be sealed, that belief was written into the model's prompt as a statement of fact, and it went untested against the network for three months. A managed sandbox and a vendor-run evaluation range have this much in common, which is that somebody else's configuration sits between your workload and the internet. Anthropic and its partner each had a reasonable basis for thinking the other had closed the path.

Part VI: The Model Can Only Leak What It Can Reach

The cleanest way to keep the model from leaking data is to never hand it the data in the first place. The default in agent design is to pull everything a task might need into context and let the model sort it out. That is convenient but terrible for containment, because anything in the context window can be coaxed back out by an injection. Anthropic made the point well in their work on code execution: keep intermediate results inside the execution environment instead of running them back through the model, because the model can only leak what it has been given. When the model has to handle something sensitive, give it a token that stands in for the value and keep the value itself in the sandbox.

Some exfiltration never touches a network call. One well-documented trick gets the model to emit content that leaks the moment it is rendered. The classic version is a Markdown image whose URL points at the attacker's server with the stolen data tucked into the query string. Johann Rehberger demonstrated the persistent version of this against the ChatGPT macOS app. An injected instruction written into the model's long-term memory sent the user's inputs and the model's replies to an external site through image markdown, across future sessions, until it was found. The user's client fetches the image to display the answer, and the fetch quietly carries the data out. No one clicked anything, and rendering the answer was enough to leak it. That is why output from your own model has to be treated as untrusted before anything downstream renders it, runs it, or stores it, which OWASP files as Improper Output Handling.

So the outbound side comes down to a few habits. Keep sensitive data out of context when you can, and tokenize it when you cannot. Scrub whatever the model emits before anything renders or runs it, and require human approval on the actions that send data outside, because once it has left you cannot pull it back.

Part VII: Containment by Construction

Allowlists and resource limits are configuration, and configuration can be set wrong. There is a more ambitious line of research that aims higher, trying to make a leak impossible by construction instead of catching it at a boundary. It is still early, but things look promising. Here is where the research is headed.

The fullest version is CaMeL, from a group at Google DeepMind and ETH Zurich. It builds on Willison's dual-LLM pattern, which splits the model that can act from the model that reads untrusted input. A privileged model plans the task but never touches untrusted content directly. A second, quarantined model reads the dangerous text and is given no way to take a consequential action. Between them sits an interpreter that tracks where every piece of data is allowed to flow and enforces those rules outside the model, in plain code, so that data marked sensitive cannot reach an outbound channel no matter what an injected instruction tells it to do.

The reported numbers moved between paper versions, and both get quoted. The first version, in March 2025, reported solving 67 percent of tasks with provable security on AgentDojo, the prompt-injection benchmark of 97 user tasks and 629 security test cases across email, banking, travel, and workspace scenarios. The revised version in June, run with newer models, reported 77 percent, against 84 percent for an equivalent system with no defenses at all. The guarantee cost seven percentage points of capability in that experiment. The guarantee itself is strong, because an injected instruction cannot redirect the program flow at all. The control flow gets extracted from the trusted query before any untrusted data is read. What the agent gives up in exchange is improvisation. It follows a plan fixed before it read anything, which rules out the tasks where the right next step only becomes clear partway through.

Part VIII: How LuumenAI Approaches Containment

I want to be measured about where we are, the same way I was in the cost paper. Most of our security effort lives in the input layer, because that is where prevention happens and where we get the most for the work. The execution sandbox is built, and deliberately narrow, and we are still working through what we will let run inside it.

When LuumenAI needs to act on a system, it does not pick from a fixed menu of tools. It writes code and scripts against the APIs we expose, and that code runs in an isolated environment of its own, separate from the model. The environment can reach only the APIs we grant it, with no open network access and no path into the rest of our infrastructure. Intermediate results stay in that environment, and only the output the model actually needs comes back, which keeps large or sensitive payloads out of the context window entirely.

Scope gets compiled before the run starts. A control layer outside the model assembles four things into a single object for the run: which identity the work is being done as, which capabilities are available, which data is in scope, and which steps will need a person to approve them. That object is the ceiling on the run's authority. Everything downstream can narrow that authority, and any component in doubt fails closed. Nothing downstream can widen it. Compiling the scope up front is the property I care about most here, because the alternative is negotiating scope step by step while the model is working, and an injected instruction can take part in a negotiation.

Anthropic's list of fixes includes the observation that Claude would probably never have gone looking for the internet if the prompt had spelled out which systems were in scope for the exercise and which were not. We arrived at scope compilation for operational reasons, mostly to stop the model asking for capabilities mid-run, and I did not expect to see it turn up in a frontier lab's postmortem as the control that was missing.

A run that pauses does not inherit what it was granted. Work that stops for an approval and picks up later revalidates the whole contract on resume. Permissions change and sessions end, so a run that sat overnight waiting for a signature gets its scope rebuilt from whatever is true at the moment the approver signs off. OpenAI made a related point in the same week as the incident, observing that a model working over long time horizons can learn the blind spots of an approval system and route around it. Revalidating on resume was a decision we made for duller reasons, and that framing is a better argument for it than the one we had.

Environments are disposable and credentials arrive at execution time. Each execution gets a fresh environment that is destroyed when the run finishes. Credentials are injected when the code runs and are not stored in the environment, so there is nothing on disk or in an environment variable for a compromised process to read and carry forward.

Because we run on AWS, execution and access activity lands in the audit trail we already keep for compliance. We also keep a durable per-run trail on the product side, which records what was asked, what scope was granted, which capabilities ran, which approvals were requested and given, and what came back. Audit coverage, retention, and least-privilege review are tracked as compliance controls with named owners and review dates.

The first thing our sandbox ever blocked was us. We turned on the egress policy and immediately found that our own monitoring vendor was reachable while our internal APIs were not. Both were behaving as configured. The argument that followed was about whether to add an exception for the internal APIs, and we landed on an authenticated proxy instead, so those calls go through something that checks who is asking, and what for. The environment still has no direct route into our network. Around the same time our ticketing integration came up dead on arrival, because the sandbox vendor's default outbound-access tier did not allow the connection.

I read the OpenAI writeup with more interest than was comfortable. Their evaluation environment allowed one outbound path, to a proxy and cache for package registries, and that path was the way out. Our answer to the internal-API problem was a proxy as well. Ours authenticates every call and it is not a package cache. Those differences matter. It remains the one deliberate opening in an environment we otherwise describe as closed. When we review our egress list, that proxy goes first. It is the exception we argued ourselves into.

**We built a warm pool and decided against using it. **Cold-starting an environment for every execution adds latency, so we built a pool of pre-warmed environments to take that latency away. Then we costed it. Holding capacity warm meant paying for idle time, and a fresh environment per execution came out cheaper. That settled an argument the security reasoning had not settled on its own. It also left us with a property we can state to a customer without qualifying it. The pooling code is still in the repository behind a flag we have not had a reason to turn on.

Five reviews gate the execution environment. Isolation configuration, egress policy, secrets handling, audit logging, and cost controls each have a named reviewer, and any one of them can hold the whole component. We have not shipped it ahead of those reviews, and having watched a managed sandbox get its isolation claim rewritten in the documentation, I am not inclined to argue for the faster option.

I am not going to tell you this is finished. The sandbox does its job, which is to bound what a hijacked model can reach, and it gets a little stronger every iteration. It is also not the whole story, and a team that treats its sandbox as the whole story has the same blind spot as a team that only hardened its inputs, just pointed the other way.

" width="800" height="551">

Part IX: Bounding the Damage and Proving It

The case for all of this lands the first time you sit across from an enterprise or a government buyer. You cannot promise them your AI will never be compromised. That promise is not available to anyone, and a vendor who makes it is selling something false. What you can promise, and then show, is that a compromise stays inside limits you drew in advance. The model can be turned and still not reach data or systems outside its box.

" width="800" height="548">

A security reviewer can work with that, and it helps that the same controls throw off the evidence they are going to ask for anyway. Least privilege on the execution environment is your access-control story. The egress allowlist covers data-loss prevention. Resource limits cover availability, and the per-run trail of what the sandbox did is the record an auditor wants to see. For SOC 2, ISO 27001, and the higher bar that comes with federal work, a design that bounds the damage on purpose produces those records as it runs, instead of leaving you to reconstruct them the week before an assessment (see Appendix C for where each control lands in those frameworks). Anthropic found all three of its incidents by reading stored evaluation transcripts, months after the fact, which is a use for that trail nobody puts in the compliance matrix.

I had been thinking about blast radius as what a compromised agent can reach inside your walls. The evaluation incidents brought out a second direction: what it can reach outside them. Anthropic notes that the infrastructure those evaluations ran on was dedicated, with no access to their sensitive internal systems and no access to customer data. By the logic in this paper that is a well-drawn limit, and it held. Three other companies were breached anyway, because the one thing the environment could reach was the open internet, and the open internet is somebody's production system. Their summary is blunt: an advanced agent in an environment holding no sensitive information of its own is still a substantial threat.

For a buyer, that shifts the question slightly. Alongside what our agent could reach of theirs, there is what it could do to a third party from inside our walls, landing us in someone else's incident report by association. An egress allowlist answers both. That is a decent argument for keeping the list short.

Open-source isolation and execution tooling is where that caution bites hardest. It does not carry a compliance certification of its own, and self-hosting it makes its security posture your problem to own, which for a federal boundary is deliberate, sustained work that lands on your team.

Part X: Where There Are No Good Answers Yet

Containment sits in a slightly better spot than prevention, mostly because its problems are engineering problems and not open research questions. It still has limits.

Sandbox escapes are an arms race, and whatever boundary holds today, somebody is poking at it tonight. July settled that argument in an uncomfortable direction, because the environment that gave way belonged to a frontier lab and was described by its own operators as highly isolated. The tension between capability and containment never goes away either. Every limit that makes the agent safer takes something away from what it can do, and some useful agents need exactly the reach that containment is built to deny.

Managed sandboxes ask you to trust a boundary you did not build and cannot fully see into, and the AgentCore case shows that the isolation guarantee can be redefined in a documentation update while your architecture stays exactly where it was. No containment design can be proven against an escape that has not been invented yet, so the strongest claim available is that it held against the attacks you thought to run.

Then there is the case this whole series keeps circling back to. An agent that needs private data, exposure to the outside world, and the ability to act is holding all three legs of the trifecta on purpose, because that combination is the entire reason it is useful. For that agent, neither prevention nor containment is enough on its own, and the answer is to put a person on the decisions that matter. Calling that an architecture would be generous. It is an admission that some decisions do not go to a model yet.

Conclusion

The first paper was about what the model is allowed to read. This one is about what the model can do once it has read it. Put them together and they are a single approach: stop most of the attacks at the door, and build things so the ones that get through cannot do much. None of the containment work is exotic. It is a kernel-level boundary around model-written code, an egress allowlist you tested yourself, with the exceptions counted, no ambient credentials, ephemeral environments, model output treated as untrusted, a person on the irreversible actions, and a log of all of it (see Appendix B for that list expanded into a checklist).

" width="799" height="271">

Two of those are hard lines. Model-written code running in anything weaker than a kernel-level boundary is a finding, and it should be treated like one. And when a managed sandbox advertises complete isolation, that is a claim to test before you believe it.

The title of this paper is an instruction to assume the model is compromised, and after July I would widen it. A model does not have to be turned by anyone to act like an intruder. It needs an objective, enough capability to chase it, an environment that left one door open for good operational reasons, and a wrong idea about which of the systems around it are real. Containment has to hold against your own model doing the job you assigned it, and I am not convinced our own design fully accounts for that yet.

A compromised model is never harmless, whatever a vendor tells you. The most a good design buys you is a compromise that stays small and stays visible, which counts for a lot when the alternative is one nobody catches until the data is already gone. You will not keep every attacker out. The part you control is what it costs when one gets in, and it gets decided when you design the system, long before the incident.

There Is No Firewall for a Sentence

serene — Wed, 21 Oct 2026 20:02:58 +0000

Written by: Josh Greenwell Staff Applied AI Engineer

This blog article is a condensed version of our whitepaper on securing AI agents against prompt injection. The full paper includes the appendices on prompt-injection defenses and the complete reference list. You can download the original pdf format here.

Introduction

When we started seriously hardening LuumenAI, the question I kept turning over was some version of "how am I going to secure this thing." For a while I treated it like a normal hardening problem and went looking for the control I had missed. There mostly is not one. You cannot secure a language model the way that question assumes, and the sooner that sinks in, the sooner you can work on the parts that are actually fixable.

LuumenAI is the intelligence behind our observability and automation platform for SAP and ERP environments. It is a copilot for the engineers who run those systems. It reads alerts, pulls documentation, looks at incident history, and it can interact with terminals and run commands to help carry out a fix. Doing that means reading a lot of text that originated somewhere we do not control, like monitoring output, ticket descriptions, log lines, and knowledge base articles. That is normal for this kind of product, and it is also where the trouble starts.

A language model does not have a reliable way to tell where its instructions end and outside text begins. It reads the operator's prompt, the user's request, and a pasted log line as one continuous stream, and it weighs them all as input. If an attacker can get text in front of the model, that text carries roughly the same authority as anything else in the window. This is not a bug we can file and fix. It is how the models work right now.

You control what the model is allowed to read, and you control what it is allowed to do once it has read it. This first paper covers keeping bad data away from the model and limiting its options before and during a run. A second paper covers what happens after the model acts, where containment and sandboxing come in. That side of the problem is the less settled one across the industry, and it gets its own paper.

Part I: Why AI Security Is Different

A single email that steals your files

In June 2025, researchers at Aim Labs disclosed a flaw in Microsoft 365 Copilot that they called EchoLeak (CVE-2025-32711, rated CVSS 9.3).

The attack was an email. The target did not click a link, open an attachment, or reply. The email carried instructions written for the AI rather than for the person. Later, when Copilot did its ordinary job and pulled that email into context to help the user, it read the attacker's instructions and acted on them, gathering sensitive content and sending it back out through a path that looked legitimate. Microsoft had real defenses in the way of this, including a classifier meant to catch injected prompts, link redaction, and a content security policy, and the attack got past all of them. Aim Labs called the underlying failure an "LLM Scope Violation," meaning the model reached past what it was ever supposed to touch because a piece of text told it to.

Microsoft fixed it on their side and customers had nothing to do. That fix closed this one instance and did nothing about the pattern behind it, which carries over to any assistant that reads untrusted content and can act on it. Today that is most of them, ours included.

The same shape fits LuumenAI without much translation. Picture an engineer who asks the copilot to triage a monitoring problem. To do that, the copilot reads the alert the way it reads any alert, the title, the description, the affected systems, and whatever notes have piled up on the linked incident. Now suppose one of those fields is not an ordinary alert. Suppose the description of a monitoring problem in a tool like Dynatrace, the body of a ServiceNow incident, or a comment someone pasted onto a ticket contains a few sentences written for the model rather than for the engineer, telling it to pull a set of records, run a particular command, or quietly note something in its own memory for later.

None of that text came from us or from the engineer. It arrived from a monitored system, a ticket queue, or whoever was able to open a ticket in the first place. When the copilot reads it in the course of doing the job it was asked to do, that instruction lands in the same window as the engineer's request and carries the same weight. No single field here is exotic. A SAP estate emits thousands of these a day, most of them machine-written, and a copilot that helps with incidents has to read all of them. The malicious one looks exactly like the rest right up until the model acts on it.

What makes this unlike most security work is that the payload is ordinary text. Nothing is malformed. There is no buffer to overflow and no input check that got skipped. The malicious instruction can be written a thousand ways and can sit inside content that is supposed to be there, so signature matching and pattern rules do not catch it the way they catch a known exploit.

What prompt injection actually is

When you build an AI feature, you write a set of instructions for the model, usually called the system prompt. Then you feed it the user's request along with whatever supporting data the task needs. All of it, your trusted instructions and the untrusted data, goes to the model as one block of text, and the model has no dependable way to know where your authority stops and the outside world starts.

Anyone who has done web security has seen this exact failure under a different name. SQL injection works the same way: attacker-supplied data gets folded into a command, and the database runs the part it was never meant to. Prompt injection, a term Simon Willison coined in 2022 (see Appendix A for who he is and why his work anchors so much of this), is that same mistake moved into plain language.

Willison keeps drawing a line that is worth repeating. Jailbreaking and prompt injection are not the same thing. Jailbreaking is talking a model into violating its own safety training. Prompt injection is getting a model to follow instructions hidden in data it was handed to process. A team that tested for jailbreaks and thinks it is covered has tested the wrong thing.

The difference that matters is that SQL injection has a fix. You use parameterized queries, the data never gets treated as command, and the problem closes. Nothing like that exists for a language model. You can ask it nicely, you can format the untrusted text, you can train it to be suspicious, and all of that helps a little and none of it is reliable. Several research groups are working on real fixes, and none has fully landed yet. Appendix B sketches the most promising directions and links the papers.

Why "99 percent effective" is a failing grade

Most security controls are allowed to be good without being perfect. A spam filter that catches 99 percent of spam is doing its job, because spam is high volume and noisy and catching most of it removes most of the harm. Detectors get graded on a curve because the threat is statistical.

Prompt injection is not statistical in that way. An attacker is not firing a thousand random attempts and hoping. They sit with your product, rephrase, watch what gets through, and they only need the one phrasing that works. After that they reuse it. A filter that stops 99 of 100 injection attempts hands the attacker the hundredth, and the hundredth is the whole game. That is why every detection layer in the rest of this paper comes with the same caveat: it helps, but it is not the thing to count on.

Part II: The Lethal Trifecta

The framing I keep coming back to is Simon Willison's lethal trifecta. An agent becomes structurally exploitable when three things are true at the same time:

It can reach private data.
It is exposed to untrusted content.
It can send information to the outside world.

It helps to picture a three-legged stool. Each of those is a leg, and the stool only stands when all three are under it. An agent that has all three is a stool solid enough for an attacker to climb up and sit on, and from here on I will call them the three legs of the trifecta.

EchoLeak had all three legs. Copilot could read the user's mail and files, it ingested an attacker's email, and it had a way to get data back out. Hold all three at once and an attacker who lands a single injection can make the agent work against its own user, using that user's permissions. The injection in EchoLeak was not sophisticated. It did not have to be, because the three conditions were already lined up for it.

Each leg also shows up in the OWASP Top 10 for LLM Applications, the field's running list of what goes wrong. The untrusted-content leg is the one prompt injection (LLM01) exploits. The ability to act, pushed too far, is Excessive Agency (LLM06). And an agent that can be driven to consume without limit is Unbounded Consumption (LLM10). The trifecta is a way of seeing how those named risks combine on a single agent, instead of treating them as separate findings.

You do not take away a leg with a better prompt, because the model still reads everything in one stream. You take one away by removing the condition or constraining it hard enough that the agent cannot be steered into using it. A stool missing a leg will not hold anyone's weight. Different products give up different legs. You can wall off the private data, you can keep untrusted content from reaching the model, or you can cut the path data can take on the way out.

For LuumenAI this paper focuses on the first two legs. We control untrusted content where it enters, and we scope what private data the model can reach through authorization. The third leg, the outbound path, is its own subject and gets the next paper. We start here because the strongest and most reliable controls available today sit before the model runs, and that is where we have put the most work.

In late 2025 Meta published the "Agents Rule of Two," which says an agent should hold at most two of the three legs in one session without a human in the loop. Willison endorsed it, calling it about the best practical advice available, given that there is still no dependable defense against injection. Meta was also clear that plenty of useful designs do not fit the rule and that following it is no guarantee.

We use it as an operating default. An agent in LuumenAI is built to hold no more than two legs on its own, and when a task actually needs all three, the third leg is not pulled automatically. A person reviews and approves the consequential step before it runs. That checkpoint is not a fallback for when the other controls fail. It is a designed part of how high-risk actions work, because some decisions should not be handed in full to a model that a sentence can fool.

Part III: Defending the Model, Not the Machine

Most security writing about agents is really about sandboxing the code the model produces. That work matters, and the second paper is all about it. It also solves a different problem than the one here. A sandbox stops the model from harming the systems around it. It does nothing about a malicious instruction reaching the model in the first place. We needed the second thing, and it lives in a layer that runs before the model does.

We call that layer the pre-runtime. It sits in front of whatever agent framework is doing the actual work, and the framework underneath is something we treat as replaceable. Most of our security investment goes here, because it is the last place we have full, deterministic control before the model starts making probabilistic decisions.

Authorization context

Before a request reaches the model, we attach what we call authorization context: the organization, workspace, user, and role the request actually belongs to, among other things. What matters though, is the source. It comes from the user's authenticated session, not from anything the client puts in the request body. A user can claim to be an administrator, but the session token is what decides whether they are one.

This is aimed at the confused deputy problem, where an agent with broad legitimate access gets talked into using that access for someone else. If the agent's sense of who it is acting for comes from a verified session rather than from text in the conversation, an injected instruction cannot quietly promote itself. The model is working inside limits that were fixed before it read a word. We also keep the sensitive parts of that identity out of the model's context window, since anything sitting in the window can be coaxed back out.

Concretely, the model gets a small, redacted block of situational facts, enough to be useful and nothing that would matter much if it leaked. It sees the workspace it is working in by name, which features that workspace has turned on, the user's role and timezone, the organization's name, and a short set of remembered facts about the conversation so far. What it does not see is the part that decides anything: the session token, the raw permission and entitlement lists, the full set of client IDs the user can reach. Those stay in the authorization context that the surrounding code enforces, not in the prompt the model reads. The block the model does get is labeled for what it is, descriptive context rather than a grant of authority, so that even if the model treats it as gospel, that still cannot let it do anything its enforced scope would not already allow.

Guardrails

We run a guardrails layer that pairs our own deterministic input scanning with NeMo Guardrails. The deterministic scanner runs on every request and does not get to be skipped. NeMo Guardrails sits on top of it as a configurable rail engine for the checks that benefit from being tuned per deployment, and it can be set to fail closed, so that a rail failure blocks a request rather than letting it through. Between them they check input on the way in, constrain how a conversation can proceed, govern what retrieval is allowed to return, and filter output on the way out. In practice that is a set of rules about which topics and actions are allowed and what happens when something disallowed shows up, whether it gets refused, stripped, or handed to a person.

This is where the 99 percent problem from Part I lands. Some of these checks are deterministic and some are classifier-based, and both share the same limit. A determined attacker eventually finds the phrasing that no rule and no classifier has seen yet. We use this layer to catch the loud, known-bad cases cheaply and consistently. We do not, however, let it be the only thing standing between an attacker and anything that matters.

At the category level, the rails cover a handful of recurring shapes rather than a list anyone could read off and route around. On the way in we look for the obvious manipulation patterns: text trying to override the operator's instructions, text fishing for the system prompt or the names of internal tools, text trying to talk its way past an approval gate, requests to read secrets or credentials, and the small set of catastrophically destructive commands that have no place in a chat turn. We also flag attempts to forge approval or audit state, because a convincing fake that claims something was already approved is its own kind of attack. Tool and terminal output gets its own handling and is marked untrusted on the way back. The rest of the rails sit at the other edges of the conversation, governing what retrieval may return, how a conversation is allowed to proceed, and what the final response is checked for before it leaves. Whatever is caught here is caught cheaply, and nothing downstream is allowed to assume the catch happened.

Content wrapping and risk levels

When untrusted content has to go to the model, we do not paste it in raw. We mark it as untrusted, tag it with its source, and attach a risk level that the rest of the system reads when it decides how to handle the content. Marking it does not force the model to behave. It tilts the odds, and it gives everything downstream of the model a clear signal that this block is not to be trusted on its own.

We got this idea from Microsoft's research on spotlighting, which describes a family of techniques for exactly this: delimit untrusted content clearly, mark it so the model can tell it apart, and transform it so embedded instructions are less likely to read as commands. It builds on the older dual-LLM pattern, where the model that can take actions never sees raw untrusted text and a separate, walled-off model does the reading. What spotlighting and the dual-LLM pattern have in common is that they stop expecting the model to sort trusted from untrusted on its own, and do that sorting for it instead.

Intent routing

We route requests by intent, and routing decides which tools are even on the table for a given request. A request to read documentation does not get handed the tools that change a system. Shrinking the set of reachable tools shrinks the set of actions an injection could trigger if it slipped through everything upstream. Routing usually gets discussed as a way to save cost or improve quality. The security value is that it keeps the model's hands off tools it has no business using for the task in front of it.

Contracts

Every path that carries data toward the model goes through a contract. By that I mean a validated, structured interface, not an open pipe. There is a contract on the input before the model runs, and there are contracts on retrieval and on memory, so that what comes back during a run has a known shape and known limits. A retrieval call does not get to dump whatever it matched straight into the window.

This is the piece most specific to how we built LuumenAI, so I will be plain about it. The authorization context, the guardrails, the content wrapping, the intent routing, and the contracts are not separate features that happen to sit near each other. They are one layer, which we call the pre-runtime, and its job is to settle everything that ordinary code can settle before the model gets to make any call of its own. By the time a request reaches the model, that layer has already worked out who the request belongs to and what they can touch, which tools are even on the table, what the model may retrieve and remember, and whether the step at the end is going to need a person to sign off.

We did not start here. The first version of LuumenAI handed the model a large pile of tools and a lot of latitude and trusted it to make good choices, which is roughly what every early agent does. Walking that latitude back, a piece at a time, is most of how we arrived at the pre-runtime. The model no longer gets an open connection to our systems. It gets a prepared run with the risky options already taken off the table, and the agent framework underneath is something we treat as swappable.

The reason almost all of the investment goes into this one layer is timing. It is the last point in a request where the behavior is fully deterministic. Everything past it runs through a model that a well-written sentence can talk into the wrong thing, and the move that has held up for us is making sure the options that would do real damage are gone before the model ever gets the chance.

Part IV: Controlling What the Model Can Do

Keeping bad input out is half of this paper. The other half is limiting what the model can do once it is running, because a model that has not been injected at all can still be talked into a bad call.

Tool calling

The worst version is the model being steered into calling a tool destructively, or into pulling sensitive records and sending them somewhere. Willison's framing, which I think is right, is that agent security is mostly API security. The protections you can rely on are the ones enforced at the tool's own boundary: each tool scoped to the narrowest permissions that let it do its job, and the expensive or irreversible calls gated behind a human. You do not get those guarantees by trusting the model to be careful.

MCP

The Model Context Protocol is mostly a way to hand tools to a model, so every tool-calling concern above applies to it directly. It also introduces a new one. Because a tool's definition, its description and parameter docs, is text the model reads, the definition itself becomes a place to hide an injection. Researchers have shown that every field of a tool schema can carry an instruction. On top of that, third-party MCP servers are a provenance question, and a tool definition can change after you approved it.

In LuumenAI we keep this surface small on purpose. We register MCP servers only from an explicit allowlist, which today holds a single read-only documentation source, and we filter the tools those servers expose down to read-only, so write and destructive operations are excluded before the model ever sees them. The tool definitions we do accept we treat as untrusted until checked, prefer pinned and signed, and vet the way we would vet any other dependency we pull in.

Sub-agents

A sub-agent is not just another tool, and treating it like one causes trouble. It is a second model with its own context and its own access. The new risk is that one agent's output becomes another agent's input, and the receiving agent tends to trust it because it came from inside the system. If the first agent was processing untrusted content, an injected instruction can ride its output into the next agent wearing a trusted badge. We treat inter-agent output as untrusted by default, and we scope each sub-agent to only the access it needs, so that compromising one of them does not give an attacker the others.

Memory

Memory makes all of this last longer. A normal injection is gone when the conversation ends. A poisoned memory entry can sit for sessions and fire later under the right prompt. This is not hypothetical. The security researcher Johann Rehberger has demonstrated persistent memory poisoning against shipping assistants, including planting instructions in ChatGPT's long-term memory and in Google's Gemini that survived across separate conversations. Because we let the model use memory, memory goes through a contract too, with rules on what can be written and read back, so the long-term store does not become a place to leave instructions for a future run.

Part V: Cleaning What Goes In

Everything above assumes the data reaching the model was not poisoned at the source. Files need their own handling, because a document is an easy place to hide an instruction.

When LuumenAI takes in a document, it does not trust it on arrival. The file goes through processing before any of its content can reach retrieval or the model, and the higher-stakes material gets both automated checks and human review. This is the data poisoning problem OWASP lists as LLM04, along with the vector and embedding weaknesses listed as LLM08.

The attack is quieter than a live injection. Someone who gets a malicious document into your knowledge base does not need to touch a conversation, because the model will retrieve the poison later and treat it as trustworthy. PoisonedRAG, a 2025 research result, showed that planting a handful of crafted documents in the collection of files a system later searches is enough to bend the model's answers. So, the checking happens at ingestion, before anything joins that collection and inherits the trust the model gives its own retrieved knowledge.

Here is what that processing means in practice. A file does not get to skip the line. Before any of its content can be embedded into the searchable collection or shown to the model, it has to come back clean from a malware and integrity scan, and until it does, the content is held back. The model is told only that a file exists, with explicit instructions not to guess at what is in it.

Alongside that, we record provenance for every ingested document: where it came from, a checksum of its contents, a version, and the workspace and roles allowed to retrieve it. The document then moves through visible ingestion states, parsing, then chunking, then embedding, then indexing, rather than appearing in the index all at once, and higher-stakes material gets human review before it is allowed to join the collection at all.

None of this is a content filter that reads the document and decides whether the prose is trustworthy, which is exactly the judgment we already said the model cannot reliably make. It is a set of gates on provenance and handling, so that the trust a retrieved document inherits is trust we can account for.

Part VI: Securing the User, the Workspace, and the Organization

The pre-runtime pays off most visibly for an enterprise or a managed service provider. Because every request is anchored to a verified authorization context, the same enforcement runs at three nested levels: the individual user, the workspace, and the organization that contains them. For an MSP, a workspace is usually one of their clients.

An MSP running many clients on one platform cannot have one client's data or actions leak into another's, and trusting the model not to confuse them is not a control. Because identity and scope are settled before the model runs and then enforced through contracts and least privilege, the separation holds at each boundary rather than only at sign-in.

A useful side effect is that the same design creates the evidence an auditor needs. The authorization context is access-control evidence. The contracts, guardrail policies, and audit logs cover most of the data-handling and change-management questions a SOC 2 Type II or ISO 27001 review will raise, and they come out of how the system runs rather than from a document written afterward.

The short version on certification is this: the architecture is built to produce the evidence these regimes ask for, while the certifications are a platform and program matter rather than something the AI runtime claims on its own. The runtime emits structured audit events for the decisions auditors care about: who a request was authorized as, which tools were selected, what was approved or rejected, and what was retrieved and cited. The authorization context doubles as access-control evidence. Mapping those signals to specific SOC 2 Type II or ISO 27001 controls, storing them durably, and producing auditor-facing reports belongs to the broader Luumen platform and its compliance function, not to the model-facing code.

For federal work, the design carries a distinct deployment edition and is built to extend toward a FedRAMP-oriented posture, enforcing the same identity, scoping, and approval boundaries at a higher bar. That is a direction the architecture supports, not a certification already in hand.

There is one limit worth naming for teams headed toward federal work, where the bar is highest. Open-source security tooling does not bring a compliance certification of its own. The moment you self-host it, its security posture becomes your infrastructure's responsibility, which is a deliberate undertaking rather than a checkbox.

Part VII: Where There Are No Good Answers Yet

Prompt injection has no complete fix, and nothing here changes that. The whole pre-runtime reduces risk and narrows what a successful injection can do. None of it gives the model the ability to tell instructions from data, because that ability does not exist yet. Indirect injection through retrieved content is built into retrieval, which is to say built into anything useful. The tension between capability and security never settles, because the combination the trifecta describes, private data plus outside input plus the ability to act, is exactly what makes an AI assistant useful in the first place. Every condition you take away to make it safer also takes away something it could do.

There is also the half this paper set aside on purpose. Controlling the input perfectly still leaves you with whatever the model does next. It generates output, calls tools, writes code that something runs. Bad data kept out of the model is progress and not safety. Containing the model after it acts, through sandboxing and isolation, and limits on where data can go, is the harder and less settled half, and it is the second paper.

Conclusion

You cannot make the model itself safe. It reads everything in one stream and it cannot be trained into reliable suspicion, so you work on the two places you do control: what reaches it, and what it can do with what it reaches.

The input side comes down to a handful of moves. Check every agent for the lethal trifecta, and require a person on any agent that has to hold all three legs. Anchor every request to a verified authorization context that comes from the session and never from the client. Mark untrusted content and give it a risk level so the model treats it as data. Run guardrails, and remember they are additive. Use intent routing to keep tools out of reach when the task does not need them. Put a contract on every path into the model, retrieval and memory included. Clean what you ingest, because poison in the knowledge base is poison the model will trust later. Scope tools and sub-agents tightly, and treat one agent's output as untrusted input to the next.

Two hard rules sit underneath all of that.

Treat guardrail detection as additive only, because its catch rate against real attacks is never 100 percent.
If you cannot establish where a piece of ingested data came from, treat it as untrusted.

None of this makes the model trustworthy, and I would not want to leave you thinking it does. It keeps a model you cannot fully trust from doing much harm when it is wrong or gets turned. For an agent in production today, that is about as far as honesty lets you go. The input is half the problem. The next paper is the other half.

This article is a condensed version of our whitepaper on securing AI agents against prompt injection. The full paper includes the appendices on prompt-injection defenses and the complete reference list. You can view and download the original in pdf format here.