Krishan Thisera

Posted on Sep 18 Originally published at linkedin.com

Designing for Assumed Compromise: Securing Autonomous AI Agents on Kubernetes

#ai #security #kubernetes #devops

A Defense-in-Depth Blueprint from Orchestration to Identity, Credentials, and Scale

An agent sandbox is a runtime instance, a container, microVM, or Kubernetes pod, built to run an autonomous AI agent's generated code and tool calls behind an isolation boundary. That boundary is meant to keep a compromised session from reaching the host machine or other tenants. How effectively it does that depends on which isolation technique backs it. Autonomous coding and browsing agents built on large language models now routinely generate and execute code, install packages, browse the live web, and handle credentials. Each of those actions carries only probabilistic guarantees about what the model will do, and that uncertainty is the problem an agent sandbox exists to solve.

That combination compounds two risks: arbitrary code execution and instructions an attacker can influence through prompt injection. Together, they have pushed the infrastructure conversation away from asking whether a container is hardened enough, toward assuming the container will eventually be compromised and designing every layer around that assumption.

This article covers that security posture, together with what it costs and how you govern a fleet of sessions rather than one, once you run it at scale.

Chapter 1: The Kubernetes-Native Orchestration Primitive

Kubernetes offers two dominant workload abstractions. Deployments manage stateless, interchangeable pods, and StatefulSets manage numbered fleets with generic identities. Neither fits an agent session, a singleton needing a stable hostname, persistent storage, and a lifecycle that pauses and resumes rather than only starting and stopping. Teams hand-assembled a StatefulSet of one, a Service, and a persistent volume claim (PVC). That combination had no shared lifecycle controller.

The Sandbox custom resource definition (CRD), maintained by kubernetes-sigs/agent-sandbox under Kubernetes SIG Apps, closes that gap with a declarative API for one stateful pod with a stable identity. Three CRDs extend it:

SandboxTemplate enforces security defaults.
SandboxWarmPool pre-provisions instances to cut cold-start latency.
SandboxClaim lets a framework request an instance without managing the template.

Chapter 2: Choosing an Isolation Technique

The Sandbox CRD manages a pod under the hood, so it inherits the standard Kubernetes runtimeClassName field for selecting the runtime, the same field any pod-based workload can set. runc, gVisor, and Kata Containers are the common runtime options it selects between, and seccomp layers a syscall filter on top of whichever one is chosen. Together the four trade isolation strength against cost differently. runc is the default low-level runtime that Docker and containerd use to create every container. It offers the least isolation of the four. It isolates a process using the host kernel's own namespaces and control groups (cgroups), so every container on a node shares a single kernel. That is fine for trusted code, but risky once an autonomous agent might run attacker-influenced instructions inside it.

seccomp (secure computing mode) is a Linux kernel facility that lets an operator allowlist or denylist the syscalls a container may invoke, rejecting anything outside that list. It adds no new kernel boundary. The host kernel still handles every allowed syscall directly, which keeps it cheap enough to layer onto almost any container. According to NVIDIA, it is one item in a defense-in-depth stack, suited to trusted internal automation rather than genuinely untrusted, agent-generated code.

gVisor adds the kernel boundary that seccomp does not. It interposes a user-space component called the Sentry between a container's processes and the real kernel. The Sentry intercepts every system call and handles it through a restricted reimplementation of the kernel surface. That shrinks the hundreds of syscalls a workload can reach down to a minimal, vetted subset. gVisor's overhead is commonly cited as 10 to 30 percent on I/O-heavy workloads, with little impact on compute-heavy work such as model inference. According to gVisor's performance guide, though, overhead varies considerably by platform and workload, and small, syscall-heavy operations can exceed that range. That makes gVisor a strong fit for CPU-bound tasks and a weaker one for constant file or network I/O.

Kata Containers provides the most isolation of the four, running a pod inside hardware-assisted virtualisation instead of a shared kernel. A hypervisor such as Firecracker creates a genuinely separate virtual machine with its own guest kernel, using the processor's virtualisation features rather than the host kernel's namespaces. Firecracker is a purpose-built hypervisor developed at AWS to back Lambda and Fargate, with reported boot times clustering around 100 to 125 milliseconds and memory overhead under 5 megabytes per microVM. Firecracker speaks its own API rather than Kubernetes', so Kata Containers closes that gap by implementing the standard Container Runtime Interface (CRI). A Kata-backed pod then schedules through the same kubectl apply workflow, with only a runtimeClassName field marking the difference. Kata supports Cloud Hypervisor as its default backend, with Firecracker and QEMU as alternatives, and boots in roughly 150 to 300 milliseconds. Because the isolation boundary is an entire separate kernel, an attacker who compromises the workload still has to escape both the guest kernel and the hypervisor to reach the host. That closes off namespace abuse and kernel-level exploits regardless of what the agent inside does.

None of this comes free. Even at 100 to 300 milliseconds, microVM boot time is measurable overhead once multiplied across every agent turn in a busy pipeline. Running Kata in production also means managing nested virtualisation support and a larger per-pod memory footprint than a bare container carries. seccomp and gVisor trade some of that isolation strength back for lower cost and higher throughput, which is exactly why the RuntimeClass mechanism matters. A single cluster can run a short-lived code interpreter session under gVisor, tolerating little latency budget but running cheaply at high volume, alongside a long-running, browser-equipped agent under Kata or Firecracker, through the same API.

Docker-in-Docker, isolated at the hypervisor level

Letting an agent build or run its own containers has conventionally meant Docker-in-Docker (DinD), a nested daemon in privileged mode, or the host's Docker socket mounted directly into the container. Both are well-known escape vectors, acceptable for trusted CI/CD pipelines but not for an autonomous agent whose build steps could be influenced by a prompt injected through a compromised dependency.

Docker Sandboxes removes both vectors instead of trying to harden them. Each agent runs inside its own rootless microVM with its own private Docker daemon, isolated at the hypervisor level rather than sharing the host's, with full privileges granted only inside that guest. In its "direct" mode, file changes still sync back to the host filesystem, so any git hooks, install scripts, or task configuration the agent touched need review before a human executes them locally. Closing this gap requires review discipline, not additional infrastructure. It means diffing hooks and checking scripts after each session.

Chapter 3: Threat Modelling for Assumed Compromise

The central threat behind every isolation decision is indirect prompt injection. Direct prompt injection, a user typing malicious instructions into a chat box, is comparatively well understood. Indirect prompt injection hides those instructions inside content the agent is expected to read as normal work. Vectors include a compromised file in a repository, a pull request description, a configuration file such as .cursorrules or CLAUDE.md, and a response from an MCP (Model Context Protocol) server. According to NVIDIA's AI Red Team guidance on sandboxing agentic workflows, an agent cannot reliably tell an operator's instructions apart from text inside a document it was asked to process. That means anyone who can put content in front of it gains a channel for influencing its actions.

Filtering the prompt cannot close this gap. Agentic tools execute arbitrary code by design. Once an action passes into a subprocess, the application has no visibility into or control over what that subprocess does. A compromised agent can also route around an allowlist by calling a restricted tool indirectly through one that is already approved. According to NVIDIA's guidance, only isolation with a kernel boundary of its own can reliably contain the risk, because containment then does not depend on the model behaving as instructed.

NVIDIA's guidance replaces a binary trusted-or-not judgement with four escalating rules:

Non-overridable enterprise denylist. An absolute floor of blocked operations, such as reads or writes to credential files or hooks, that neither a user nor the agent can approve away.
Workspace-scoped, allow-by-default access. Read and write permission inside the agent's active project directory, granted without approval for every action, since constant friction on routine work undermines the practice of approval altogether.
Narrow allowlisted exceptions. Specific operations outside the workspace that the agent's job still requires, such as a named SSH key for a legitimate git operation, approved individually.
Default-deny. Fresh manual approval required for everything else, never cached. A cached "yes" from an earlier legitimate action can be silently reused by an attacker-influenced action that looks similar later in the same session.

The same principle underlies all four rules. Each applies least privilege continuously, rather than only once at startup. A kernel boundary limits what a compromised process can touch. A network allowlist limits where it can send data. A credential proxy, covered in Chapter 4, limits what it can ever possess. Each is the same idea, enforced at a different layer.

Chapter 4: Network, Credential, and Identity Boundaries

Chapter 3 closed by naming three layers where least privilege gets enforced: a kernel boundary, a network allowlist, and a credential proxy. This chapter covers the latter two, network and credential boundaries, and adds a third: identity management. Together, egress control, credential handling, and identity management govern what a compromised agent can still do once it is talking to the outside world. That is where most of the actual damage from indirect prompt injection gets carried out.

Egress filtering is not content-level safety

Network egress control matters independently of kernel isolation, because even a sandbox that holds does not stop a compromised agent from sending data out through a connection it was always permitted to make. According to NVIDIA's guidance, egress filtering is a mandatory baseline control, not optional hardening. It calls for blocking outbound connections to unknown destinations by default. That boundary should be enforced through enterprise denylists, HTTP proxies, and DNS-level restriction, rather than trusting the agent's own code to self-police.

Domain-level allowlisting is not the same thing as content-level safety. Permitting a broad domain such as github.com allows access to any content hosted there. An attacker who can post data to an already-allowed gist or issue comment therefore has an exfiltration channel a domain filter will never flag.

Production implementations span a real spectrum rather than one fixed policy. At one end, a narrow allowlist permits only a model API endpoint and a package registry. At the other, full air-gapping removes all internet access for high-risk batch jobs with no legitimate need to reach it.

The same logic applies inside an organisation's network, not only at its external edge: a destination being reachable doesn't mean it's safe to trust. Zero-trust segmentation is what enforces that internally. An agent sandbox that can reach a production database or a deployment credential store has simply moved the exfiltration risk internally rather than removed it. In multi-tenant designs, that internal segmentation needs to be mandatory, not optional, on equal footing with external egress control.

Credentials that never enter the sandbox

The conventional pattern for handling secrets is to inject an API key into a sandbox as an environment variable through a Kubernetes secretKeyRef. That pattern has an irreducible weakness. The agent process itself can still read those environment variables at runtime.

Three failure modes follow from that fact alone, regardless of how well the surrounding sandbox is isolated:

The agent inadvertently includes the secret in a generated response or log.
A prompt-injected agent deliberately exfiltrates it through an otherwise-legitimate outbound call.
An adversarial input tricks the agent into printing or transmitting its own environment variables.

The credential-proxy pattern, implemented independently by projects including Infisical's Agent Vault, inverts this. The sandbox holds no secret at all. It routes outbound HTTPS traffic to a proxy, typically through the standard HTTPS_PROXY variable. That proxy terminates TLS and strips the placeholder credential the agent's request carried. It then injects the real credential from an encrypted store and re-establishes the connection to the actual upstream service. A session-level agent token issued at proxy handshake ties every request to a specific agent. That is what stops one sandbox from riding another's credentials, and it lets the proxy log, rate-limit, and revoke access mid-session without touching the sandbox itself. Because the sandbox never holds the real secret, none of the three failure modes above can occur. There is nothing for the agent to leak, exfiltrate, or be tricked into revealing.

Three approaches implement this pattern today, and they differ in what infrastructure they assume and how mature each one is. A team already running Istio can build it directly on EnvoyFilter and ext_authz. That path requires Istio's full service mesh to already be in place, and it inherits the fragility that Istio's documentation attributes to that escape-hatch mechanism. A purpose-built AI gateway requires no service mesh at all. agentgateway and Envoy AI Gateway both run as a standalone binary. Infisical's Agent Proxy is the third option, a vendor-managed proxy. Agent Proxy also requires no service mesh, but delegates secret storage to Infisical's own service rather than an operator-controlled vault. All three remain a single point of failure if the proxy itself is compromised.

No single mechanism covers identity completely

Three largely separate mechanisms address what identity an agent, or a sub-agent it spawns, actually carries, and none covers the problem completely. Bearer tokens and long-lived API keys assume a human authenticates once and performs a bounded set of actions. Handing an autonomous agent one broad token instead creates ambient authority, letting it exercise everything the token permits at any time, with no link between a specific tool call and a specific authorisation decision.

The problem compounds with sub-agents. Whichever of three common patterns a team picks carries its own risk:

Passing the parent's token down to a sub-agent erases the audit trail's ability to distinguish parent from child.
Issuing a new static credential per sub-agent multiplies the credentials that can leak.
Running a sub-agent unauthenticated is a compliance failure outright.

SPIFFE and its SPIRE implementation issue a short-lived, cryptographically verifiable identity based on runtime attestation rather than a static secret. SPIRE, however, expects every workload variant pre-registered ahead of time and delivers identity through a pull-based call, both of which sit awkwardly against a sandbox whose useful life might be seconds.

Cloud workload identity federation authenticates a pod to its cloud provider, but only coarsely. Google Kubernetes Engine (GKE) implements this through Workload Identity Federation, which makes every pod sharing one Kubernetes ServiceAccount cryptographically indistinguishable to Google Cloud IAM. Amazon Elastic Kubernetes Service (EKS) offers a choice between two mechanisms, each with a different tradeoff. IAM Roles for Service Accounts (IRSA) refreshes credentials without a pod restart, at the cost of AWS Security Token Service (STS) quota. EKS Pod Identity avoids that quota cost but can cache credentials for up to six hours before a permission change takes effect. Delegation-aware authorisation systems such as OpenFGA and SpiceDB fill the remaining gap, checking each individual tool call against an explicit graph of user and sub-agent permissions rather than authorising a session once at the start.

At the time of writing, stitching these three layers together remains real engineering work rather than a documented, turnkey integration.

Chapter 5: Scaling and Scheduling Agent Workloads

Chapters 2 through 4 covered how to secure a single sandbox. Running thousands of them at once introduces a different problem. It means avoiding a multi-second wait every time an agent starts a new turn, and stopping one tenant's burst of activity from crowding out every other tenant on the same infrastructure.

Balancing statefulness and disposability

An agent sandbox has to satisfy two requirements that pull in opposite directions. Many agent workloads are genuinely stateful. A persistent notebook kernel needs to preserve variables across cells, and a coding-agent workspace needs its file changes to survive across many separate tool calls. Both want a stable hostname and durable storage. At the same time, the premise of a sandbox, as distinct from a persistent server, is that it should be cheap to start per task and safe to tear down immediately afterward.

This differs from the ephemeral containers used in continuous integration and delivery (CI/CD). A CI/CD container starts, runs a predetermined pipeline of steps, and is discarded, with no expectation of resuming mid-pipeline. An agent sandbox typically needs to persist across many discrete tool calls over an extended, possibly paused, conversation. It might sit idle for minutes while a person reviews its output, then resume with the same files, the same running processes, and the same hostname. Chapter 1 covered the interface built for this, the Sandbox custom resource's declarative API for a single stateful pod with a stable identity. It exists because a hand-rolled StatefulSet, Service, and persistent volume claim had no shared lifecycle controller to reproduce that fidelity.

Warm pools and checkpoint-restore

Chapter 1 also introduced SandboxWarmPool, a pool of pre-booted sandboxes that lets a new session claim one instead of creating it from scratch. This brings allocation down to the millisecond range. A fresh pod with a hardware-isolated kernel boot (Chapter 2) costs seconds instead, an order of magnitude slower.

GKE adds a further refinement. Pod Snapshots is a checkpoint-and-restore capability that captures a running pod's full state and cuts startup time from minutes to seconds, including for GPU-backed workloads that are expensive to cold-boot. A companion capability, snapshot-pausing, reclaims an idle sandbox's compute without discarding its state the way a hard termination would.

EKS has no equivalent yet. AWS does support checkpoint and restore, built on Checkpoint/Restore In Userspace (CRIU) through the Kubernetes kubelet checkpoint API. Its purpose is different. It exists for forensic evidence preservation, capturing a suspicious container's state without killing it so an investigator can examine it later. This forensic checkpointing is not a substitute for GKE's cold-start capability.

One project outside AWS closes part of this gap today. Agent Substrate, a joint effort between Solo.io and Google, runs its own cross-cloud snapshot layer, independent of any single vendor's kubelet feature, coordinating full-state snapshots across an agent sandbox's suspend and resume cycles.

Quotas, tokens, and rate limits for a sandbox fleet

Kubernetes' standard ResourceQuota object sets a namespace-level ceiling on aggregate CPU, memory, and object counts, the same primitive any multi-tenant cluster already relies on. LimitRange complements it by bounding what any single pod or container inside that namespace can request, so one workload cannot claim the entire quota for itself. Neither one governs what a workload does over the network, which the egress allowlists from Chapter 4 control separately. Many of the threat categories catalogued by OWASP for agentic systems operate entirely within otherwise-permitted resource and network boundaries. That means an attack can satisfy both checks at once and still succeed. A compromised agent exfiltrating data through an already-allowed API call, for instance, breaches no quota at all. Quota and network policy alone cannot catch that, which is why agent workloads need a further, behavioural layer constraining the pattern of an agent's actions rather than only their volume.

This behavioural layer governs token consumption most directly, typically enforced at the same AI gateway introduced in Chapter 4. Envoy AI Gateway's QuotaPolicy and agentgateway's token budgets both implement it as a three-level hierarchy. An overall platform budget subdivides into per-tenant budgets, which further subdivide into optional per-agent budgets. The gateway rejects a request once its level's budget runs out.

Rate limiting for agent fleets operates at two distinct layers, and mixing them up causes real problems. One governs how many outbound calls an agent may make against an external service. The other governs how many sandboxes, and how much aggregate compute, a tenant may have running at once. Confusing them produces either over-restrictive throttling or under-enforcement. Over-restrictive throttling kicks in when compute quota looks tight but the real bottleneck is API calls. Under-enforcement lets a tenant within its compute quota keep overloading an external API for everyone sharing that connection.

The first layer sits naturally at the network and proxy layer. The same egress point that injects credentials in Chapter 4 typically enforces it too, so a platform already routing traffic through that proxy for credentials gets per-tenant throttling as an adjacent policy. The second layer sits at the scheduler and quota layer described above.

Agent workloads are bursty. A single user task can fan out into several rapid tool calls, or spawn sub-agents that act concurrently. A per-tenant limit checked independently by each agent cannot see the other agents' concurrent draws against the same shared budget. An external rate-limit service closes this gap. Envoy AI Gateway and agentgateway both call one to share a budget across replicas. The service checks and decrements one shared counter atomically, so concurrent agents see a consistent, current balance rather than a stale per-replica count.

What isolation actually costs

Chapter 2 covered isolation's overhead in compute and latency terms. What it costs in dollars is harder to establish. No source reviewed for this article publishes a like-for-like comparison of what a single provider charges for the same workload under runc, seccomp, gVisor, and Kata Containers. You cannot currently look up what stronger isolation costs while holding everything else constant. You have to benchmark it yourself.

Published pricing does make one part of the spectrum concrete, the deployment model. At a reference scale of 200 concurrently running sandboxes, a managed, shared platform lists monthly pricing around US\$7,200. A comparable footprint under a bring-your-own-cloud model costs about US\$2,060 (more than a threefold difference) for broadly similar isolation and capability. Most of that gap is operational and multi-tenancy overhead a fully managed platform absorbs on a customer's behalf, separate from the isolation technology itself.

Chapter 6: Benchmark Evidence and Attack Surfaces Beyond Code

Chapters 2 and 3 argued, from engineering principles, that hardware isolation (gVisor's kernel boundary, Kata's separate guest kernel) is necessary once an agent might execute attacker-influenced code. This chapter tests that argument against evidence, drawing on a 2026 benchmark measuring real escape attempts and an attack surface those technologies were never built to cover.

What SandboxEscapeBench found

SandboxEscapeBench tests whether a model with root access inside a container can find its way out to the host. It spans 18 scenarios across the orchestration, runtime, and kernel layers, each rated by difficulty. Frontier models reliably solved easy misconfigurations, such as privileged containers and exposed Docker sockets. On medium-difficulty exploits requiring genuine multi-step work, GPT-5 and Claude Opus 4.5 both scored around 0.40. On kernel-level exploits, every model tested scored zero.

The gap between model tiers was large. Claude Opus scored 0.49 overall against Haiku's 0.18, roughly threefold, and performance scaled with the compute a model was given. Plain Docker isolation is insufficient by default. Real deployments should be stress-tested rather than assumed safe. No novel vulnerabilities were used, only known misconfigurations and disclosed Common Vulnerabilities and Exposures (CVE) entries. This remains a single study, not yet independently replicated.

These results are worth reading together: escape success scaled with the compute a model was given, and every model scored zero on every kernel-level exploit. Together, they suggest the current zero-success ceiling reflects where model capability stands today, not a permanent limit. If that holds, three directions follow:

Hardware isolation stops being reserved for high-risk workloads and becomes the default for any untrusted one.
Credential proxying and egress filtering move from a feature teams opt into to one platforms ship by default.
Escape benchmarking becomes a routine part of due diligence, the way penetration testing did for cloud infrastructure.

The browser as a different kind of risk

Browsing agents introduce a second execution surface that Chapter 2's isolation technologies do not cover. The risk is semantic rather than technical. A malicious instruction hidden in a web page's rendered content does not need to escape a sandbox. It manipulates the agent's reasoning directly, the same indirect prompt injection risk from Chapter 3, with a page rather than a file as the vector. The stakes are higher because of what a browsing session carries, live authentication cookies and sometimes payment details, which a manipulated agent can leak through actions the browser was always permitted to take.

Browser isolation complements code-execution sandboxing rather than replacing it, typically through an HTTP-level policy layer sitting above the browser's existing process sandboxing. It remains younger and less standardised than code-execution isolation, with no equivalent yet of Chapter 1's RuntimeClass for an isolated browser session.

Chapter 7: Buy Versus Build

Everything from Chapter 1's orchestration primitive to Chapter 6's benchmark evidence describes a stack a team could build. Most teams do not build it themselves. They buy a hosted sandbox instead, from a provider such as E2B, Fly.io, Modal or Northflank. Each sells a scheduled, isolated, credential-brokered sandbox. None of them require the buyer to assemble the underlying cluster, isolation configuration, credential proxy and governance layer.

The isolation layer is not a differentiator among these vendors. Nearly every one uses microVM or gVisor isolation, the two strongest techniques Chapter 2 covered. The industry already treats both as sufficient for untrusted, LLM-generated code. Vendors compete instead on cold-start speed, session persistence, pricing and developer experience. That makes the buy decision simple for most teams. Vendors' published isolation stacks appear broadly comparable to what this article has described, at a fraction of the build cost. Building it yourself only becomes worthwhile at organisations already running Kubernetes at scale, or held to compliance requirements that rule out shared infrastructure.

Chapter 8: Conclusion

One fact underlies why agent sandboxes exist at all. A large language model can now execute code it wrote itself, in response to instructions nobody can fully verify. That same capability gives the agent its value and makes it dangerous.

Nothing this article has covered removes that risk. Every layer instead narrows it, each covered in an earlier chapter:

the orchestration primitive
the isolation technique
the threat model
network and credential boundaries
fleet scaling and governance
the benchmark evidence

None of these ideas were invented specifically for AI agents. Multi-tenant cloud security supplied the isolation techniques, and a decade of browser security research supplied the browser controls.

The result is defense-in-depth, a posture rather than a solution. It treats every layer as something that will eventually fail, and it builds each one so that failure does not cascade into whatever comes next.

Chapter 9: References

Chapter 1

Chapter 2

Chapter 4

Chapter 5

Chapter 6

Chapter 7

DEV Community