DEV Community

Sandscale
Sandscale

Posted on Originally published at sandscale.dev

Containers, gVisor or Firecracker: how to isolate code your AI agent wrote

Originally published on sandscale.dev.

An AI coding agent writes code and runs it, often with no one reading it first. The code may be buggy, may pull in a dependency nobody vetted, or may have been steered by text in a web page the agent read. You need a boundary around that code so a mistake or an attack stays inside it.

There are three common ways to draw that boundary: a plain container, a container run under gVisor, and a microVM such as Firecracker. They differ in what the untrusted code can touch directly, and the differences are easy to flatten into "VMs good, containers bad". That is not quite right, and this post tries to say why. It uses each project's own documentation, quotes only figures those documents publish, and says where we are stating an opinion.

The threat model

Start with what you are defending and from whom.

The attacker. Treat the code in the sandbox as hostile, even if the person who started the run is not. Typical ways it gets there: prompt injection through content the agent reads, a malicious or compromised package, or generated code that does something destructive by accident. The Firecracker design document takes the same stance toward guests: "all vCPU threads are considered to be running malicious code as soon as they have been started" (FCD).

What you want to protect.

  • The host machine and its other tenants (the other runs on the same hardware).
  • Credentials the agent holds, such as API keys and tokens.
  • Your network: internal services reachable from the sandbox.
  • Your bill and availability: a runaway process can consume CPU, memory or disk.

The ways out. gVisor's security documentation breaks an attack surface into classes. Two matter here:

  • The system API: the system calls, files and sockets the code is allowed to use. gVisor notes that bugs here "are also the most common form of exploit" (GVS).
  • The system ABI: hardware and privileged-code behavior that does not go through the intended API. gVisor cites a hypervisor flaw to make the point that "hypervisors are not immune to this vector" (GVS).

A third class is hardware side channels, which can affect any code on a machine, sandboxed or not. gVisor says it generally does not protect against them (GVS). The Firecracker paper likewise says anyone adopting Firecracker "must understand" side-channel attacks "and have plans to mitigate them" (FCP, section 3.4).

Isolation is only one layer of the answer. It stops code from reaching the host. It does not decide which hosts the code may call, which secrets it can read, or what an approved action is. Those need network policy, secret handling and review, covered near the end.

Layer 1: containers (shared kernel)

A container is an ordinary Linux process (or group of processes) with a restricted view of the system. The kernel provides the restrictions:

  • Namespaces wrap a global resource, such as process IDs, mount points or the network stack, so the process sees its own instance (man-pages, namespaces(7)).
  • cgroups group processes so that their use of resources such as CPU and memory can be limited and monitored (man-pages, cgroups(7)).
  • Capabilities split root's powers into pieces. Docker drops most of them by default (DKR).
  • seccomp filters which system calls the process may make (seccomp(2)). Container runtimes ship a default profile.

The key property: the container's processes make system calls into the same kernel that runs everything else on the host. Isolation is enforced by that kernel's own code. If a kernel bug is reachable through an allowed system call, the code in the container has the same route to the host as any other process. The Firecracker paper describes the same trade-off: containers sit directly on the kernel's sandboxing features, while VM-style isolation relies on the VMM and hardware virtualization (FCP, section 2).

None of this makes containers unsafe for every use. For code you wrote and reviewed, they are a good fit, with excellent compatibility and almost no overhead beyond the process itself. The risk grows as the code becomes less trusted, because the exposed surface is the whole Linux system call interface plus the files and sockets reachable through it. You can shrink it with a tight seccomp profile, dropped capabilities, read-only filesystems and user namespaces. gVisor's documentation points out that in practice it "can be extremely difficult (if not impossible) to reliably define a policy for arbitrary, previously unknown applications" (GVI), which describes the position of a platform running code it has never seen.

Layer 2: gVisor (a kernel in user space)

gVisor puts a second kernel between the application and the host. In its own words, it is "an application kernel that implements a Linux-like interface", written in Go and running in user space (GVI).

The main pieces:

  • The Sentry is the application kernel. It implements system calls, signal delivery, memory management, the threading model and more. When the application makes a system call, a platform redirects it to the Sentry, which handles it. gVisor states that the Sentry does not pass system calls through to the host kernel (GVS).
  • A restricted host interface. The Sentry itself runs with a tight seccomp policy and makes only a small set of host system calls. It cannot open new files or create new sockets directly for the most part (GVS). The Sentry is a pure Go binary with no CGo (GVS).
  • The Gofer is a separate process that manages the container's filesystem access; file operations go through it (GVS).
  • A platform intercepts the application's system calls. gVisor documents three: KVM, systrap and ptrace. Systrap uses seccomp's trap feature to hand each system call to gVisor, and replaced ptrace as the default in mid-2023. The KVM platform uses the KVM facility so the Sentry acts as both guest OS and VMM, but "there is no virtualized hardware layer" and the sandbox "retains a process model" (GVP).

So an exploit in the application first has to find a flaw in the Sentry's implementation of a Linux-like interface, and then escape a Sentry that is itself confined. That is defense in depth, with an independently written kernel in the way.

The costs, as gVisor describes them: the design "comes at the price of reduced application compatibility and higher per-system call overhead" (GVI). It "only implements a subset of the Linux syscall ABI"; the missing part is "mostly comprised of alternatives to existing syscalls that gVisor does support", and io_uring is called out as not fully supported (GVC). The project adds that compatibility issues are not common in practice for most workloads, citing its use in products like Google Cloud Run and DigitalOcean App Platform (GVC), and that "the only real way to know if it will work is to try."

For agents, the compatibility question is concrete. Agents run package managers, compilers, test suites, language servers, headless browsers and sometimes Docker inside the sandbox. Each of those uses a wider and odder slice of the kernel than a typical web service. It is worth testing your own toolchain, not assuming.

Layer 3: microVMs (hardware virtualization)

A microVM gives the code its own virtual machine: its own guest kernel, running on virtualized hardware. The host's KVM facility, with the CPU's virtualization extensions, enforces the boundary between the guest and the host.

Firecracker is a virtual machine monitor (VMM) that uses KVM to create and run microVMs. Its README describes microVMs that "combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers" (FCR). The design choices that matter:

  • A minimal device model. Firecracker "excludes unnecessary devices and guest-facing functionality to reduce the memory footprint and attack surface area of each microVM" (FCR). Guests get VirtIO network and block devices, a serial console and a small set of legacy pieces (FCD). The device emulation code is one of the main things a hostile guest can reach, so fewer devices means less to attack. gVisor makes the related point that a VM's security "depends to a large extent on what is exposed from the host kernel and userspace support code" (GVS).
  • One process per microVM, with threads for the API, the VMM and each vCPU (FCD).
  • The jailer. In production, Firecracker is started through a wrapper that puts the Firecracker process into a chroot, separate PID and network namespaces, with privileges dropped and a seccomp-bpf profile (FCJ; FCP section 3.4.1). The seccomp allowlist is documented in the jailer docs.
  • Host configuration still matters. The README states that the security of the microVMs "depends on a well configured Linux host operating system" and links a production host setup guide (FCR).

The Firecracker paper reports, for its minimal guest kernel configuration, "memory overhead of less than 5MB per container", boot "to application code in less than 125ms", and creation of "up to 150 MicroVMs per second per host" (FCP). These are the authors' published figures for their configuration in 2020, not a promise for any particular deployment, and the paper's boot measurement uses a minimal init and a single vCPU. The design doc gives a different configuration-specific figure: with a minimal Linux kernel, single-core CPU and 128 MiB of RAM, "5 microVMs per host core per second" (FCD).

What you give up: a guest kernel to boot and keep patched per image, a VMM and host kernel (KVM) that are themselves software with bugs, and a need for KVM on the host. That last point means hardware virtualization on the host. gVisor's documentation warns that nested virtualization "may bring challenges" for performance (GVI) and that its own KVM platform runs best on bare metal (GVP). Firecracker also supports a limited feature set. As of the 2020 paper, it says it does not emulate legacy devices or PCI, cannot boot arbitrary kernels, and could not boot Windows without significant work (FCP).

Firecracker's design document also notes that it "does not perform any network traffic filtering. All egress traffic from a guest is therefore considered untrusted, and should be filtered at the host-level" (FCD). The VM boundary does not give you a network policy.

Side by side

This table is qualitative on purpose. Where a number would go, it is in the sections above with its source.

Container Container under gVisor MicroVM (Firecracker)
Who handles the code's system calls The host kernel The Sentry, in user space (GVS) A separate guest kernel
What the code can reach on the host The host's Linux syscall interface, narrowed by seccomp, capabilities and namespaces The Sentry's implementation, and through it a small set of host calls The VMM's device emulation and KVM
Main bet on security Kernel correctness plus a tight policy A smaller, memory-safe reimplementation, confined Hardware virtualization plus a minimal VMM
Compatibility with arbitrary Linux software Highest A subset of the syscall ABI (GVC); test your toolchain A real Linux guest, so high; no GPU or exotic devices unless the VMM supports them
Per-call overhead None added Higher per-system call overhead (GVI) System calls stay inside the guest kernel; I/O crosses the VMM
Fixed overhead per sandbox Smallest Small, "process-like" resource model (GVI) A guest kernel and a VMM process per sandbox; Firecracker reports low figures for a minimal setup (FCP)
Host requirements Any Linux Any Linux; KVM platform optional (GVP) KVM (or a supported alternative)
Operations Image build and patching; policy tuning Runtime install; track compatibility Guest kernel and image management; host virtualization setup; VMM patching
Side channels Not addressed Not addressed (GVS) Not addressed by default; the paper tells adopters to plan mitigations (FCP)

gVisor's own FAQ gives the caution that applies to the whole table: one "should not assume that the mere use of virtualization hardware makes a system more or less secure" (GVS). A well-built user-space kernel can beat a sloppy VMM setup, and a minimal VMM on hardware virtualization is a strong boundary. The product of the design and its operation is what you get.

What none of these solve

Isolation answers one question: can the code reach beyond its box? It leaves several others open.

  • Egress. A sandbox that can call any address can leak data, fetch payloads, or hit your internal services. Decide what the box may reach, and filter at the host or with a policy layer. Firecracker says this itself (FCD).
  • Secrets. If an API key is in the box's environment, hostile code can read it. Inject secrets only where needed, for as long as needed, and keep them out of prompts and logs.
  • What the agent is allowed to do. An agent with a valid token can do real damage inside its own box's permissions. Approval steps for risky actions, and a record of every action, are separate controls from the isolation boundary.
  • Resources. Set CPU, memory, disk and time limits per run.
  • State between runs. A box reused across tasks or users carries leftovers. One box per run, destroyed afterward, is simpler to reason about.

Choosing

A rough guide, with opinion marked as such:

  • Containers fit code you or your team wrote and reviewed. Add a tight seccomp profile, dropped capabilities and a non-root user.
  • gVisor fits when you want stronger isolation than a container without requiring hardware virtualization, and your workloads fit within its supported syscalls. Test the exact tools your agents use first.
  • MicroVMs fit when the code is fully untrusted and varied, when you want a full Linux guest for compatibility, and when you can provide KVM on your hosts. In our view, this is the safer default for running arbitrary agent-written code, with the trade-off that you now operate guest kernels and a VMM.
  • Layers combine. Firecracker's jailer is itself a set of containers-style restrictions around the VMM, and gVisor stacks seccomp under its Sentry. "Which one" is rarely the whole decision.

Where Sandscale sits

Sandscale runs each sandbox as its own Firecracker microVM with its own kernel, one per run, started from the API or MCP. The sandboxes page describes the lifecycle (pause with memory kept, snapshots, stop), and the security overview covers the controls around the boundary: secrets in Vault, DLP, approvals and a hash-chained audit log. Agent policy describes how tool calls are checked.

We chose microVMs for the reasons above, and we also take the limits above seriously: a microVM does not by itself filter egress, protect secrets or decide what an agent may do. If you are comparing options, run your own toolchain through each one before you decide.


Top comments (0)