When an AI agent executes code, the useful state is often not just a directory on disk.
It may also include:
- an interpreter with modules already imported;
- a shell with environment variables and working-directory state;
- processes listening on local ports;
- caches held in memory;
- an application paused halfway through a task.
Starting another container from the same image does not reproduce that state. Copying the filesystem does not reproduce it either.
For reinforcement-learning rollouts, this distinction matters. If several trajectories must branch from the same point, rebuilding the environment for every branch wastes time and can change the state being evaluated. What we want is closer to fork() at the level of an isolated machine: create several sandboxes from one running environment, including memory.
This post explains how we approached that problem in OpenDsec, an open-source, self-hosted sandbox platform for code-executing agents and RL workloads.
Project context: OpenDsec is an independent implementation inspired by the architecture described in the DeepSeek Elastic Compute (DSec) paper. It is not DeepSeek code, not an official release, and is not affiliated with or endorsed by the paper's authors.
A sandbox has more than one kind of state
A useful mental model separates sandbox state into four parts.
1. Immutable image state
This is the base OCI image: the operating system, language runtime, libraries, and preinstalled tools.
It is easy to reproduce, but it only describes the starting point.
2. Mutable filesystem state
The agent may create files, install packages, clone repositories, or modify configuration. Overlay filesystems make this layer relatively cheap to capture and reuse.
OpenDsec can also package filesystem changes into a new image through pack_diff.
3. Memory and device state
This contains running processes, page contents, CPU state, virtual device state, open sockets, and in-memory caches. A filesystem snapshot does not preserve any of it.
For a microVM, this is the state that must be captured by the virtual machine monitor.
4. Control-plane state
The platform still needs to know who owns the sandbox, which compute node holds it, what resources it consumes, and how requests reach it.
A fast VM snapshot is not enough if restoring it requires rebuilding a large routing database or coordinating every control-plane component.
The lifecycle we need
OpenDsec exposes four related operations for Firecracker microVMs:
- Pause stops execution while preserving the sandbox state.
- Snapshot records memory and virtual machine state.
- Restore creates a runnable microVM from that snapshot.
- Fork produces one or more new sandboxes from a running source.
The important property is that a fork includes memory. If a Python process has already imported a large dependency, the child starts with that dependency present. If a service is listening inside the sandbox, the child contains the corresponding process state.
The source sandbox should also experience only a short interruption while a consistent point-in-time state is captured.
A simplified Python example looks like this:
import asyncio
from opendsec import DSecClient, DSecRunArgs
async def main():
async with DSecClient() as client:
sandbox = await client.run_sandbox(
DSecRunArgs(
backend="microvm",
image="root/py:3.12",
cpu_cores_limit=1,
memory_limit_mb=1024,
)
)
try:
await sandbox.run_shell(
"python3 -c 'import numpy; print(numpy.__version__)'"
)
children = await sandbox.fork(count=4)
for child in children:
result = await child.run_shell("python3 -c 'print(6 * 7)'")
print(result.stdout)
await child.stop()
finally:
await sandbox.stop()
asyncio.run(main())
Same-node and cross-node forks are different problems
A cluster cannot assume that every child will fit on the source node.
Same-node fork
When source and child stay on one compute node, memory pages can be shared copy-on-write. The initial child does not need a complete physical copy of every page. Pages are copied only after one side modifies them.
This improves both fork latency and memory density, especially when many rollout environments branch from the same prepared state.
The trade-off is locality: the children initially depend on memory managed by that node, and capacity on one machine is finite.
Cross-node fork
When placement selects another node, the destination cannot use the source node's local memory pages directly. It must obtain the snapshot, prepare the filesystem view, and restore the microVM locally.
In OpenDsec, the destination pulls the snapshot from the source edge. The control plane does not proxy the snapshot bytes. This keeps bulk transfer out of the stateless API layer and avoids turning the control plane into a data-plane bottleneck.
Cross-node fork is naturally slower than a same-node copy-on-write fork, but it allows a prepared environment to fan out across the cluster.
Placement cannot trust stale global state
Cluster schedulers work with observations that are already slightly old.
OpenDsec uses a stateless API server for authentication, quota checks, and initial placement. A watcher aggregates node health and available capacity. However, the selected node's edge daemon makes the final admission decision using its actual local resources.
This avoids a common race:
- the scheduler observes enough free memory;
- several requests choose the same node;
- the node's real capacity changes before all requests arrive.
The edge can reject an admission, allowing the caller to retry placement rather than overcommitting the node based on stale information.
Each sandbox ID also encodes its owning edge. Requests can therefore be routed without maintaining a central per-sandbox routing table. Restarting the API server does not interrupt microVMs already running on compute nodes.
Components communicate over gRPC with mutual TLS.
Runtime access must survive the lifecycle
Restoring a VM is only useful if the platform can still execute commands and transfer files inside it.
OpenDsec runs a small component called Aether inside each sandbox for the runtime channel. Chronus manages persistent shell sessions. Exec, files, HTTP, processes, and PTY access use the same runtime API across Firecracker microVMs, containers, and full QEMU VMs.
The backend changes, but the client-facing interaction model does not.
This also matters for compatibility. OpenDsec includes an E2B-compatible gateway, so existing E2B SDK programs can be pointed at a self-hosted deployment with environment variables. It is not yet complete: directory watching, signed file URLs, unauthenticated browser access to sandbox ports, and template building are currently unsupported.
What we measured
The following are OpenDsec project measurements, not numbers from the DSec paper.
Our test environment used two AMD EPYC 9Y24 processors (384 hardware threads), 1.5 TB of memory, NVMe RAID0, and bare-metal Ubuntu 24.04.
| Operation | Observed value |
|---|---|
| Firecracker microVM cold start | about 0.1 s |
| Restore from a memory snapshot | about 55 ms |
| Resume after pause | about 50 ms |
| Source VM interruption during fork | about 10 ms |
| Cross-node fork, 8 children | about 0.58 s |
| Concurrent restore, 64 children | about 0.2 s |
| Burst on one 384-thread node | 5,120 microVMs in 37 s |
These results describe our hardware and implementation, not a universal performance guarantee. Storage, image contents, memory size, networking, kernel configuration, and concurrent load all affect the outcome. The repository's specification documents the architecture and measurement context.
Isolation and networking still matter
Fast cloning must not weaken sandbox boundaries.
OpenDsec supports Firecracker microVMs, containers, and full QEMU VMs. Firecracker is the primary backend for workloads that need a stronger isolation boundary than a shared-kernel container.
Outbound networking is denied by default. Policies can allow specific domains, CIDRs, and ports. Our implementation uses nftables and a DNS proxy rather than the eBPF approach described in the paper.
Images can live on NFS, CephFS, or JuiceFS. This is another deliberate difference from the paper, which discusses 3FS.
The failure boundary is the host
Memory-preserving fork is useful, but it does not make a sandbox durable across every failure.
Today, if a compute host reboots, the sandboxes on that host are lost, including paused microVMs. The availability of the image store depends on the shared filesystem selected by the operator. QEMU PCI passthrough has also not yet been tested with real devices.
These limitations are important when deciding whether the platform fits a workload. A rollout environment that can be recreated is different from a long-lived production service whose in-memory state must survive a machine failure.
When memory-preserving fork helps
It is most useful when:
- many trajectories share an expensive initialization phase;
- state is held in running processes rather than only in files;
- reproducible branching matters;
- the environment must remain on private or offline infrastructure;
- hundreds or thousands of isolated copies must start together.
It is less useful when a clean process starts instantly, all relevant state is external, or a managed sandbox service is preferable to operating Linux hosts and KVM.
Closing thought
For agent infrastructure, “start another sandbox” and “continue from this exact execution state” are fundamentally different operations.
Images solve repeatable beginnings. Filesystem snapshots preserve disk changes. Memory-preserving fork creates branches from a running point in time.
OpenDsec is exploring that third operation as an open-source, self-hosted building block. The code, Quickstart, architecture documents, and known limitations are available in the OpenDsec repository. Feedback from people running agent or RL workloads is especially welcome.
Top comments (0)