Why We Brought This Tool Into Our Lab
We care about sandbox costs because agent execution turns small unit prices into a material infrastructure bill. A coding agent does not run one command. It installs dependencies, retries failed tests, holds a workspace between turns, and sometimes waits while another model reasons. At volume, the lifetime of those environments matters as much as their CPU consumption.
Isolation is not a complete sandbox
Firecracker provides a KVM-backed isolation primitive, but a self-hosted sandbox still requires an execution service, security controls, and infrastructure operations.
Our question was simple: can we replace a managed sandbox bill with infrastructure we control without accidentally becoming a sandbox-platform company?
Firecracker supplies an important part of that answer. On the KVM path we evaluate here, it runs a guest Linux kernel inside a lightweight virtual machine. Its deliberately small device model reduces exposed functionality compared with a general-purpose VMM. We get a hardware virtualization boundary rather than asking untrusted programs to share the host kernel directly.
We do not get a complete agent execution service.
We still need image builds, a guest execution agent, admission control, workspace handling, deadlines, output limits, network policy, cleanup, and an authenticated customer-facing API. Firecracker's host API configures machines; it does not provide an equivalent of sandbox.commands.run().
Our evidence boundary matters: we reviewed the implementation-facing material and external execution accounts, but did not complete a local KVM deployment for this article. Our review environment lacked the necessary KVM access, binaries, images, and privileged networking controls. The walkthrough below is a reproduction recipe, with explicitly simulated output—not a claim that we measured boot latency or validated production isolation.
That distinction changes our recommendation. We consider Firecracker a credible primitive for a self-hosted sandbox, not a deployment we can certify from a successful API response. Our starting references are the Firecracker project, its quickstart, and its production host guidance.
Hands-On Walkthrough: Setup, Execution & Output
We separate the first experiment from the production design. The first experiment should prove that a trusted guest boots on the intended host. It should not accept customer code.
Establish the host and artifact contract
We require Linux with working /dev/kvm, a supported CPU architecture, and permission for the VMM identity to access KVM. A cloud instance being “a Linux VM” is insufficient: its provider and instance configuration must expose hardware virtualization.
Our initial inventory is deliberately mundane:
uname -a
uname -m
ls -l /dev/kvm
test -r /dev/kvm && test -w /dev/kvm
findmnt -t cgroup2
/usr/local/bin/firecracker --version
/usr/local/bin/jailer --help
We would run the permission checks as the intended service identity, not only as root.
We install Firecracker and jailer from the same pinned release, using the upstream release archive for the host architecture. We verify the release artifacts before installation and retain their hashes. We do not use an unversioned download or assume that a URL ending in a binary name exists for every release.
For the guest, our artifact contract is:
- A compatible kernel image for the selected architecture.
- An ext4 root filesystem with a working init process.
- Built-in kernel support for the root block device and filesystem.
- Serial-console support for the initial experiment.
- Virtio-vsock support if we later use vsock for execution requests.
- A versioned manifest tying the kernel, filesystem, VMM, and guest agent together.
Our build path would use a pinned Linux source tree and Firecracker's matching guest configuration as the kernel baseline. We would build the appropriate kernel target for the architecture, rather than assume one image format fits every host.
For the filesystem, we would construct a minimal distribution tree, install the interpreter and guest agent, add an init service, and package it into ext4. For example, once a trusted staging tree exists:
# Run in a dedicated build directory, not on a production sandbox host.
set -euo pipefail
test -x rootfs-tree/sbin/init
truncate -s 1G rootfs.ext4
mkfs.ext4 -F -d rootfs-tree rootfs.ext4
sha256sum vmlinux rootfs.ext4 > guest-artifacts.sha256
The 1 GiB file size is an example build choice, not a measured storage requirement. These commands package a tree; they do not magically create its init system or execution agent.
We would first boot upstream quickstart artifacts, then replace the kernel and filesystem separately. That keeps a broken image build from masquerading as a KVM problem.
Exercise the Unix-socket API
Here is our proposed trusted-guest smoke test, which we have not executed or validated against a pinned Firecracker API specification. Before running it, we would check the endpoints, payloads, and lifecycle sequence against the selected release and supply verified, compatible guest artifacts. We intentionally leave networking out.
This launches Firecracker directly, without jailer. We would never use this diagnostic path to run untrusted agent code.
#!/usr/bin/env bash
set -euo pipefail
FC="${FC:-/usr/local/bin/firecracker}"
KERNEL="$(realpath "${KERNEL:?Set KERNEL to a compatible kernel image}")"
ROOTFS="$(realpath "${ROOTFS:?Set ROOTFS to a trusted ext4 image}")"
WORK="$(mktemp -d /tmp/effloow-fc.XXXXXX)"
SOCK="$WORK/api.sock"
PID=""
cleanup() {
if [[ -n "$PID" ]]; then
kill "$PID" 2>/dev/null || true
wait "$PID" 2>/dev/null || true
fi
rm -rf "$WORK"
}
trap cleanup EXIT
test -r /dev/kvm
test -w /dev/kvm
test -x "$FC"
"$FC" --api-sock "$SOCK" >"$WORK/console.log" 2>&1 &
PID=$!
for _ in $(seq 1 100); do
[[ -S "$SOCK" ]] && break
kill -0 "$PID"
sleep 0.05
done
test -S "$SOCK"
put() {
local endpoint="$1" payload="$2"
curl --silent --show-error --fail-with-body \
--unix-socket "$SOCK" \
-X PUT "http://localhost/$endpoint" \
-H 'Content-Type: application/json' \
--data-binary "$payload" \
-o "$WORK/response" -w "$endpoint HTTP %{http_code}\n"
}
MACHINE='{"vcpu_count":1,"mem_size_mib":256}'
BOOT="$(python3 - "$KERNEL" <<'PY'
import json
import sys
print(json.dumps({
"kernel_image_path": sys.argv[1],
"boot_args": "console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda ro"
}))
PY
)"
DRIVE="$(python3 - "$ROOTFS" <<'PY'
import json
import sys
print(json.dumps({
"drive_id": "rootfs",
"path_on_host": sys.argv[1],
"is_root_device": True,
"is_read_only": True
}))
PY
)"
put machine-config "$MACHINE"
put boot-source "$BOOT"
put drives/rootfs "$DRIVE"
put actions '{"action_type":"InstanceStart"}'
sleep 2
tail -n 20 "$WORK/console.log"
printf '%s\n' "Start acknowledged; command readiness has NOT been verified."
Simulated successful API output:
machine-config HTTP 204
boot-source HTTP 204
drives/rootfs HTTP 204
actions HTTP 204
Start acknowledged; command readiness has NOT been verified.
A successful start response does not prove that init completed. A read-only guest also needs appropriate writable temporary filesystems for its services. We would inspect the serial log and then require an actual command response from the guest agent.
Add jailer before untrusted execution
Our production path would start with the jailer reference, matched to the pinned binary.
We need a unique instance ID, an explicit unprivileged UID/GID, a controlled chroot base, configured resource controls, and any required network namespace. Kernel, disk, and socket paths must make sense inside the jail. Host absolute paths from the diagnostic script cannot simply be copied into jailed API requests.
We would verify the resulting UID, namespace membership, cgroup placement, device access, and file ownership before admitting a job. Starting through jailer is a step in the security model, not proof that the whole model is correct.
What We Could Not Verify and Deployment Risks We Identified
Our environment lacked usable KVM and privileged host controls, so we could not validate boot, jailer behavior, or enforcement. We stopped there rather than replace those checks with a mock socket and call the result a sandbox test.
The remaining issues are concrete deployment risks we identified, not failures we personally reproduced.
A VMM API is not a command-execution API
We still have to build the guest agent. It must validate request lengths, manage subprocesses, separate stdout and stderr, enforce output caps, return exit status, and survive malformed requests.
We would run execution workers without guest-root privileges. We would also retain a host-side deadline that destroys the entire microVM. A compromised guest must not be able to negotiate away its termination.
Vsock can keep command transport separate from IP networking, but it does not make guest responses trustworthy. We would treat every returned byte as untrusted input.
Guest process limits and host process limits are different
We would audit this distinction first when testing resource controls.
A host cgroup's pids.max bounds host tasks, including VMM threads. It does not count every process running inside the guest. We need guest-side process controls as well.
Likewise:
- Guest RAM configuration is not the VMM's complete host memory budget.
-
memory.maxneeds allowance beyond configured guest RAM. -
cpu.maxconstrains aggregate host CPU consumption, not an individual guest subprocess. - Guest filesystem capacity, host disk usage, I/O, and log growth require separate controls.
We would not copy a “128 MB sandbox” setting into a 128 MB host memory ceiling and assume it works. Under load, we would inspect cgroup memory events, CPU throttling, host pressure, and guest failures together.
Networking remains our responsibility
A network namespace is not an egress policy. NAT is not an authorization boundary.
Our first untrusted execution profile would have no virtual network interface. If package installation becomes necessary, we would add a separately tested network path with explicit policy for:
- Host and private-network access.
- Cloud metadata endpoints.
- DNS and alternate resolvers.
- IPv4 and IPv6.
- Cross-sandbox traffic.
- Package mirrors and outbound proxies.
We would avoid the common quickstart-to-production shortcut of broad forwarding and masquerading rules. Those rules establish connectivity; they do not establish safe connectivity.
Snapshot restore requires compatibility and state-isolation tests
We have no local snapshot-restore measurement or verified restore sequence to publish. We would use the pinned release's snapshot documentation rather than improvise endpoints.
Our acceptance tests would cover artifact compatibility, backing disks, namespace and TAP reconstruction, stale connections, guest clocks, and readiness after restore. Snapshots containing credentials or customer state would be sensitive artifacts.
We would also test entropy and identity regeneration when cloning prepared state. We would not fork a memory image containing reusable secrets and assume each child becomes independent automatically.
Guest services can dominate startup
A small VMM does not guarantee a fast application environment. General-purpose init services, device waits, SSH startup, and runtime imports can dominate the path to useful execution.
Our workaround would be a minimal, observable boot chain—not blanket removal of randomness-related services. We would measure where startup blocks before changing entropy handling or weakening guest configuration.
Scale, Latency & Cost vs. Alternatives
We cannot publish a raw Firecracker p50 or p95 from this review. We have no completed local run.
In our managed sandbox comparison, we use 717 ms create and 662 ms resume as E2B's command-executed readiness baseline. These timings describe the managed service, not raw Firecracker boot latency; our local KVM reproduction remained unperformed.
We would reject a comparison that puts a local InstanceStart acknowledgment beside that command-ready measurement.
| Option | What we get | What we still own | Our decision criterion |
|---|---|---|---|
| Raw Firecracker | KVM-backed microVM primitive and host API | Guest agent, fleet, images, policy, quotas, cleanup | We need infrastructure control and have sustained demand |
| Managed sandbox such as E2B | Integrated execution service and lifecycle API | Application integration, permissions, usage controls | We need delivery speed and a smaller operations surface |
| Self-hosted container sandbox | Familiar packaging and orchestration | Shared-kernel risk, hardening, execution control | Workload trust permits that boundary |
| Additional sandboxing layer such as gVisor | A mediated syscall boundary | Compatibility testing, orchestration, policy | We need a different security/operations trade-off |
We would benchmark four paths separately: fresh boot, snapshot restore, warm-pool allocation, and a subsequent command in an existing guest.
We would start the timer when we admit a request and stop it when we receive a nonce-bearing command result. We would report p50, p95, failures, and timeouts under increasing concurrency. Artifact preparation and queue delay would remain visible rather than disappear outside the timer.
A warm pool can reduce allocation delay, but it consumes capacity before requests arrive. It is not evidence of a faster cold boot.
Our break-even calculation
We model the business decision as:
Managed monthly cost = billable sandbox-hours × effective hourly rate
+ plan fees
Self-hosted monthly cost = hosts + storage + network
+ engineering/operations
+ capacity reserve
Consider illustrative assumptions, not current vendor quotes:
- Three hosts at $250 each: $750/month.
- Storage, networking, and monitoring: $150/month.
- Twelve operations hours at $100/hour: $1,200/month.
- Total: $2,100/month.
- Managed equivalent: $0.08 per sandbox-hour.
The arithmetic break-even is:
$2,100 / $0.08 = 26,250 billable sandbox-hours/month
That is roughly 36.5 continuously occupied sandboxes over a 30-day month. It is not a capacity claim for those three hosts. We would still have to prove that the chosen CPU, RAM, I/O, and failure reserve can support the workload.
Ignoring labor lowers the apparent threshold to 11,250 hours. That is precisely how a self-hosting proposal becomes misleading.
We would also amortize initial engineering separately. Burstiness, idle sessions, retention, billing semantics, and negotiated pricing can change the answer substantially.
Our Final Verdict: When to Deploy, When to Skip
Our verdict: shortlist Firecracker as an isolation primitive; do not mistake it for a finished agent sandbox.
We would deploy after validation if:
- We have predictable, sustained execution demand.
- We already operate suitable Linux/KVM infrastructure.
- We need control over images, network boundaries, or data placement.
- We can own guest-agent security and host patching.
- We have measured command-ready latency under realistic contention.
- We have verified resource enforcement and cleanup after failure.
We would hold off if:
- We need a customer-facing execution API immediately.
- Our traffic is sporadic or operational ownership is unclear.
- We cannot verify KVM availability on the intended host.
- We expect jailer to supply network policy automatically.
- Our savings calculation excludes engineering and spare capacity.
- We cannot destroy a wedged or hostile guest independently of its agent.
Our next validation step would use one hardened host, one pinned artifact set, and a destructive acceptance suite: guest memory exhaustion, fork pressure, output flooding, disk filling, timeout enforcement, controller crashes, denied network probes, and snapshot compatibility failures.
Until those checks pass, “self-hosted isolation” is an architecture proposal—not a production guarantee.
We keep related infrastructure evaluations in our tools collection. If the real decision is whether to build this control plane or buy an execution service, our architecture services focus on that boundary: the workload, the threat model, and the operational bill we will actually have to pay.
Top comments (0)