How memory checkpoints and sandbox forking let you build once, checkpoint the warm state, and run as many parallel workers as you need. No reinstalls. No reloads.
Running eight parallel ML experiments sounds efficient. Watching each one reinstall numpy from scratch does not.
The logs told the story: eight sandbox workers, each spending its first 40 seconds on pip install numpy and loading the dataset before a single line of training code ran. The training scripts were different.
The setup wasn't. I had paid for the same 40 seconds of work eight times.
The training scripts were the candidates. The setup was not.
That is not a performance bug. It is a mental model problem.
This article is the fix.
"Starting Fresh" Is Not the Same as "Starting Clean"
Most experiment setups quietly make an assumption: real worker isolation means booting from a clean image every time. No shared state. No residue from previous runs.
That assumption is not wrong. It is just more expensive than it needs to be.
Workers running parallel experiments need independent filesystems and independent process trees. They need a known starting state. What they do not need is to reinstall numpy, reload the same dataset from disk, or re-seed a Python environment that was identical across all of them at the start of the iteration.
"Starting fresh" became a proxy for isolation.
But the actual requirement is narrower: get every worker to the same state, then let them diverge. That is a different primitive than booting from a base image.
Filesystem Snapshots vs Memory Snapshots
Before I discovered how Tensorlake handles this, I assumed all snapshots worked the same way Docker images do.
A filesystem snapshot captures disk state only. When you restore from one, the sandbox cold boots: the VM initializes from scratch, the OS comes up, Python starts, your process begins again. The installed packages are there on disk, so you skip the apt install and pip install steps. But you still pay for boot time and for every piece of process initialization that was not baked into the image.
The interpreter loads. Modules get imported. Data gets pulled into RAM.
I learned this the hard way.
My first implementation used filesystem snapshots because they were the default. I launched eight workers from the same filesystem snapshot and watched every one of them cold boot, import NumPy, load the dataset, and rebuild the execution environment.
I had skipped the package installation step, but I still paid for everything that came afterward.
Then I read the docs again and saw CheckpointType.MEMORY.
A memory snapshot captures the filesystem plus the entire VM memory state, including all running processes at the exact moment of capture. Restore from one and the sandbox resumes warm. The Python interpreter is already running. Modules imported before the checkpoint are already loaded. Variables that were in memory are still there.
It is closer to Unix fork() than to a Docker image. You are not restoring a filesystem. You are branching from a specific execution point.
Tensorlake exposes filesystem and memory checkpoints as distinct primitives, making it possible to choose between cold restores and warm restores depending on the workload.
from tensorlake.sandbox import Sandbox, CheckpointType
# Cold-boot restore: captures disk state only
snapshot = sandbox.checkpoint(checkpoint_type=CheckpointType.FILESYSTEM)
# Warm restore: captures disk + VM memory + running processes
snapshot = sandbox.checkpoint(checkpoint_type=CheckpointType.MEMORY)
The default when you call sandbox.checkpoint() with no arguments is filesystem. For the fork pattern described in this article, you want memory.
One tradeoff to understand upfront: memory snapshots lock the resource configuration at capture time.Image, CPU count, memory limit, entrypoint, and secrets all come from the snapshot when you restore.
You cannot change them at restore time.
If you need different resource allocations per worker, use filesystem snapshots and accept the cold boot. For most parallel experiment setups, the locked resources are fine since you sized the base environment for the workers when you created it.
Secrets are the exception.
They survive the checkpoint but are not locked to it. Secrets are passed as environment variables to run(), so you can pass a new value at restore time and overwrite whatever was captured in the snapshot. Rotating an API key across forked workers does not require a new snapshot.
The Fork Pattern
Once you have a memory snapshot, the fork pattern is straightforward. Here is what it looks like end to end:
Every worker starts from the same warm execution point. They share nothing at runtime. The base sandbox can be terminated right after the checkpoint. The snapshot_id persists independently.
Step 1: Create one base sandbox and do all shared setup inside it.
from tensorlake.sandbox import Sandbox, CheckpointType
base = Sandbox.create(
image="tensorlake/ubuntu-minimal",
cpus=2,
memory_mb=4096,
)
# Install shared dependencies once
base.run("pip", ["install", "numpy", "pandas", "--break-system-packages"])
# Verify the environment is ready
base.run("python3", ["-c", "import numpy; import pandas; print('Environment ready.')"])
Step 2: Capture the warm state.
snapshot = base.checkpoint(checkpoint_type=CheckpointType.MEMORY)
print(f"Snapshot captured: {snapshot.snapshot_id}")
# The base sandbox can be terminated. The snapshot persists independently.
base.terminate()
Step 3: Restore N workers in parallel from the same checkpoint.
from concurrent.futures import ThreadPoolExecutor
def run_experiment(script: str) -> str:
# Each worker restores a fresh copy of the warm environment
worker = Sandbox.create(snapshot_id=snapshot.snapshot_id)
result = worker.run("python3", ["-c", script])
worker.terminate()
return result.stdout
candidates = [script_v1, script_v2, script_v3, script_v4, script_v5]
with ThreadPoolExecutor(max_workers=len(candidates)) as pool:
results = list(pool.map(run_experiment, candidates))
Each Sandbox.create(snapshot_id=...) call restores an independent copy of the warm environment. Workers share no filesystem and no runtime state. They just started from the same point.
The snapshot.snapshot_id is a persistent string. Save it to a file. Use it in the next session, in the next iteration of your loop, in a different process entirely. The warm state survives until you explicitly delete the snapshot.
Where This Actually Matters
ML Experiment Racing
Andrej Karpathy published the autoresearch repo in early 2026.
The core idea: an LLM reads your current best training script, proposes N code modifications, you race all N in parallel sandboxes, keep the winner, and loop. The loop runs overnight. Each accepted modification becomes the new baseline for the next iteration.
The naive implementation pays the setup cost N times per iteration. If the training script needs numpy, scipy, and a small dataset loaded into memory, that is real time per candidate, per loop. With 8 iterations and 3 candidates each, you have paid identical setup cost 24 times.
With Tensorlake's Snapshot Fork Pattern, you do the setup once.
Take a memory checkpoint. Fork all candidates from that checkpoint.
They start warm and only diverge where the proposed modification changes behavior.
The structure of one iteration looks like this:
# Build the warm baseline once per iteration
base = Sandbox.create(image="tensorlake/ubuntu-minimal", cpus=2, memory_mb=4096)
base.run("pip", ["install", "numpy", "scipy", "--break-system-packages"])
base.run("python3", ["-c", "import dataset; dataset.load_into_memory()"])
# Checkpoint once: all candidates share this starting point
snapshot = base.checkpoint(checkpoint_type=CheckpointType.MEMORY)
base.terminate()
# Race candidates from the warm snapshot in parallel
# Each worker starts where the base left off, not from a fresh image
with ThreadPoolExecutor(max_workers=len(candidates)) as pool:
results = list(pool.map(lambda s: run_in_sandbox(snapshot.snapshot_id, s), candidates))
# Keep the winner, discard the rest, repeat
winner = min(results, key=lambda r: r["val_loss"])
The snapshot.snapshot_id is reused across all candidates in the same iteration. Setup cost is paid once per loop, not once per candidate.
RL Rollouts with Reproducibility
RL rollouts have a hard requirement: same seed, same action sequence, same trajectory. Every time, without exception. That guarantee breaks the moment workers share any state. A shared pip cache can introduce version skew between runs. A shared /tmp carries residual files from previous episodes. Even calling env.reset() correctly does not help when state outside the environment object persists between episodes.
With per-rollout sandboxes forked from a common snapshot, the isolation is structural rather than enforced by convention. There is no shared filesystem between workers.
The seed goes directly into the Python script that runs inside the sandbox, not into the host process. Keeping it there means the host's random state stays completely out of the episode:
import json
from tensorlake.sandbox import Sandbox
def gym_harness(seed: int) -> str:
# Script runs inside the sandbox, not the host
return f"""
import gymnasium as gym, json
env = gym.make("CartPole-v1")
obs, _ = env.reset(seed={seed})
env.action_space.seed({seed})
trajectory, total_reward = [], 0.0
for _ in range(200):
action = env.action_space.sample()
next_obs, reward, terminated, truncated, _ = env.step(action)
trajectory.append((obs.tolist(), int(action), float(reward), bool(terminated)))
total_reward += reward
obs = next_obs
if terminated or truncated:
break
print(json.dumps({{"seed": {seed}, "total_reward": total_reward, "steps": len(trajectory)}}))
"""
def run_rollout(snapshot_id: str, seed: int) -> dict:
worker = Sandbox.create(snapshot_id=snapshot_id)
result = worker.run("python3", ["-c", gym_harness(seed)])
worker.terminate()
return json.loads(result.stdout)
One specific gotcha here: env.reset(seed=seed) only seeds the observation and transition RNG. The action space has its own separate RNG that requires env.action_space.seed(seed) independently. Miss the second call and trajectories vary across runs with no obvious error. The mismatch is silent and will look like flaky results until you trace it to the gymnasium source.
Parallel Browser Agents
Browser warmup is slow: login flows, OAuth, session cookies, waiting for JavaScript-heavy pages to stabilize. If you are running 20 browser workers in parallel, you do not want each one repeating that from scratch.
Tensorlake's ubuntu-vnc sandbox image makes this practical because the authenticated browser itself becomes part of the checkpoint.
Complete the auth flow once, checkpoint the authenticated browser state, then fork your workers from there:
from tensorlake.sandbox import Sandbox, CheckpointType
# Authenticate once in a single browser sandbox
browser_base = Sandbox.create(image="tensorlake/ubuntu-vnc", cpus=4, memory_mb=4096)
# ... drive Chrome via CDP, complete login, reach stable page state ...
# Checkpoint the authenticated browser in memory
auth_snapshot = browser_base.checkpoint(checkpoint_type=CheckpointType.MEMORY)
browser_base.terminate()
# All 10 workers start with the logged-in browser state already in memory
workers = [Sandbox.create(snapshot_id=auth_snapshot.snapshot_id) for _ in range(10)]
The same pattern that eliminates repeated pip install across ML workers eliminates repeated OAuth flows across browser workers. The primitive is the same. Only the warmup content changes.
Suspend vs. Snapshot: Two Different Things
Suspend and snapshot both preserve sandbox state. They solve different problems and it is worth being clear about which one you reach for.
Suspend pauses this specific sandbox and holds its state for later resumption under the same sandbox ID. It uses no compute while suspended. When you resume, the sandbox comes back in under a second with the same process IDs, filesystem, and in-memory state intact. Named sandboxes auto-suspend on timeout instead of terminating, which means you do not lose a long-running agent session because a task ran slightly over its time budget.
Snapshot captures a reusable artifact you can restore into new sandboxes. The artifact persists after the source sandbox is terminated. Restore from it once or many times.
Suspend is for pausing and resuming a single workstream. Snapshot is for branching from a known state into N parallel workers.
The sandbox keeps running while the snapshot is being captured. The snapshot artifact persists regardless of what happens to the source sandbox afterward.
Decision Matrix
Why This Works: The Underlying Principle
Unix fork() exists for the same reason. Process creation is expensive, so instead of spawning a child from scratch, you copy the parent's entire memory state and let the child diverge from there.
Git branches for the same reason: you checkpoint a known state and branch from it because re-deriving full history for every branch would be wasteful.
AI experiment infrastructure got filesystem snapshots first. But filesystem snapshots only moved the cost boundary to disk. The running process, loaded modules, in-memory data, and interpreter state all had to rebuild on every new worker anyway.
Memory checkpointing moves that boundary further. The running process is part of what gets captured. Load a 500MB dataset into a pandas DataFrame before the checkpoint, and every forked worker starts with that DataFrame already in RAM. Pre-compile JIT functions and the cache is there too. The interpreter does not restart. Nothing reloads.
The expensive part of an experiment is rarely the experiment. It is everything that had to be true before the experiment could start.
That cost is not fixed. It just looked fixed because the tooling treated it that way.
What the Platform Handles
Three numbers that matter for the fork pattern specifically.
Speed. The tensorlake/ubuntu-minimal image starts up in a few hundred milliseconds; tensorlake/ubuntu-systemd (full init system) takes around one second. Forked workers restore from a memory snapshot warm, skipping boot and process initialization entirely. Suspended named sandboxes resume in under a second without losing any memory or filesystem state. The published SQLite benchmark (100k inserts, 2 vCPU / 4GB) shows Tensorlake at 2.45s against Vercel at 3.00s, E2B at 3.92s, Modal at 4.66s, and Daytona at 5.51s.
Scale. Tensorlake supports fanning out to thousands concurrent sandbox environments — the number the Harbor integration targets for RL rollouts and eval pipelines running from a single snapshot. The overall project limit is 5 million sandboxes.
Isolation. Sandboxes run on MicroVMs backed by Firecracker and CloudHypervisor. LLM-generated code never shares a kernel with other tenants or with your host process. Tensorlake is SOC 2 Type II and HIPAA compliant, with EU data residency and zero data retention options available.
Getting Started with Tensorlake
Install the SDK and grab an API key from cloud.tensorlake.ai (free tier available):
pip install tensorlake
export TENSORLAKE_API_KEY=your_api_key
Minimal working fork pattern you can run right now:
from tensorlake.sandbox import Sandbox, CheckpointType
from concurrent.futures import ThreadPoolExecutor
# Step 1: Warm the base environment
base = Sandbox.create(cpus=2, memory_mb=4096)
base.run("pip", ["install", "numpy", "--break-system-packages"])
# Step 2: Checkpoint the warm state
snapshot = base.checkpoint(checkpoint_type=CheckpointType.MEMORY)
base.terminate()
# Step 3: Fork workers from the snapshot
def run(script: str) -> str:
w = Sandbox.create(snapshot_id=snapshot.snapshot_id)
out = w.run("python3", ["-c", script]).stdout
w.terminate()
return out
scripts = ["print('v1')", "print('v2')", "print('v3')"]
with ThreadPoolExecutor(max_workers=3) as pool:
results = list(pool.map(run, scripts))
print(results)
All three workers start warm. snapshot.snapshot_id is a persistent string you can store and reuse across sessions.
Conclusion
Those eight pip install numpy calls were not a performance bug.They were a modeling mistake. I had built my experiment loop around "start fresh" when what I actually needed was "start identical."
The difference is real. Starting fresh means paying full setup cost per worker. Starting identical means paying it once and forking from there.
Tensorlake's memory checkpoints provide one implementation of this pattern: build the environment once, capture the execution state, and fork as many isolated workers as you need from the same warm baseline.
You do the expensive shared work once, freeze the execution state, and branch as many workers as you need from that exact point. Each worker gets its own isolated environment. The setup cost is paid once.
This applies to most parallel AI workflows: ML candidate racing, RL rollouts, browser agent pools, CI evaluation pipelines. The pattern is the same across all of them.
If I could go back to those eight workers all stuck on pip install numpy, the fix would have been four lines. Create the base. Warm it. Call checkpoint(MEMORY). Restore from there. The 320 seconds of identical setup would have happened once. The other seven copies would never have existed.
That is what the pattern is for.
References
- Tensorlake Sandboxes: Introduction: Overview of how MicroVM sandboxes work, boot times, and supported images
-
Tensorlake Snapshots Documentation: Full reference for
checkpoint(), snapshot types, restore, clone, and delete operations - Tensorlake Sandbox Lifecycle: State machine for ephemeral vs named sandboxes, suspend/resume, and timeout behavior
- Tensorlake Agentic Autoresearch Loop: Full working example of the Karpathy-inspired autoresearch pattern using Tensorlake sandboxes
- Tensorlake RL Reproducible Environments: Deterministic rollout guarantees with per-rollout sandbox isolation and seed embedding
- Tensorlake Agentic Swarm Intelligence: Map-reduce pattern for parallel agent swarms in isolated sandboxes
-
Tensorlake Drive Chrome over CDP: Running Chrome inside
ubuntu-vncsandboxes and driving them via the Chrome DevTools Protocol - Tensorlake Harbor Integration: Running Harbor evaluations and RL rollouts with 10k+ concurrent sandbox environments
- Tensorlake SQLite Benchmark: Public benchmark comparing sandbox filesystem performance across Tensorlake, Vercel, E2B, Modal, and Daytona
- karpathy/autoresearch: The original autoresearch repo by Andrej Karpathy that inspired the ML experiment racing pattern
- Tensorlake SDK Reference: Complete Sandbox handle method reference for Python and TypeScript
Top comments (0)