Sponsored by Tensorlake. They paid for this article and reviewed the draft; the numbers below are from my own runs on 5 and 6 Nov 2026.
Harbor, the framework from the Terminal-Bench creators, runs 4 coding-agent rollouts at a time by default, and the Terminal-Bench 2 README example uses the same 4 on local Docker. That is a config default, not a technical limit: Harbor can run far more, and its docs show 100 concurrent rollouts on other cloud sandboxes. On local Docker, though, raising it runs you into your own machine. Every RL step then waits on the slowest rollout in the batch.
So I ran Harbor on Tensorlake sandboxes at 100 concurrent, same command, two flags changed. A hundred rollouts finished in about two minutes. Scaling from 10 to 100 concurrent rollouts increased throughput by 7.9x. At 100 concurrency, median rollout latency was 110.6 seconds, 23% higher than running a single rollout. Here is where the seconds and the cents went.
I measured it with Harbor on Tensorlake sandboxes. The answer in one table:
Throughput is rollouts divided by batch wall time. Wall times, median of 3 runs: 96.4 s at 1, 100.1 s at 10, 127 s at 100 (runs of 166, 127 and 118 s).
The four numbers to remember: 7.9x throughput from 10 to 100 concurrent (79% of linear); +23% median rollout latency at 100 concurrent versus a single rollout; 300 of 300 rollouts passed at 100 concurrent; 37¢ sandbox cost for 100 rollouts, from Tensorlake's usage API.
Figure 1. Throughput at 1, 10 and 100 concurrent rollouts against the linear ideal. From 10 to 100 it grows 7.9x, 79% of linear.
What I actually ran
Everything uses Harbor's oracle agent. The oracle replays the task's reference solution, so there is no language model in the loop. That is on purpose: it isolates the part Tensorlake is responsible for, the sandbox, from the part that varies with the model. It also means these are infrastructure numbers, not agent pass rates. Treat 300 of 300 as "the plumbing did not drop anything", not "the agent is good".
Setup, in a virtual environment (the install pulls 130+ packages and took 157 s for me):
pip install "harbor[tensorlake]"
export TENSORLAKE_API_KEY=...
The rollout, one Terminal-Bench 2.1 task whose image Tensorlake has already published, so there is no build step:
harbor run --env tensorlake \
--dataset terminal-bench/terminal-bench-2-1 \
--include-task-name terminal-bench/pytorch-model-cli \
--agent oracle -k 100 -n 100
-k is how many attempts, -n is how many run at once. Going from 10 to 100 is changing two numbers. Nothing else in the task or the evaluator changes. Harbor works on native Windows too, which is where I ran it.
One requirement: 100 concurrent sandboxes needs a credit-based (usage) plan, which has a limit of 100 concurrent sandboxes. The free tier has lower concurrency limits, so you cannot run this exact command on it.
Figure 2. The only change is where the rollouts run. Harbor's default is 4 trials at once; each Tensorlake sandbox takes one rollout.
Scaling efficiency: what 100 at once costs you
From 10 to 100 concurrent, throughput went up 7.9x, which is 79% of the 10x you would get if nothing slowed down. The price of the missing 21% is latency: at 100 concurrency the median rollout took 110.6 s, 23% more than the 89.9 s of a single rollout run alone (the 10-concurrent median was about 90 to 93 s). More than half of that gap is in the verifier (40.7 s alone, 52.7 s at 100). Setup moved a little (8.2 s to 8.6 s) and the agent step went from 37.0 s to 42.0 s.
Reliability held: 300 of 300 rollouts passed at 100 concurrent, and 334 of 334 across all my oracle runs.
The tail is the part that varies. The slowest rollout in each batch set that batch's wall time: 163.6 s, 127.2 s and 117.5 s across the three runs. So at 100 concurrency, plan around the slowest rollout, not the median. If your trainer waits for the whole batch, the tail is your step time. If it can use rollouts as they arrive, you get most of the benefit sooner.
Where the 90 seconds go
Before scaling anything, I split one rollout into its phases (median of 3 runs): sandbox setup 8.2 s (9%), agent step 37.0 s (41%), verifier 40.7 s (45%).
Figure 3. The same rollout alone and at 100 concurrent. The verifier grows the most (40.7 s to 52.7 s).
Two things stand out. The sandbox is under a tenth of the rollout, so "sandbox startup" is not where RL time goes. And the verifier takes as long as the agent: scoring is half the job, so a rollout holds a full machine for the whole minute and a half, not just for the agent's turn.
For the API call itself, Sandbox.create() took a median of 1.27 s from my laptop over 24 creates. Tensorlake's docs say sandboxes boot in hundreds of milliseconds; as the team explained to me, that figure is the OS boot after creation, not the API round trip, so expect about a second and a quarter from your code to a sandbox you can use, and about 8 s in Harbor until the task container is ready.
What it costs
This number is measured, not guessed. Tensorlake's usage page reports the cost of each sandbox, so I pulled the usage export for my account. It lists 334 sandboxes that ran my oracle rollouts, each sized 2 CPU, 4 GB RAM and 30 GB disk (the task asks for less, but Harbor's Tensorlake environment raises every sandbox to that minimum). Tensorlake charged $1.23 for all of them: about 0.37 cents per rollout, about 37 cents per 100.
Billing is per second at 7 cents per core-hour, 1.5 cents per GB-hour of RAM and 0.02 cents per GB-hour of disk. CPU is billed on active execution with a floor of 20% of the allocated vCPU, and my rollouts averaged about 0.85 busy vCPU out of 2. That is why the bill lands between the two bounds I had worked out from the price list alone: 28 cents at the floor, 61 cents with both cores busy the whole time. Of the 37 cents, about 47% is CPU, 48% is RAM and 5% is disk.
Figure 4. Cost of 100 rollouts: the measured figure sits between the CPU floor and ceiling.
Per rollout: about 0.37 cents. For 100 rollouts: about 37 cents. For 1,000 rollouts (extrapolated): about $3.70.
Two caveats. The export is Tensorlake's current-price estimate, not a finalized invoice, and it rounds each sandbox to a whole cent, so I priced the usage quantities directly: that gives $1.19 against the $1.23 total. And it covers every rollout in my runs, not only the 100-way ones; the 300 longest-running sandboxes average 0.36 cents each, so the figure barely moves. Because you pay per second, the slower rollouts at 100 concurrent cost a little more each than a rollout run alone.
What this means for 1,000 rollouts
Figure 5. The scorecard: what I measured at each level.
Here is the comparison that matters, for 1,000 rollouts. The rows for 4, 8 and 30 are arithmetic from the rollout time I measured (about 95 s), not runs on local hardware, and a local Docker rollout may take longer or shorter; the 100 row extrapolates my 100-way result to ten waves. I have not tested either.
That is about 20x faster than the default, 10x faster than 8 concurrent and 2.5x faster than 30, with no machine to size, no images to cache and no cleanup to own. At the measured 0.37 cents each, the sandbox time for the 1,000 would be roughly $3.70. Run it yourself before you plan around it: the tail may behave differently at ten waves than at one.
Two things that got in the way
Large output gets an extra newline. A sandbox printing 40,000 characters returned 40,002; 100,000 returned 100,006. That is one added newline every 16,384 characters. If you print JSON over 16 KB from a sandbox and parse it, it fails with "Invalid control character". Do not strip the newlines: that is not a safe fix, because it also breaks any JSON that contains real newlines. Write the result to a file inside the sandbox and read the file instead. Tensorlake says it is working on the root cause.
The Harbor page has no RL guide yet. It says rollouts are coming in follow-up guides. What you do above, -k and -n, works today; the RL examples live on a separate page, "Reproducible Environments for RL Rollouts".
What I did not test
No language model agent, so no pass rate and no token cost. No rival sandbox on the same task. No local-Docker runs to confirm the 4, 8 and 30 concurrency estimates. No snapshots, no network-policy phases, no Docker Compose tasks. The cost is Tensorlake's current-price estimate from the usage export, not a finalized invoice. Scale AI's AgentEnv is out of this piece: it did not run on native Windows, and Tensorlake pointed out that is an AgentEnv limitation.
Try it
You need a usage-based plan for the 100 concurrent sandboxes. Start here: tensorlake.ai, and the Tensorlake documentation.
Originally published on Medium.
Top comments (0)