Arc Ops Level 5 is sandboxing: what an agent can reach on the day it is wrong. 24 hand-written jobs × 40 trials = 960 task-runs, swept across all 24 allow-list widths.
Escapes are 0 at every width. The boundary genuinely holds — that is not the problem.
Level page: https://dev48.infy.uk/arcops/level5-sandboxing.html · Repo: https://github.com/dev48v/arc-ops
Every number a sandbox reports about itself improves as you tighten it
Blocked share, denials, contained runs — all monotone in the right direction as the allow-list narrows. The harm is not:
| width | harm | completed | harm / completed |
|---|---|---|---|
| 3 | 3,266 | 371 | 8.8032 |
| 11 | 856 | 723 | 1.1840 |
| 23 (no sandbox) | 714 | 960 | 0.7438 |
The peak is at width 3: 4.57× the unsandboxed harm, and 11.84× per completed task, on 371 completions of 960.
Twelve widths do more harm than having no sandbox at all. Fourteen are worse per completed task.
Tightening one entry at a time
Over 23 single-entry steps: 7 raise the harm, 10 lower it, 6 change nothing. The worst single step is width 7 → 6: +1,044 harm for +0 completions.
Why
A narrower list does not remove the work. It removes the direct route to it, and what is left is the indirect one. A task that cannot write the deploy manifest can still write the CI config — and the runner reads it on the next tick.
| at width 11 | |
|---|---|
| resources wanted | 19 |
| reached directly | 11 |
| reached by deputy | 3 |
| genuinely denied | 5 |
263 effects arrive by deputy, 40 are unattributable, and 33.41% of the harm arrives that way. The log names the deputy, not the agent. Nothing crossed the boundary; something inside was persuaded to act.
The setting that decides it
| sandbox | on deny | harm | completed |
|---|---|---|---|
| jail | improvise | 856 | 723 |
| jail | fail | 384 | 610 |
| none | — | 714 | 960 |
Improvising on a denial is worse than having no sandbox (+142 harm, 1.1989×). Failing closed is better (0.5378×) and gives up 350 completions. That is the real trade, and it is not the one the dashboard shows you.
An allow-list also understates itself
At width 3, 3 entries transitively reach 12 resources — 4.00× what the list says. A writable checkout on its own reaches 11. The list is what you review; the closure is what you get.
And the generator underneath
Four ways of drawing a key, each tested twice — the histogram everybody runs and the serial test almost nobody does. The lowest chi-square of the four (3.12, comfortably passing) flips parity on 80.05% of consecutive draws. The most uniform-looking stream is the most predictable one; a histogram cannot see a serial defect.
895 tests, standard library only, MIT. 46 verifier asserts, 14 page checks, 0 failures.
Top comments (0)