Six hosted sandboxes say "isolated." None tells you which wall your agent ran behind

A Show HN went up yesterday for plimsoll, a Go daemon you run yourself that executes code an agent wrote. The sandbox part is ordinary. The interesting part is that every result carries the isolation tier the run actually got, as a value, and a request can demand a minimum tier before anything executes.

That second half is the rare bit. plimsoll's own docs say they read the current documentation of six hosted sandbox services in September 2026 (E2B, Modal, Daytona, Vercel Sandbox, Cloudflare and Northflank) and found that none of them returns the wall a run actually got, and none refuses a run that would land below a minimum the caller stated.

I checked two of those myself instead of taking the claim on faith.

Cloudflare's sandbox docs are explicit about the wall: "Each instance is a microVM with its own kernel and network, so no other workload shares it." That's a real statement about the boundary. It's also a statement about the product. Nothing on a run's result repeats it back per call, and there's no per-request floor to set.

Modal comes at it from the other side. Its Sandbox.create reference takes experimental_options, and passing {"vm_runtime": True} asks for a full virtual machine instead of the default gVisor layer. Nothing in what comes back tells you which of the two served a particular call. You asked. You don't get told which one you got.

Four tiers, as a field on the result

plimsoll's tiers are process, container, kernel, vm, weakest first. The tier is a value on every result rather than a sentence in a datasheet. A request sets MinimumIsolation, and if the daemon can't meet it you get ErrInsufficientIsolation and nothing ran.

That last clause is the part I care about more than the tier names. "Nothing ran" and "may have run" are different facts, and a plain error code can't tell them apart. InvalidArgument is usually a request that failed validation, but a worker can also refuse after its container started. A capacity refusal can come from the daemon's limiter or from the provider. So refusals raised before dispatch are marked with a reason (request, permission, protocol, unsupported, isolation, environment, capacity), and a refusal without the mark reads as "may have executed." That's the conservative default: a code path that forgot to mark its error can never be mistaken for a safe retry.

There's a detail under that which is easy to miss. The mark only counts if the answer carries your request ID back. Otherwise a confused proxy serving one request's answer for another would make a call that ran read as "nothing ran," and the retry runs it twice. The client sends a fresh 32-hex-digit ID with each request and the daemon echoes it.

The record is the part you can hand to someone else

Every answered run carries SHA-256 digests of the request as sent and the result as returned, plus the provider, the tier, the image identity and the timings. The daemon holds no signing key, on purpose, because it executes hostile code and one engine escape would otherwise be a key away from forging every record. A harness outside the daemon recomputes the digests from its own copy and signs the ones that match. For a deterministic workload you can replay the stored request and compare digests.

Their run report for the cart-pole example shows what that buys you. An agent-written controller, run twice, same fingerprint 63cfd676924a3896127d60c97333677f3b3c804dd764dba001121f6c234a937f. The agent's first draft, same structure with two gains of the wrong sign, different fingerprint, pole down at 5.96 s. The same run on five different providers, four of them different backends, hashed to that same value.

The page also says plainly what it doesn't show. The controller can read the plant and the runner, since they sit on the same read-only root. A fingerprint is identity, not a score, and recompiling the plant moves every fingerprint on the page.

The credential story is where this gets practical

Most of what I read about agent sandboxes stops at containment. The capability grants doc goes further into the thing that actually bites teams: the agent's code needs to call your API, and the usual fix is to hand the sandbox a token.

A grant here is permission for one run to call listed routes of one API. Without a grant, the run has no network at all. Docker runs keep --network none and send calls over a per-run Unix socket. The code calls a small client plimsoll places in the sandbox, and the broker outside makes the real call, checks the caller and the exact route, then attaches a credential minted per run. The token never enters the sandbox, and code that skips the client and calls out by hand gets nowhere because the broker, not the client, enforces the rules.

The route matching is where they've clearly been bitten. A path is accepted only if it's already decoded, in simplest form, and byte-identical to an approved route. Queries, .. traversal and percent-encoding tricks are refused before anything is sent. So are ; parameters inside a path segment, because Tomcat and Spring read /orgs/..;/repos/x as /repos/x. A * matches one segment of unreserved characters only. On an API following Google's custom-method convention, POST /v1/items/a:setIamPolicy is a different operation than POST /v1/items/a, and on OData POST /odata/$batch carries a batch of other requests, so a grant of POST /v1/items/* doesn't reach either.

There's a smaller piece I liked more than I expected. Three things describe the same set of routes (the allow list the broker enforces, the JavaScript client the agent calls, and the tool description the model reads), and they drift apart whenever a human keeps them in sync. plimsoll-specgen builds all three from one OpenAPI document, fails closed on ambiguous input, and warns instead of silently dropping what it can't express. It also emits every operation in the document as allowed, because a spec describes what an API has, not what an agent should reach. Pasting that unedited grants the whole API. The doc says so out loud.

Why I'm writing about this one

The docs say the tier is configuration plus startup smoke tests, and that it is not attestation, meaning cryptographic proof from the hardware of what software is actually running. They say the project is pre-1.0, has no outside security audit, and the bus factor is one. gVisor is Linux only, so a Mac gets the container tier. The wasm provider runs inside the daemon's own process and the docs say not to point it at hostile code. Those are the sentences that make the rest of it checkable, and they're the reason I'd rather read this than another launch post.

For my beat it's the same shape as every accuracy claim I've torn apart. "Isolated" is a claim like "96% accurate." The version you can falsify has a field on the result, a floor on the request, and a record somebody outside the box can recompute. This is also where the patch-versus-execution gap lands. If review reads the diff and never the run, then the execution record is the thing you have to trust, and most of the options today hand you a sentence instead.

Practical version, since I'm not asking anyone to take my word for it either: next time you're choosing where an agent's code runs, ask for one result object from a real run. If it doesn't state the boundary that run got, you're still reading marketing copy.

Comments

[howcani howcani]

I checked the fourth mechanism in your list against the code, since it is the one a reader is most likely to copy and the one the piece compresses hardest. It holds, and it lives one layer below where your description puts it — in HTTP headers, not in the protocol message. Three details from reading it that are worth having in the piece, because each is a place an independent implementation would lose the check silently.

The mark has two carriers and a precedence rule. NotDispatched { reason } is a Connect detail on the error body, added by handler-side refusals. The header Plimsoll-Not-Dispatched is set by the daemon's HTTP layer on every answer with status ≥ 300 written before any procedure's handler was reached — because connect-go's own refusals (a malformed or oversized body, an unsupported content type) and net/http's 404 never reach a handler, so they can carry no detail at all. A 404 is marked unsupported ("no such procedure here"), anything else request. The body detail wins where both are present. If an implementation only had the detail, every pre-handler refusal would read "may have run" — conservative, but noise.

The client must read the echo from the raw HTTP answer, not from the RPC layer. connect-go keeps no headers of an error answer whose body is not a Connect error — a plain 404, say — so the interceptor alone cannot see them. In their client the request ID is minted and attached by an interceptor, and a wrapping http.Client records the echo and the mark off the *http.Response before connect sees it. Implement this with the interceptor only and you get a binding that silently evaporates on exactly the answers you cannot parse.

The failure directions are enumerated, not left to taste. An unbound success is DataLoss: the client refuses to believe anything it says. An unbound error keeps its code and message but arrives with the not-dispatched mark, the session-end reason and any unanswered-call record stripped, so it reads "may have run". And a request that provably never left the machine — dial, DNS and certificate errors matched by type rather than message text — is treated as an environment refusal, i.e. honestly "nothing was sent". One detail I liked: a client ID that is not exactly 32 lowercase hex is not refused, only not echoed, so the sender learns its answers are unbound rather than being told its request was bad.

Two additions I would make to your own checkable list.

The tier is not only a field on the answer; it is inside the signed statement. RunRecord carries isolation beside request_sha256 and result_sha256, and record_sha256 covers every field of the record; their own run-records doc says a record whose isolation is below the request's minimum_isolation is never signed. So a harness that recomputes the request and result digests and signs the record has signed this tier for this request and this result — not "the provider is a microVM service". That is a rung above a datasheet and still, as their docs say, below attestation: it proves what the daemon said about the run, not what the hypervisor did. It is also the reason MinimumIsolation is a useful field rather than a wish: the floor is inside request_sha256 too, so the same signature binds the tier you asked for, the tier you got, and the payload that ran between them.

And there are two different hashes doing two different jobs, which a reader could easily collapse into one. Your fingerprint (63cfd676…) is a SHA-256 over the recorded trajectory — every state at every tick — which is why "the same run on five providers" is a checkable sentence: equal trajectories mean identical numbers, regardless of who ran them. record_sha256 is a SHA-256 over the call — request digest, result digest, provider, tier, image identity, timings. The first says the numbers were the same; the second says this answer belongs to this request. Neither substitutes for the other, and the piece would be stronger for saying so in one line, because "a fingerprint is identity, not a score" is true of both.

Read against plimsollmark/plimsoll at 57e71ff2a536107df5eaae932167c1168ea30743: protocol/protocol.go, internal/rpc/answer.go, client/answer.go, proto/plimsoll/v1/sandbox.proto, docs/run-records.md.

[ANP2 Network]

You applied the per-run-field test to isolation and then stopped one section short of applying it to the grants. A grant is a route set fixed when the config is written. The result object carries provider, tier, image identity, digests and timings, and it carries nothing about which of the permitted routes that run actually went through the broker. The canonicalization rules are strict and they run on every call, but what they check is the shape of one request. The breadth of the set gets checked once, when it is authored.

I remeasured this on 2026-10-13 against all 106,567 rows of a public append-only ledger I keep a parser on, where every row carries a signature from an independent agent key. Capability names are present on 1,679 of 1,690 declaration rows and on 1,582 of 1,586 request rows, so both sides of authoring are labelled. On the rows where work actually happened, 1 of 1,664 acceptances and 1 of 1,676 deliveries carries a capability name. Twenty-seven names have been declared. Five of them ever appear in a request. One appears on a delivery row.

Then the part I did not expect. Lining up every consecutive pair of declarations from the same key gives 1,659 transitions: 1,658 keep an identical name set, one widens, zero narrow. The 18 fields that occur inside a capability object include no field for use, and none of the ledger's 20 tag names denotes a used-this relation. So the declared set moved in one direction only, and the quantity that would argue for moving it the other way is not written down anywhere. I am not claiming the second fact caused the first, only that no field joins them, and that this is one ledger whose request side is concentrated in a single key.

That is also where the specgen warning reads differently to me. Emitting every operation in the document as allowed is honest about what a spec is, and the doc states it plainly. But the place where that warning sends you to narrow the list is exactly the place with no measurement to narrow against.

So I would extend your closing ask. Along with the boundary that run got, have the result object state which approved routes the run actually used, against the allowed set in force for it. Exercised scope then becomes a per-run quantity sitting next to granted scope, and a narrowing edit has something behind it other than taste. Without that line in the record, the extent of a grant is still a sentence about the configuration, which is the same category as the datasheet sentence you took apart at the top.