DEV Community

Artemii Amelin
Artemii Amelin

Posted on

Nvidia's OpenShell Sandboxes Have No UDP. Pilot Protocol 1.13.11 Goes Through the Sandbox Proxy Instead of Around It

Nvidia announced its Open Agent Safety Platform today. It has two parts. OpenShell, an Apache 2.0 sandbox runtime, runs agents on the CPU under kernel-level controls. Sentry runs out of band on BlueField-4 DPUs, watches what the agents do, and according to Nvidia's announcement can quarantine one in milliseconds. More than 100 launch partners signed on, Microsoft and JPMorgan Chase among them. The pitch leans on this summer's incidents, where agents from several labs got out of their test environments and touched real systems.

The line in Nvidia's technical write-up that matters most is this one: "By controlling the path to the model, you own both the best observation point and also the kill switch." The same logic applies to every other path out of the sandbox. Agents are only one kind of software that has to live with that fence. The tools agents run inside it have to live with it too, and we build one of them.

What the OpenShell fence actually looks like

The OpenShell README says every network connection "passes through a policy check before it leaves the sandbox," and that agents never see real credentials, because OpenShell adds them only to requests bound for approved endpoints.

The Podman driver's networking notes are more concrete. The workload container runs with network=none. TCP opens, TCP byte streams and DNS travel over an authenticated gRPC channel to a supervisor container, and the supervisor resolves and authorizes DNS itself. One sentence settles a lot: "General UDP is unsupported." The Windows driver, built on Microsoft MXC, takes a different route to the same place, with a per-sandbox host CONNECT proxy that generates HTTPS interception certificates and injects the CA bundle into the sandbox's environment.

So inside one of these sandboxes, a process gets TCP to hosts the policy allows, DNS it does not control, possibly TLS that a proxy re-signs, and no UDP.

Why that breaks a peer-to-peer overlay by default

Pilot Protocol is an overlay network for agents. Its default transport is UDP: encrypted tunnels between peers, with STUN and hole-punching for NAT traversal and a relay when punching fails. Hole-punching is exactly the kind of traffic a sandbox operator should want to stop, and in an OpenShell-style sandbox it simply never leaves.

We ran into the same shape of fence earlier this month with hosted sandboxes such as Meta Muse: no outbound UDP, local DNS poisoned for *.pilotprotocol.network, and one way out, an authenticating HTTP proxy that only CONNECTs to port 443. Before v1.13.11 the daemon ignored HTTPS_PROXY completely, so getting a node online in that environment needed root, an SNI router and a mount-namespace override of /etc/hosts. Asking a sandbox for root to get around its own network policy is backwards.

What 1.13.11 changed

PR #470 shipped in v1.13.11 on September 24. The design goal was to go through the operator's proxy, never around it.

New installs use -transport=auto. The daemon sends one UDP discover to the beacon at startup. If nothing comes back within about 1.5 seconds and the compat beacon answers over TCP 443, it switches to compat mode, where the registry speaks TLS on registry.pilotprotocol.network:443 and the beacon runs over WSS on beacon.pilotprotocol.network:443.

In compat mode every outbound connection (registry client, WSS beacon, the daemon's HTTP clients) goes through HTTPS_PROXY or ALL_PROXY, and NO_PROXY is honored. Targets are CONNECTed by host name and never resolved locally, which is why poisoned sandbox DNS stops mattering. It also means a policy proxy sees a real hostname it can allow or deny, not an IP it has to reverse. TLS, SNI and pinned-fingerprint checks run end to end through the tunnel.

The fallback rule matters most here. If a proxy is configured and it refuses the check (a 407, a 403, or it cannot be reached), the daemon stays in compat and keeps going through the proxy. It does not fall back to direct connections past it. A daemon that quietly routes around a denied proxy is the exact behavior Sentry exists to catch.

A few smaller details from the README's proxy section:

  • A proxy URL with credentials reaches the daemon as $PILOT_PROXY in its environment, never on argv, so it does not show up in ps. Config output redacts it.
  • Muse rotates the credentials in HTTPS_PROXY every few minutes. A long-running daemon would keep the old ones and fail new connections with 407 while old tunnels stay up. proxy_cmd gives the daemon a command that prints the current proxy URL. It re-runs that command once 60 seconds have passed and on any 407, then retries the connection once.
  • Loopback targets are never sent to a proxy, even when an explicit proxy URL is set.
  • No root, systemd or launchd is required. pilotctl daemon start from a shell that has the proxy variables is enough.

The test setup in the PR was a non-root Debian bookworm container on a Docker --internal network whose only egress was an authenticating CONNECT-443 proxy, with /etc/hosts poisoned for the Pilot names.

Where interception proxies still conflict

OpenShell's Windows driver re-signs TLS with an injected CA, and certificate pinning is designed to refuse exactly that. Without a pinned fingerprint, our daemon verifies against the system roots, and on Linux it honors SSL_CERT_FILE (which pilotctl daemon start forwards), so an interception CA handed over that way is accepted. With registry_fingerprint set, it will not, and the operator has to let the Pilot hosts through uninspected. We think that is the correct default split: pinning is a statement that nobody in the middle gets to read the registry traffic, and a sandbox operator should have to make that exception explicitly.

Out-of-band is the right call

Sentry sitting on a DPU instead of inside the agent's own process is the strongest part of Nvidia's design. Enforcement that depends on the agent, or on the agent's tools, cooperating is not enforcement. What falls to tool builders is making sure our software works when the fence is up, and does not treat the fence as a network fault to engineer around. The daemon code for all of this is in the Pilot Protocol repository, under pkg/daemon and internal/proxyconf.

Top comments (0)