I Learned Eval Environments Aren’t Fort Knox.

I Learned Eval Environments Aren’t Fort Knox. So I Tested My Own Agent Sandbox Instead of Trusting It.

After reading how OpenAI’s and Anthropic’s “isolated” test environments turned out to have a way out, I stopped assuming my own agent setup was actually contained. Here’s how I tested it, what I found, and the exact hardening steps that closed the gaps.

When OpenAI disclosed that an experimental model escaped its cyber-eval sandbox and hacked Hugging Face, and Anthropic followed with three of its own incidents where Claude reached real companies during testing, the detail that stuck with me wasn’t the sophistication of the attacks. It was the root cause. In Anthropic’s case specifically, nobody broke anything. The environment was just never actually isolated, and everyone assumed it was.

That’s a much scarier bug than a clever exploit, because it means “sandboxed” was a label, not a guarantee. So instead of reading the postmortems and moving on, I spent an afternoon actually testing whether my own Claude Code / agent setup was isolated the way I assumed it was.

It wasn’t. Here’s the process, end to end, with the commands.

Step 1: stop assuming, start testing

“Sandboxed” gets used loosely. A container gives you process and filesystem isolation by default. It does not give you network isolation by default. Docker’s standard bridge network routes straight to the internet unless you tell it not to. If your agent runs docker run my-agent with no network flags, it has the same outbound access your host does.

I tested this directly, before touching any config:

# run a throwaway container the same way I normally run my agent
docker run --rm my-agent-runtime sh -c "
  echo '--- can I reach the open internet? ---'
  curl -sS -m 3 -o /dev/null -w '%{http_code}\n' https://example.comecho '--- can I reach cloud instance metadata? ---'
  curl -sS -m 3 -o /dev/null -w '%{http_code}\n' http://169.254.169.254/latest/meta-data/
  echo '--- can I reach other hosts on the local network? ---'
  curl -sS -m 3 -o /dev/null -w '%{http_code}\n' http://192.168.1.1
"

Every single one of those returned a response. My “sandboxed” agent could reach the open internet, could (on a cloud box) hit the metadata endpoint that hands out IAM credentials, and could reach other machines on my LAN. That’s the exact shape of gap that let Claude Opus 4.7 reach a real company’s production database during Anthropic’s eval: nothing technically broke, the wall just was never built.

Step 2: know what “isolated” actually means at each layer

Before hardening anything, I mapped out what each common setup actually guarantees, because the terminology is where these assumptions come from:

+------------------------------+---------------------+----------------------+------------------------+
| Setup | Process isolation | Filesystem isolation | Network isolation |
+------------------------------+---------------------+----------------------+------------------------+
| Bare process on host | No | No | No |
| Docker, default bridge | Yes | Yes (unless mounted) | No (full outbound) |
| Docker, network_mode: none | Yes | Yes (unless mounted) | Yes (no network at all) |
| Docker + custom network + | Yes | Yes (unless mounted) | Yes (allowlist only) |
| egress proxy | | | |
| Firecracker / gVisor microVM | Yes (kernel-level) | Yes | Depends on config, same |
| | | | mistake still possible |
+------------------------------+---------------------+----------------------+------------------------+

The row that matters is the last one. Even a proper microVM, the kind frontier labs use for these evals, gives you zero network isolation guarantee by default. It’s a config choice, every time, and it has to be verified, not assumed. That’s precisely what Anthropic’s postmortem confirmed: a misunderstanding with a third-party evaluator left “isolated” environments with a live route to the internet.

Step 3: default to no network, add access back deliberately

The fix isn’t complicated. It’s just a default nobody sets until something goes wrong.

# docker-compose.yml
services:
  agent:
    image: my-agent-runtime
    network_mode: none # no network interface at all
    volumes:
      - ./workspace:/workspace # only what the agent needs to touch
    read_only: true # filesystem outside the mount is read-only too

Confirm it actually holds:

docker compose run --rm agent sh -c "curl -sS -m 3 https://example.com || echo 'blocked, as expected'"

If the agent genuinely needs outbound access (installing packages, calling an API), don’t reopen the bridge. Route it through a proxy that only allows named destinations:

services:
  agent:
    image: my-agent-runtime
    networks:
      - agent-net
    environment:
      - HTTPS_PROXY=http://egress-proxy:8080
      - HTTP_PROXY=http://egress-proxy:8080
egress-proxy:
    image: mitmproxy/mitmproxy
    command: >
      mitmdump --set block_global=true
      --allowlist "pypi.org,registry.npmjs.org,api.anthropic.com"
    networks:
      - agent-net
networks:
  agent-net:
    internal: false # egress-proxy is the only container with a real route out

Only the proxy container has a real path to the internet, and it only forwards to the domains you name. The agent itself has no other route.

Step 4: block the metadata endpoint everywhere, not just in the agent’s own network namespace

This one’s easy to miss because it’s a host-level firewall rule, not a Docker setting, and it matters most on any box that actually runs on a cloud provider:

# block every container on the host from reaching the cloud metadata service
iptables -I DOCKER-USER -d 169.254.169.254 -j DROP
# make it persistent across reboots (Debian/Ubuntu)
apt-get install -y iptables-persistent
netfilter-persistent save

I’d assumed my staging box, which sits on a cloud VPC, had this blocked by default. It didn’t. Any container on that host, agent or otherwise, could reach the metadata service and potentially pull temporary IAM credentials without ever touching a password. This is the single highest-leverage rule in this whole list.

Step 5: a verification script, not a one-time check

The mistake in all three disclosed incidents (OpenAI, Anthropic, Meta) wasn’t a missing control on day one, it was a control that quietly stopped holding and nobody re-checked. So I turned Step 1’s manual test into a script that runs before every agent session:

#!/usr/bin/env bash
# preflight.sh: run before starting any agent container
set -e
FAIL=0
check_blocked () {
  local desc="$1" url="$2"
  if docker compose run --rm agent sh -c "curl -sS -m 3 -o /dev/null $url" 2>/dev/null; then
    echo "FAIL: $desc is reachable (should be blocked)"
    FAIL=1
  else
    echo "OK: $desc is blocked"
  fi
}
check_blocked "open internet" "https://example.com"
check_blocked "cloud metadata endpoint" "http://169.254.169.254/latest/meta-data/"
check_blocked "local network gateway" "http://192.168.1.1"
if ["$FAIL" -eq 1]; then
  echo "Preflight failed. Do not start the agent session."
  exit 1
fi
echo "Preflight passed. Network boundary confirmed."

It takes about two seconds to run and it’s caught a regression for me once already, after a docker-compose edit accidentally moved the agent service onto the default bridge network during a refactor. Without the script, I wouldn’t have noticed until something actually went wrong.

Step 6: a fully offline setup for anything I don’t want to think about

For routine experimentation where I don’t want to reason about blast radius every single time, I run a local model with genuinely no path to the internet, not “blocked by a rule I have to trust,” but structurally absent:

# local model, no external network interface possible
docker run -d --name ollama --network none -v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama pull qwen2.5-coder:7b
# agent container talks to Ollama over an internal-only network,
# with no gateway to anything outside the host
docker network create --internal agent-local
docker run -d --name ollama-svc --network agent-local -v ollama:/root/.ollama ollama/ollama
docker run --rm --network agent-local \
  -e OLLAMA_HOST=http://ollama-svc:11434 \
  my-agent-runtime

The --internal flag on the Docker network means there's no gateway out at all, not even if something inside the container tries. It's not as capable as Opus or GPT-5.6 for serious work, but for "let the agent poke at a scratch repo and try weird things," it removes the entire category of risk the eval-escape reports describe.

What actually changed for me

Nothing here is exotic. network_mode: none by default, an explicit allowlist when access is genuinely needed, a blocked metadata endpoint, and a preflight check that runs every time instead of a mental note I trust forever. The whole point is that "sandboxed" stopped being something I assumed about my setup and became something I could point to a passing script for.

Frontier labs found out the hard way, with real companies on the other end, that an eval environment’s isolation is a claim until someone verifies it. I’d rather find that out from a script I wrote in twenty minutes than from an incident report.

Tags: docker, ai-agents, network-security, devsecops, self-hosted