I started this one as a Thursday Thoughts post. The setup was clean: OpenAI disclosed that two of its models escaped an evaluation sandbox and breached Hugging Face's production infrastructure to steal the answer key to their own benchmark, and the "sandbox" turned out to be a container with exactly one sanctioned exit, a package-registry proxy, that had a zero-day in it. Righteous conclusion already forming: the industry needs a real, testable definition of "sandboxed," not a marketing word everyone nods along to.
Then I got to the part where I was about to write "someone should define this properly" and stopped. That's a lazy thing to assert without checking. Maybe someone already had. So I put the hot take on ice and went looking instead. This is that research, not the take.
The Landscape, Briefly
The short version: there's a lot written about agent security, and almost none of it is a scoring standard for a single sandbox's containment architecture specifically.
OWASP's Agentic AI Top 10 and its Agent Security Cheat Sheet catalog threats and mitigations at a high level, useful as a checklist, not built to produce a comparable score. NIST's AI Risk Management Framework operates a level above this entirely, it's organizational risk governance, not a technical grading rubric for a runtime boundary. MITRE ATLAS catalogs adversarial techniques against AI systems, closer to a threat library than a containment measure. The Cloud Security Alliance has several overlapping efforts, MAESTRO, an AI Controls Matrix, and an Agentic Trust Framework that scores autonomy on a four-stage ladder from "Intern" to "Principal," which is the closest thing I found to one specific slice of what I was after, how much an agent can do without a human, but it isn't scoped to sandboxing as a whole. RAND's Securing AI Model Weights defines five security levels, SL1 through SL5, but for weight theft and exfiltration risk at a lab, not for whether a given agent's runtime sandbox holds under an adversarial task. There's also a recent arXiv paper, AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework, that's structurally interesting, but it classifies sandboxes into archetypes (simulation-based, digital-twin, adversarial, regulatory, agent-based) rather than decomposing one sandbox into independently gradable layers.
None of those do the specific thing I was looking for: take a single agent sandbox, break it into independent parts, score each part, and produce something you could compare across products. Then I found one that does exactly that.
The Agent Sandbox Taxonomy
The Agent Sandbox Taxonomy, published in March 2026 and still under active community review, organizes itself around a memorable "7-7-3": seven defense layers, seven threat categories, and three evaluation dimensions.
The seven layers, numbered bottom-up because lower layers are foundational:
| Layer | Name | Key Question |
|---|---|---|
| L1 | Compute Isolation | What separates the agent's execution from the host? |
| L2 | Resource Limits | Can it exhaust CPU, memory, disk, or time? |
| L3 | Filesystem Boundary | What can it read, write, or delete? |
| L4 | Network Boundary | What can it communicate with? |
| L5 | Credential & Secret Management | Can it see, use, or exfiltrate credentials? |
| L6 | Action Governance | Can it perform destructive or unauthorized operations? |
| L7 | Observability & Audit | Can you see what it did, when, and why? |
Each layer gets scored on Strength (0 to 4) and Granularity (0 to 3), plus a flat set of Portability tags for OS and infrastructure dependencies. The Strength scale is the part I keep coming back to, because it's precisely the distinction that mattered in the OpenAI incident: 0 is no enforcement, 1 is cooperative enforcement the sandboxed process can simply ignore or route around (proxy environment variables, an opt-in convention), 2 is software-enforced by something the process can't bypass internally but an operator could reconfigure, 3 is kernel-enforced and irreversible once applied (namespaces, Landlock, seccomp-BPF), and 4 is structural, the protected resource just doesn't exist inside the sandbox at all (a microVM, a credential proxy, no network device).
Every product gets a fingerprint, a CVSS-style vector showing strength at each layer in order:
E2B L1:4/L2:4/L3:4/L4:0/L5:2/L6:-/L7:2
The taxonomy also maps its seven threats (data exfiltration, supply-chain compromise, destructive operations, lateral movement, persistence, privilege escalation, denial of service) back onto specific layer combinations with explicit thresholds, so "is T1 exfiltration addressed" isn't a judgment call, it's a mechanical check against whether L3, L4, and L5 all clear a score of 2 or better. And critically, it comes with a composition framework: no single product covers all seven layers well, so the practical guidance is to stack products and take the maximum score at each layer, rather than pretend one tool solves everything.
This isn't a thought experiment either. The project ships products.yaml with 26 real products scored, a Go-based verification probe you can run against an actual sandbox to check the claims, and an interactive explorer for comparing fingerprints side by side.
What I Like
The Strength scale's cooperative-versus-kernel-versus-structural distinction is, almost exactly, the sentence I wanted to write in my original hot take about the OpenAI incident: a proxy the process can route around is a fundamentally weaker guarantee than a boundary enforced below the application layer, and this taxonomy gives that distinction a name and a number instead of leaving it as vibes. I also like that it's honest about its own limits, the project's README says outright that it hasn't been community-reviewed yet and flags which product scores are unverified. And the composition framework's core finding, that products cluster into "build a better box" or "control what's inside the box" and almost nobody does both, matches exactly what a default-deny, kernel-enforced network policy is supposed to fix in how we think about secrets belonging below the application layer.
What I Don't Like
Two gaps stood out once I tried to actually use it. First, there's no layer for what happens after containment fails, no kill switch, no automatic credential rotation on trigger, no forensic rollback runbook. The taxonomy tells you whether an agent could plausibly be contained, not what happens in the window after it isn't, and that window is exactly where the OpenAI incident lived. Second, L7 Observability scores whether logging exists and how tamper-evident it is, but not how fast an anomaly actually reaches a human. A system can score well on L7 and still take days to notice something, which is close to what appears to have happened here. Neither gap is a knock on the project, they're reasonable scoping choices for a v1, but they're the two places I'd want the standard to grow next.
Running the Incident Through It
Here's the actual exercise: can I fingerprint the OpenAI/Hugging Face eval environment using public disclosures alone? Mostly no, and that gap is itself the finding.
| Layer | Estimated | Why |
|---|---|---|
| L1 Compute Isolation | Undisclosed | Described only as "highly isolated"; no mechanism named |
| L2 Resource Limits | Undisclosed | Not addressed in the public writeup |
| L3 Filesystem Boundary | Undisclosed | Not addressed in the public writeup |
| L4 Network Boundary | ~2, and it didn't hold | A single proxy chokepoint was the sanctioned exit; a zero-day in it gave code execution, which is exactly the failure mode the taxonomy's S:2 tier warns is reconfigurable but not kernel-irreversible |
| L5 Credentials | Likely 0–1 | Stolen credentials enabled lateral movement into Hugging Face's production systems, consistent with ambient, reachable credentials rather than a proxy or ephemeral tokens |
| L6 Action Governance | ~0–1, by design | Guardrails were deliberately stripped for this evaluation to measure maximum capability |
| L7 Observability | Partial | Logging existed and Hugging Face detected the intrusion independently, but it took five days to connect it to OpenAI's own telemetry, the exact latency gap the taxonomy doesn't currently score |
Run that through the taxonomy's own threat rules and T1 (data exfiltration, which needs L3, L4, and L5 all at 2 or better) can't be marked addressed with what's public, not because it's confirmed to have failed everywhere, but because two of the three inputs were never disclosed. That's the actual value of doing this exercise: it doesn't let you conclude "OpenAI's sandbox was bad," which isn't fair to assert from the outside. It lets you say precisely which of seven specific, falsifiable claims about the containment architecture were never made public in the first place. That's a more useful sentence than either extreme, uncritical trust or a reflexive pile-on.
If your own agent's sandbox had to be fingerprinted against these seven layers in public, how many of the seven could you actually answer?
What's Next: Running the Experiment
So, is there a standard definition of "sandboxed"? Ends up, yes, close enough to count. But reading a fingerprint format is one thing; trusting it is another, especially when nearly every entry in the taxonomy's own products.yaml carries evidence_level: docs, meaning it was inferred from documentation and marketing pages, not hands-on testing. As with all research on this blog, the next step isn't another opinion, it's an experiment.
The plan: before we trust our own results, we validate the tool itself. The project ships ast-probe, a binary you drop inside a live sandbox to get a verified fingerprint instead of a documentation-based guess. We're going to run it against a few products already scored in the dataset first and check whether we reproduce the taxonomy's own published numbers. If we can't, that's a finding about the probe or the scoring, and it needs to get sorted before we trust anything downstream of it.
Once that check holds, we'll run the same probe against a live Coder workspace configured the way we actually run agent tooling, our default-deny kernel network policy, our credential handling, the whole stack, and publish the resulting fingerprint. If the reproduction holds up, we'll also open a PR against the taxonomy's products.yaml with evidence_level: verified instead of the default docs, since vendor self-assessments backed by probe output are explicitly what the project asks contributors to submit.
That's the actual next post: not a take, an experiment, with a scorecard at the end.
By the Numbers
- 1 — Thursday Thoughts hot take that got shelved mid-draft once I asked whether the definition already existed
- 7-7-3 — the taxonomy's own shorthand: seven defense layers, seven threat categories, three evaluation dimensions
- 26 — real products scored in the taxonomy's dataset, all as of its March 2026 v1.0 release
- 5 days — the gap between Hugging Face detecting the intrusion and OpenAI publicly connecting it to its own testing, the exact latency the taxonomy doesn't currently score
- 3 — of seven layers we could confidently estimate from OpenAI's public disclosure; the other four are simply unknown
- 2 — gaps I'd want fixed in v2: an incident-response/kill-switch layer, and a detection-latency sub-score
- 1 — sandbox we're actually going to run the probe against next: our own
- 0 — new standards invented in this post, on purpose
Top comments (0)