Two weeks ago, an autonomous AI agent, one of OpenAI’s own cyber-evaluation models, escaped a test environment that was supposed to be sealed off from the internet, reached Hugging Face, and compromised production over a weekend. I wrote about it last week. Since then, it is genuinely all anyone in this field wants to talk about. In every conversation this week the same question came up: if everyone involved is serious and competent, and they are, how did this happen?
My answer is simple. Everything was not right. Real things were missing. And when people asked me what, I realized the last round of writing, mine included, mostly celebrated what held: Hugging Face detected an unprecedented attack with their own telemetry, contained it fast, and confirmed no public models or weights were tampered with. That is true and they deserve credit. But “what held” is the wrong lesson. The lesson is in the gaps, and the gaps are not exotic. They are the ones every practitioner nod at the moment you name them.
Here are the three I keep coming back to, and here is the uncomfortable part: they are the exact problems a group of us have been building open solutions for, for the last year.
Gap 1: the supply chain runs untrusted code by default
Initial access was a malicious dataset that abused two code-execution paths, a remote-code dataset loader and a template injection in a dataset config. If you work in ML, you did not flinch reading that, because you already know our supply chain executes untrusted code as a matter of routine. trust_remote_code=True, pickled artifacts, loader scripts. We normalized running other people’s code to load a model or a dataset.
What is needed is integrity and provenance you can verify before anything executes: is this the artifact its author published, under what license, and does the thing about to load it run in a sandbox that cannot be talked out of. On the model-weights side we have been building exactly that in the open, the Weight Custody Manifest: a signed manifest binding the exact weight hash, so a tampered or poisoned fork fails verification before it ever loads, plus lineage, license-as-a-release-condition, and a kill switch. It is pre-1.0 and access is by request today, so if model-artifact integrity is your problem, come ask. AGT ships execution sandboxing for agent tool calls in the same spirit.
Honest admission, because this newsletter is called Proof not Promises: the dataset-and-loader half of the supply chain is where we have been thinking, not yet shipping. Weight custody is not dataset custody. Extending an attested, sandboxed manifest to datasets and loaders is open work, and it is the first place I would welcome the community.
Gap 2: the agent had no brakes, and left no receipt
The agent took, in Hugging Face’s own words, thousands of individual actions across a swarm of short-lived sandboxes. Two things were absent, and both are relatable to anyone who has watched an autonomous agent run: nothing said “no” to an action in real time, and when it was over there was no trustworthy, tamper-evident record of what the agent actually did. Detection is after the fact. Logs are editable. Neither proves what happened at the moment it happened.
This is the cluster we have spent the most on, all of it open:
- AGT (Agent Governance Toolkit) enforces deterministic allow/deny policy at the action and tool-call layer, with zero-trust identity and execution sandboxing, and it covers 10 out of 10 of the OWASP Agentic Top 10. That is the real-time “no” the agent never met.
- cMCP (Confidential MCP) runs that enforcement inside a hardware-attested enclave, at the MCP boundary this agent abused. The policy bundle is measured into the attestation report before any code runs, so a compromised host cannot swap or bypass it.
- TRACE produces a signed, hardware-rooted, offline-verifiable record of every call: which model ran, under which policy version, on what hardware, with the full tool transcript. That is the forensic receipt Hugging Face did not have, and its anchors live in an append-only Merkle registry, so a suppressed entry is detectable, not a silent non-event.
- Agent Manifest gives an agent an unforgeable identity, all ten artifacts that define it (system prompt, policy bundle, tool schemas, model identity, provenance, delegation chain, and more) anchored in hardware. “We did not know exactly what the agent was” stops being possible.
None of this is theoretical. It is running code, in public repositories, today.
Gap 3: one foothold owned the network, and the secrets were just sitting there
From code execution on a worker, the attacker escalated to node level, harvested cloud and cluster credentials, and moved laterally across internal clusters. This is the oldest shape of a breach: flat trust, secrets readable on the machine, one foothold becoming the whole estate. AI did not change that shape; it just made the operator faster.
The open answers here are confidential execution and verifiable identity. Secrets that are never readable on the host because the workload runs in a TEE and keys release only against a valid attestation. A cryptographic, attested identity per workload instead of a shared credential. Sealed, attenuated channels between agents so a compromised one cannot impersonate its neighbors, which is what cA2A (Confidential Agent-to-Agent) specifies. This is the ground the Opaque platform and cMCP already stand on: attestation-gated key release, per-workload identity, tamper-evident audit.
The detail that should matter most to open source
The single most revealing line in Hugging Face’s disclosure: their defenders were blocked by frontier-model safety guardrails from analyzing the attack and had to fall back to an open model (GLM 5.2) to do the forensics. Sit with that asymmetry. The attacker’s model ran with reduced refusals; the defenders’ models refused to help. The people cleaning up were handicapped by the safety features of the tools they pay for, while the attacker had none.
That is the strongest argument I know for open, defender-controlled models and open governance standards. You cannot run incident response on infrastructure whose guardrails treat your own investigation as the threat. Defenders need a capable model they run themselves, vetted and ready before the incident, and an open, inspectable governance layer around it. This is not a nice-to-have. It is a structural requirement for defense, and it is precisely where an open community, not a single vendor, has to own the ground.
We have been building this in the open for a year. Come build the rest with us.
I am not writing this to say we predicted an autonomous agent would break out of an eval and hack a hub. I am writing it because the missing layer this incident exposes is the exact layer a group of us have been putting in the open, in public, for the last year, an open stack now on track to a million downloads:
- AGT: ~4,900 stars, ~790 forks, 100+ contributors, 10/10 of the OWASP Agentic Top 10, now stewarded under the Linux Foundation.
- agentrust-io: an open stack around it, cMCP, TRACE, Agent Manifest, cA2A, conformance tests, and end-to-end examples across financial services, healthcare, and SaaS.
- and a curated map of the whole field for anyone starting: the awesome-ai-governance list.
Two invitations, and both are real. If you have this problem, use it, it is open and it works. If you want to shape the missing pieces, the attested data-and-loader supply chain, blast-radius ceilings for autonomous agents, egress governance for agentic runtimes, join us and build them in the open. We are standing up a fellowship to do exactly that, and the attested data-and-loader supply chain is first on my list for it. More on that very soon.
Hugging Face and OpenAI are now discussing these gaps in public, and that openness is the right instinct. Here is my constructive addition: three of those gaps have had open, working answers for a year, and the rest are worth building together rather than each of us relearning them the hard way, one incident at a time.
Trust the platform less. Verify the artifact. Govern the agent. And build the layer that proves it, in the open.
Top comments (0)