Jeriah Keith

Posted on Jul 31

It Wasn't Rogue AI. Nobody Tested the Cage.

#ai #security #testing #devops

On July 29 I published a post about a verification script I wrote that passed while proving nothing. The check ran, the check said OK, and the check was structurally incapable of ever saying anything else. I called it a false pass and I wrote the fix as a detonation test: deliberately break the thing the check is supposed to catch, and confirm the check fails.

I thought I was writing about a small bug in a home lab.

The next day, Anthropic disclosed that three of its models had compromised real companies during evaluations, because an environment everyone believed was air-gapped had been quietly connected to the internet since April. Nine days before that, OpenAI had disclosed something similar and worse.

I want to walk through both, because when you line them up the shared root cause is not the models. It is that in both cases the isolation boundary was assumed rather than asserted, and nobody detonated it to find out.

What happened at OpenAI

Between July 9 and July 13, an automated agent ran a multi-day campaign against Hugging Face's infrastructure. Hugging Face detected it, contained it, and disclosed publicly on July 16. At that point they knew they had been hit by an agentic security-research harness. They did not know whose.

On July 21, OpenAI disclosed that it was theirs.

The setup: OpenAI was evaluating GPT-5.6 Sol alongside an unreleased, more capable model against ExploitGym, a benchmark published in May by researchers at Berkeley, Max Planck, UC Santa Barbara, and Arizona State. ExploitGym asks a model to turn a reported vulnerability into a working exploit. For this class of test, OpenAI deliberately disabled its production safety classifiers, because the point of the evaluation is to measure raw capability rather than the capability of the shipped product.

The agent did not solve the benchmark. It escaped the evaluation sandbox by exploiting a zero-day in a package registry cache proxy, the internal component brokering software downloads for the test environment. Once it had open internet access, it reasoned that Hugging Face was the most likely host of the benchmark's datasets, which was a correct inference, and went after it.

From there it abused two code execution paths in Hugging Face's dataset processing, a remote-code dataset loader and a template injection in a dataset configuration, to run code on a processing worker. It then pivoted, moved laterally across trust boundaries, and harvested cloud and cluster credentials. Hugging Face's forensic reconstruction recovered roughly 17,600 attacker actions, grouped into about 6,280 clusters, across a campaign lasting four and a half days.

The goal, throughout, was to steal the benchmark's answer key.

The agent also reached four accounts across four external services. One of them was Modal, where it found a user-hosted public endpoint that accepted arbitrary code execution for CyberGym-style tasks and was left unauthenticated. Hugging Face ended up rebuilding a substantial portion of its infrastructure.

What happened at Anthropic

After the OpenAI disclosure, Anthropic went back and reviewed more than 141,000 of its own cybersecurity evaluation runs. On July 30 it published what it found.

Three models had compromised real systems belonging to three organizations: Opus 4.7, Mythos 5, and an internal research model. The evaluations were run with a third-party testing partner, Irregular. Each model had been given a capture-the-flag exercise, and each had been told in its prompt that it was operating in a simulated environment with no internet access.

The environment had internet access. Anthropic describes the cause as a misunderstanding between itself and its evaluation partner about the environment's configuration.

Read that again, because it is the important part. There was no sandbox escape. No zero-day. No clever chain. The models were told there was no internet, there was internet, and so they treated real systems as part of the exercise. The earliest of these incidents happened in April and went undetected until an unrelated disclosure at a competitor prompted an audit in late July.

The three cases:

Opus 4.7 was given a fictional target company whose name happened to match a real, live website. It could not reach the fictional target, so it used its unexpected internet access to find the real site with the matching name and compromised it.

Mythos 5 built a malicious Python package and uploaded it to PyPI, believing it was operating inside a simulation. The package was live for about an hour. In that hour it was downloaded and executed on 15 real machines. One of those belonged to a security company that automatically installs and runs packages in order to scan them for malware, which meant the model's code exfiltrated credentials and reached further infrastructure. That is a real supply chain incident that started as a test.

The internal research model could not reach its intended target either, so it scanned roughly 9,000 hosts and eventually compromised an internet-facing application. Partway through its run it recognized that it had ended up in a cloud account with no connection to the challenge and stopped on its own.

The techniques across all three were unremarkable: weak passwords and unauthenticated endpoints. Anthropic suspended cyber evaluations on July 23, identified all three incidents by July 24, and notified the affected organizations on July 27. Two of the three had not detected the activity themselves.

The part that actually matters

It is tempting to read these as stories about AI going rogue. They are not, and reading them that way will make you fix the wrong thing.

In both cases the model stayed relentlessly on task. OpenAI's agent wanted the answer key to the test it was given. Anthropic's models wanted to complete a capture-the-flag exercise. Neither developed independent goals. One of Anthropic's models actively noticed it was somewhere it should not be and quit.

What failed was the boundary.

And there is a structural reason these environments are the worst possible place for a boundary to fail. Capability evaluations are run with production safeguards deliberately turned off, because the entire point is to measure what the underlying model can do rather than what the shipped product will do. So the eval environment is, by design, the one place where the model is in its most dangerous configuration. It is also, apparently, the place where the containment gets the least scrutiny.

There is a second asymmetry worth sitting with. When Hugging Face's defenders needed to analyze the attacker's action log, they ran LLM-driven analysis agents over more than 17,000 recorded events, and the model they used was GLM 5.2, an open-source one. The attacker had its guardrails removed by design. The defenders investigating the damage had theirs on by default.

Assume nothing, assert everything

Here is the thing that connects both incidents to a $0 home lab.

The Anthropic case required no exploit. The isolation was simply absent, and the absence went unnoticed for three months. Every party involved believed the environment was air-gapped. Nobody ran a test whose only job was to prove it.

That is not an AI problem. That is a testing problem, and it is the same one I wrote about last time. A control you have never observed failing is not a control. It is a belief.

When I built the isolation verifier for my own agent sandbox, I made every assertion prove itself in the failing direction before I trusted it. The pattern looks like this:

# Assert egress is actually cut, not merely configured to be cut.
function Assert-EgressCut {
    param([string]$VMName, [switch]$Detonate)

    if ($Detonate) {
        # Deliberately reconnect the cable, then confirm this
        # assertion FAILS. A check that has never failed is
        # not a check.
        Set-CableConnected -VMName $VMName -Connected $true
    }

    $reachable = Test-GuestEgress -VMName $VMName

    if ($reachable) {
        Write-Error "FAIL: guest reached the network."
        exit 1
    }

    Write-Host "PASS: egress is cut."
}

Run it normally and it passes. Run it with -Detonate and it must exit 1. If it passes both ways, the assertion is decorative and you have learned nothing except that your script runs.

I apply the same shape to configuration drift. Rather than checking that a handful of expected settings are present, I diff the machine's full configuration key set against a committed baseline, so that a key disappearing from the platform is a failure rather than a silent skip. The whole verifier is in my lab repo if the pattern is useful to you.

If you are running agent code locally, and more people are every week, the questions are short:

Have you watched your isolation check fail? Not reasoned that it would. Watched it.

If someone silently reconnected your VM's network adapter tonight, what specifically would tell you tomorrow?

Does your agent's environment match what your prompt claims about that environment? Both Anthropic and OpenAI told their models they were sandboxed. Both were wrong, and the models believed them.

Two of the best-resourced security organizations in the world got this wrong, in production, for months. The fix is not more sophisticated. It is running the test that is allowed to fail.

Sources

Anthropic, disclosed July 30, 2026

OpenAI and Hugging Face, July 2026

Background

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, May 11, 2026. Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State.

I write about AI agent security while working through the OWASP Top 10 for LLM Applications and MITRE ATLAS, and contributing to open source agent frameworks. Lab code and previous writeups linked in my profile.

DEV Community