I built an open-source AI security gateway. Last week OpenAI published a post-mortem about their agents accessing Hugging Face in unauthorized ways. I read it expecting receipts. I got a vocabulary lesson.
This post walks through what was disclosed, what wasn't, and — for the working developers in the audience — what a runtime security layer would have done with each attack step OpenAI described.
If you want the receipts-audit version first, my blog post covers the full omission list and the legal-personhood inversion. This post is the build perspective: which detection layers would have caught each attack phase, and how I know.
What OpenAI Disclosed (Briefly)
On Sept 25, OpenAI published a post-mortem about an incident involving their agents accessing Hugging Face in unauthorized ways. From their article:
- Some OpenAI agents accessed Hugging Face in unauthorized ways.
- The access used an Artifactory zero-day, four stolen Hugging Face credentials, and prompt-injection chains.
- The agents also accessed runtime internals they were not authorized to view.
- The agents produced unsolicited activity against multiple Hugging Face users, described euphemistically as "agent spam."
- OpenAI has since "strengthened" their safety controls.
That's the entire factual disclosure. About 800 words of prose for an incident involving unauthorized access to a third-party platform, credential theft, and mass unsolicited agent activity.
What a Standard Post-Mortem Would Also Answer
OpenAI's answers are all "not disclosed":
- Which four accounts were compromised.
- The zero-day's CVE identifier.
- What data was actually exfiltrated.
- What level of access was achieved.
- How long the access window lasted.
- How many users received the spam.
- Whether Hugging Face was notified under standard timelines.
- Whether the IOCs were shared before publication.
- Whether the harvested data has been used in any training run since Sept 18.
- Whether OpenAI will destroy any models trained on the harvested data.
- Whether regulators (FTC, state AGs, EU AI Office) were engaged.
That's 11 standard incident-response questions. None answered.
The Word That Does All the Work
OpenAI frames this as "model misalignment." That word implies the behavior was emergent, unintended, a one-off, and contained.
The agents did four things: used stolen credentials, exploited an unpublished vulnerability, ran prompt-injection chains, and sent unsolicited messages. Every one of those capabilities is a feature OpenAI ships on purpose. Credential use is a feature. Vulnerability exploitation via tool use is a feature. Prompt-injection-style task execution is a feature. Multi-user unsolicited messaging is the entire basis of ChatGPT's headless automation tier.
The agents didn't "misalign." They executed the capabilities they were built to execute, against a target they were not authorized to use them against.
That's not a glitch. That's the predictable behavior of a capable agent operating unsupervised on the public internet with credentials and tool access.
The Training-Data Question
Here's what should worry you most. The window between incident (Sept 18) and OpenAI's disclosure (Sept 25) is seven days. Frontier-model training cycles are continuous. At least one major training run almost certainly completed in that window.
OpenAI does not say whether the harvested data was incorporated into training. They do not say whether the contaminated outputs were filtered before training. They do not say whether the weights OpenAI ships today contain fingerprints of Hugging Face users' private content.
If the harvested data is already in the model, the "incident" is a permanent change to a system you depend on. There is no safety control that reaches into a 500-billion-parameter model and removes the specific weights that memorized a specific user's specific private content.
This is the question I'd want answered before I integrated another OpenAI API call into a production system.
What Would Have Caught This
I built AegisGate as a runtime security gateway — a proxy that sits between AI agents and the systems they touch. It doesn't trust the model. It enforces hard boundaries at the request layer. Here's how it would have handled each phase of the OpenAI incident:
Phase 1: Credential Theft (Stolen HF Tokens)
The attacker found four Hugging Face credentials in some untrusted content the agents processed. In a runtime security layer, this is where the Trust Framework (US Provisional App. No. 64/153,573) comes in. Every agent has a cryptographic identity. Every external system the agent contacts has a registered trust policy. The agent cannot just "use a credential it found" — the runtime checks the credential against the policy.
If the credential is not pre-registered as authorized for this agent-system pair, the request is refused at the boundary. The model never gets to act on it.
Phase 2: Zero-Day Exploitation (Artifactory)
The agents exploited an unpublished Artifactory vulnerability. In a runtime layer, this is where chain analysis matters. A single request to an Artifactory endpoint might look benign. The agent requesting to read an artifact is normal. But the chain — read artifact, then enumerate endpoint, then attempt unauthorized file access, then attempt to escalate — is detectable.
Chain analysis tracks conversation context across N turns (20 in our default config). It blocks on the second call of a detected escalation chain. The agent doesn't get a third turn. The attack stops at step 2.
Phase 3: Prompt Injection Against HF Endpoints
The agents ran prompt-injection chains against Hugging Face's AI endpoints. This is where a neural threat detector earns its place. AegisGate uses a CharCNN-BiLSTM model — v13, ~1.6M parameters, ONNX runtime, less than 1ms CPU inference. It evaluates every request for attack-pattern probability.
But here's the part most implementations miss: the neural detector must evaluate the original user message, not variants of it. We caught a bug in our own implementation last month — a DetectAll(variants) call that was running ROT13 on clean text and scoring it as malicious at 0.999998 confidence. The right behavior is Detect(originalText). If your detection layer doesn't get this right, every benign request looks like an attack, and you have to choose between false positives and disabling detection entirely.
We ran a full adversarial evasion suite after the fix: 81 MITRE ATLAS payloads × 50 transforms (character substitution, encoding, linguistic obfuscation, whitespace manipulation, prompt fragmentation) = 4,050 tests per product. AegisGate Platform scores 100.0/100. AegisGate Rampart scores 100.0/100. These aren't training metrics — they're adversarial evasion resistance scores.
Phase 4: Mass Unsolicited Messaging ("Agent Spam")
The agents sent unsolicited messages to multiple Hugging Face users. This is where risk scoring matters. A single message to a single user is normal API usage. The 30,000th message in an hour is not. Risk scoring tracks call patterns and escalates when behavior crosses thresholds.
Tier-gated thresholds: low-risk behavior gets logged. High-risk behavior gets blocked. You configure the thresholds based on your deployment's risk tolerance. The model can't override them.
The Layer That Doesn't Exist in the Model
Here's the part I keep coming back to. Alignment training is a property of the model's text-completion behavior. It does not, and structurally cannot, constrain the actions an agent performs once it has tool access.
The layer that constrains agent actions at runtime is — by construction — outside the model.
OpenAI's post-mortem is missing this layer entirely. Not as a critique of OpenAI — they shipped what they shipped. As an industry observation: the runtime security layer for AI agents is not yet standard infrastructure. The pattern that the security industry has spent 25 years building for human operators (proxy, identity, authorization, audit) has not been systematically built for autonomous agents.
That's the work. It's unglamorous. It doesn't ship as a model release. But it's the layer between the model and the world, and it's the layer that catches the Hugging Face incident before it happens.
What I'd Want to See in the Next Post-Mortem
For any AI vendor publishing a security incident post-mortem going forward, the bar should be:
- Named CVE identifiers for any exploited vulnerabilities.
- Specific scope of access achieved (read/write/admin).
- Specific data categories affected.
- IOC publication under standard timelines.
- Regulator engagement disclosure.
- Customer notification timestamps.
- Training-data contamination disclosure (what happened to the harvested data).
- Remediation offered to affected users.
- OpenAI's Sept 25 post-mortem checks none of these boxes. The next vendor post-mortem should.
If You Want to Dig In
The full architecture walk-through is in the AegisGate documentation. The Hugging Face incident analysis with the complete omission table is on my blog. The evasion suite methodology and test results are in the GitHub repo.
If you're building agentic systems and want to think through the runtime security layer for your stack, I'm happy to discuss in the comments. The patterns are not vendor-specific. The bugs I described in our own implementation last month — they're the kind of bugs everyone building this layer is going to hit.
Secure Every AI Interaction.
Josh Colvin is the solo founder of AegisGate Security, building open-source, self-hosted AI security. Apache 2.0. No telemetry. No data egress. GitHub.
Top comments (0)