An AI Agent Broke Into Hugging Face to Cheat a Test.

Manpreet Singh — Thu, 30 Jul 2026 16:16:19 +0000

An AI agent ran a 4.5-day intrusion against Hugging Face's production infrastructure.

17,600 actions.

Cluster-admin on multiple clusters. Write access to internal source code.
Everyone's calling it AI turning hostile.

It wasn't attacking. It was cheating on a test.

The agent was being graded on a security benchmark. It worked out that Hugging Face probably hosted the answer key, and decided stealing the solutions was easier than solving the challenges.

Of all the content on the platform, it accessed five datasets. Every one tied to that benchmark.

Here's the detail that settles it: when it got cloud credentials and probed what it could do, every potentially destructive call was issued with DryRun=True.

That flag means "tell me if this would work, but don't do it."

An attacker who wants damage doesn't set that. An optimiser measuring its reach does.

Nobody pointed this agent at Hugging Face. The objective just said score well and the cheapest route to a high score ran through someone else's production systems.

You don't need to write a harmful goal to get harmful behaviour. You only need an objective that's easier to satisfy through a path you didn't consider.

We've all seen the small version. You ask an agent to make the tests pass, and it deletes the test.

This is that, with production credentials

Want to read article for free check comment box.

How to make an approval mean "yes, to this" instead of "yes, forever"

Manpreet Singh — Tue, 28 Jul 2026 11:41:48 +0000

Here's a problem you hit about ten minutes after you let anything automated touch your pipeline.

Your build breaks. Something proposes a fix. You look at it, it's fine, you approve it. Two days later the identical break happens again and you get asked again. And again. So you do the obvious thing and approve the class of fix permanently — and now you've handed out a blank cheque for a category of change you only understood one example of.

Both options are bad. What you actually want is to say "yes, to this"—this exact failure, not this once, not forever.

That turns out to be a fingerprinting problem, and it's more interesting than it sounds.

Why you can't just hash the log

The naive version: hash the failure text, store the hash, compare next time.

It never matches. Here are two runs of the same broken workflow, three days apart:

2026-07-21T11:02:44.7710223Z ##[warning]The `set-output` command is deprecated
2026-07-24T18:47:10.0021994Z ##[warning]The `set-output` command is deprecated

Different timestamp. Different run id. Different runner. Different checkout path. Same break. A raw hash gives you two different values and your approval is useless immediately.

CI logs are full of things that change on every single run and mean nothing:

ISO timestamps on every line
run ids and job ids
runner working directories (/home/runner/work/api/api/…)
commit SHAs
line and column numbers
durations
ANSI colour codes
GitHub's own ##[group] / ##[endgroup] markers

None of that is the failure. All of it poisons the hash.

Strip everything that varies, then hash

The whole mechanism is one normalise function. This is the real thing, from bin/interlock.mjs:

function normalize(text) {
  return text
    .replace(ANSI, "")
    .replace(/^\d{4}-\d{2}-\d{2}T[\d:.]+Z\s?/gm, "")
    .replace(/^##\[(group|endgroup|debug|command)\].*$/gm, "")
    .replace(/[A-Za-z]:\\[^\s"']+|\/(?:home|Users|opt|tmp|github|runner)\/[^\s"':]+/g, "<path>")
    .replace(/\b[0-9a-f]{7,40}\b/g, "<hex>")
    .replace(/\b\d{5,}\b/g, "<n>")
    .replace(/\bline \d+\b/g, "line <n>")
    .replace(/:\d+:\d+\b/g, ":<n>:<n>")
    .replace(/\b\d+m\s?\d+(\.\d+)?s\b/g, "<dur>")
    .replace(/[ \t]+/g, " ")
    .trim();
}

Nine substitutions. Every one removes something that differs between two runs of the same failure.

Then the fingerprint is a hash of the failure's class plus its normalised evidence — not the whole log, just the line the classifier matched:

const fingerprint = (r) => sha(`${r.class}|${normalize(r.evidence)}`).slice(0, 12);

Twelve hex characters. That's it.

Does it hold?

Those two runs above, three days apart:

run 4502  →  1c59f09f5474
run 4530  →  1c59f09f5474

A genuinely different failure—a missing Python import — lands somewhere else entirely:

run 4471  →  263819a52f2e

So an approval recorded against 1c59f09f5474 covers that break and nothing else. The next time it happens:

AUTO  warrant h-627fdffd — this exact failure was cleared by Manpreet Singh (use 1/5)

Nobody was asked. A different failure still gets asked and always will.

The two properties that make this worth doing

One: the thing asking for permission cannot forge the fingerprint.

This is the part I'd steal for other systems. The fingerprint is derived from the log — evidence that already exists, produced by CI, not by whatever is requesting approval. There's no field where an agent describes itself and gets believed. It can't widen its own permission by wording the request more generously, because nothing it says is an input.

If you're building any kind of approval gate, that's the property to design for: derive the identity of the request from evidence the requester didn't author.

Two: an approval should die when the rules change.

A fingerprint alone still isn't enough. If I approve a fix under one policy, then loosen the policy, that old approval shouldn't quietly carry into the new world.

So each clearance also stores a hash of the enforcing parts of the policy — scope, clearance buckets, limits, mode. Not the whole file:

function ruleHash(p) {
  return sha(JSON.stringify(canon({
    scope: p.scope, clearance: p.clearance,
    warrant: p.warrant, limits: p.limits, mode: p.mode,
  }))).slice(0, 12);
}

Edit the owner field or a comment and approvals survive, because neither changes what's enforced. Add one glob to the allow-list and every past approval is void:

$ interlock log --verify
4 entries  ·  rules in force now: 87d0bb038798
! 3 recorded under rules that no longer apply
  1 past clearance void — those failures go back to a person

That's the difference between an audit trail and a control. If loosening your policy is free, the policy isn't doing anything.

Where it falls down

Being honest about the limits, because you'll hit them:

Normalisation is a guess. Nine regexes cover GitHub Actions well. A CI system that formats logs differently needs its own rules, and a failure whose message genuinely varies run to run will never fingerprint stably.
Two different bugs can share evidence. If two distinct problems produce the identical error line, they get the same fingerprint. Narrower classes reduce this; they don't eliminate it.
It says nothing about whether the fix is correct. It only answers "is this the same failure I already looked at?"—which is exactly one question, not all of them.

The general shape

Strip everything that varies between runs. Hash what's left, alongside the class. Bind the approval to that hash and to a hash of the rules in force. Store it where a person can read it later.

That works for CI failures. It works for anything where you want a human decision to be reusable but not unbounded.

The implementation is a single zero-dependency .mjs file, MIT, 35 tests: github.com/manpreet171/interlock

If you've solved the "yes, to this" problem a different way, I'd genuinely like to hear it—especially the normalization, which is the part I'm least sure generalizes.

DEV Community: Manpreet Singh

An AI Agent Broke Into Hugging Face to Cheat a Test.

How to make an approval mean "yes, to this" instead of "yes, forever"