DEV Community: Jeriah Keith

119 Tests Passed. The One Named After the Bug Passed Too.

Jeriah Keith — Sun, 16 Aug 2026 09:28:03 +0000

My test suite came back green. 119 passed, 3 skipped, under a second.

At that exact moment, my tool was telling people to do the one thing a test in that suite promised it never told anyone to do. The test was named after the promise. It ran. It passed.

That gap between what a test is named and what it actually checks is the most expensive thing I have found in this project. I do not think it is rare, and the shape of it transfers to almost any codebase.

The 30-second version of what the tool does

I built a scanner that checks a company's email authentication setup from public DNS records, then generates a plain-English page telling them what to publish to fix it. Small accounting firms, mostly. They take that page to whoever runs their DNS and paste in the records.

That last part is the whole risk. The output is not a report someone reads and nods at. It is a set of instructions someone follows. If the instruction is wrong, a real business loses real mail.

The relevant feature is a staged rollout, a ladder. Each rung is one DNS record to publish, in order, with a stage label attached.

The advice it got wrong

One DMARC setting matters here: a tag called t.

A domain can publish "reject anything that fails authentication" and simultaneously publish t=y, which means "but don't actually do that yet, I'm still testing." Think of t=y as a parking brake. The policy is set. The brake is on.

Removing that brake is safe once your mail is properly signed. Removing it before that is how a company starts bouncing its own invoices.

So the tool has a rule. If a domain has the brake on and no working mail signing, do not tell them to take the brake off. Tell them to leave it alone and go fix signing first. That rung exists, it works, and it is the first thing on the page:

stage : Not yet - keep the t=y tag until DKIM is signing
value : v=DMARC1; p=quarantine; t=y; rua=mailto:r@example.com

Correct. Brake intact.

Here is the second rung on that same page, for that same unsigned domain:

stage : Next - after 2 weeks of clean reports, and only once DKIM is signing
value : v=DMARC1; p=reject; sp=reject; rua=mailto:dmarc@x.test; fo=1;

No t tag anywhere in it. Step one says keep the brake on. Step two hands over a record with the brake gone and the policy escalated, on the same page, to the same client.

The test that was written to prevent this

This is the part that should bother you, because it is the part I would have sworn was covered:

def test_no_rung_offered_to_an_unsigned_domain_removes_its_test_mode_brake(
    label: str, record: str, weights: Weights
) -> None:
    before = assess(record, weights, dkim=False)
    if not before.dmarc.policy_test_mode:
        pytest.skip("not in test mode; no brake to remove")

    brake_removed = strip_test_mode_tag(record)
    for rung in dmarc_rollout("x.test", before):
        assert rung.value != brake_removed

Read the name. Then read the assertion.

The name claims: no rung removes the brake.

The assertion checks: no rung equals this one specific string. That string is the input record with the tag deleted and every other byte left identical.

Those are not the same claim, and the distance between them is exactly where four bad outputs were living. The offending rung was v=DMARC1; p=reject; sp=reject; rua=mailto:dmarc@x.test; fo=1;. It has no brake. It is also not the input-minus-the-tag, because the policy changed and the other tags were regenerated. Different string. != holds. Test passes. Bug ships.

The test was not missing. It was not skipped. It ran on exactly the right input, in exactly the right scenario, and reported success.

Three more guards, all blind, all reasonable

This is the part I want any engineer reading to sit with. It was not one weak test. It was four, and each one was defensible on its own.

Guard 1. The string comparison above. It checked one exact value out of an unbounded set of wrong values.

Guard 2. "No suggested record drops the tag." Same technique, same blind spot. Two tests, one assumption, zero independence. Two guards that share an assumption are one guard wearing a disguise.

Guard 3. Publish the advice, then re-scan. Good design, this one: take the record the page tells you to publish, publish it in a simulated zone, run the whole assessment again, confirm the domain is still braked. It passed because it read only the first rung. The first rung was the correct one. It never looked at rung two.

Guard 4. "Following any step must not introduce a critical finding." The finding it watches for is triggered by the policy value alone. On a domain already set to quarantine, that finding had already fired. You cannot introduce something that is already present. The guard was structurally incapable of failing on this input.

Four layers. One live path straight through all of them.

How I actually found it

Not by reading the code. I had read that function. I had an opinion about it. My opinion identified one faulty code path.

What I did instead was enumerate. I wrote a throwaway script that took every relevant input state, ran it through the ladder, and printed every single rung it produced into a table. One column ran each generated record back through the real parser and asked, plainly, "does this still have the brake on?"

Eighteen rungs. Four with the brake removed.

input	stage	brake in output?
`p=quarantine; t=y; rua=…`	`Next - ...only once DKIM is signing`	gone
`p=quarantine; t=y; adkim=s; ruf=…`	`Next - ...only once DKIM is signing`	gone
`p=reject; t=y` (no reporting)	`Repair`	gone
`p=reject; t=y; pct=20; rua=…`	`Repair`	gone

Three separate code paths, not one. My reasoning had found a third of the problem and felt finished.

Now look at the bottom two rows. Those are labeled Repair. Not "only once DKIM is signing." Just Repair. Flat, unqualified, no warning attached anywhere. The safety caveat had been added to the escalation steps and never to the repair steps, because when that caveat was written nobody was thinking about repairs. A client reads a step called "Repair," assumes repairs are safe, and pastes it in.

The root cause was mundane, by the way. A branch appended its rung and did not return. Execution fell straight through into the three branches below it.

What I take from this

A test's name is documentation, and documentation drifts. The name was written when the intent was fresh. The assertion was written to the example in front of me that day. Nobody reconciled them afterward, because the test was green and green means done.

Assert on the property, not on a value. != "one_known_bad_string" is not a guarantee. Parsing the output and asking "does this have the property I care about?" is. Same test, ten more lines, and it would have caught all four.

Count your independent guards, not your total guards. I had four. Two shared a technique, one read only the first element, and one was watching a condition that was already true. That is not four layers of defense. That is one, and it was the weak one.

Enumerate the output space. Every real error in this project, mine and the AI assistant's I use on it, came from reasoning about what code would do instead of running it and printing what it did. Reasoning gave me one code path. A table gave me three. The table took fifteen minutes.

Mutation-test in both directions. A guard that should fire must be shown to fire. A guard that should stay quiet must be shown to stay quiet. An assertion no one has ever seen fail is a hypothesis, not a test.

Where it stands

I caught this in a working tree, mid-change, before the feature was finished. Not from a client call. That is the good outcome, and I only got it because I stopped trusting a green checkmark long enough to enumerate what the thing actually emits.

The fix is not obvious, which is why it is still open. Carrying the brake tag forward onto an escalation step means handing someone a record that says "reject everything, but not yet," which is confusing on its own terms. Suppressing those rungs entirely means the page goes quiet about a real next step. Both are defensible. That is a product decision, not a test decision, and I would rather sit with it a day than patch it because a table embarrassed me.

The test is getting rewritten either way. Right now its name is a promise its assertion does not keep, and that is worse than having no test at all. A missing test is an obvious gap. A lying test is a gap you have already convinced yourself is covered.

Go look at your suite. Find a test whose name makes a broad claim. Read what it actually asserts.

I would bet you find at least one.

For the curious

The sweep. The whole thing was about forty lines. Every input state, through the ladder, one row per rung, and one column that did the actual work:

def parses_as_test_mode(value: str) -> bool:
    """Run a candidate record back through the real parser."""
    res = dmarc_check(FakeResolver({("_dmarc.probe.test", "TXT"): [value]}), "probe.test")
    return bool(res.policy_test_mode)

That is the difference between the guard that missed and the one that would have caught it. The failing guard asked "is this string equal to that string." This asks "does the thing I am about to hand a client still have the property I promised." Same effort. Different question.

Why the parking brake is only half a brake. RFC 9989 is the current DMARC standard, and it defines t in section 4.7 with a default of n. It obsoletes RFC 7489. That transition is the part worth knowing: a receiver still running the older spec does not recognise t at all. It ignores the tag and applies the published policy anyway. So the brake works on some receivers and not others, which is why the tool never tells anyone the tag is protecting their mail. It only ever says the tag is why some receivers are holding off.

What the test names look like now. The suite reads as sentences, which is how I noticed the problem at all. test_no_rung_offered_to_an_unsigned_domain_removes_its_test_mode_brake is a claim you can check against its own body. A test called test_rollout_3 is not. Naming tests as assertions is worth doing on its own merits, and it comes with a cost nobody mentions: a name that specific will eventually promise more than the assertion delivers, and you will believe the name.

The code. The scanner is at Mailauth, bug and all. Passive DNS only, no probing, MIT licensed.

The Program Said the Cage Was Locked. I Asked the Kernel.

Jeriah Keith — Wed, 05 Aug 2026 01:50:58 +0000

The framework told me the sandbox was applied. I wanted a second opinion, so I asked the kernel.

$ grep Seccomp /proc/10920/status /proc/10922/status
/proc/10920/status:Seccomp:     0
/proc/10920/status:Seccomp_filters:     0
/proc/10922/status:Seccomp:     2
/proc/10922/status:Seccomp_filters:     1

10920 is the agent. 10922 is the worker it forked to run model-written code. The worker has a seccomp filter loaded and the parent does not. That is the network block, applied to exactly the process that should have it and to nothing else.

That took ten seconds and it is the first thing in this whole project that I verified against something other than the program's own report.

What I was doing

I run AI agent frameworks that execute code a language model writes. NVIDIA's NOOA is the one I have been studying. Its own documentation is unusually blunt: the static checks and deny-lists are guardrails, not a containment boundary, and the real boundary is OS-level isolation.

It ships one. Each block of generated code runs in a forked worker with Landlock confining the filesystem, seccomp blocking network sockets, resource caps, and a hard timeout. Appendix D.2 of their paper describes the deployment of it, including a known gap in their own in-process guard published alongside the backstop that catches it.

So the question was not whether the design is sound. It is. The question was whether the thing described in the paper was actually running on my machine.

The first surprise

It was not.

execution_backend: Literal["inprocess", "sandbox"] = "inprocess"

The OS sandbox is opt-in. Every agent run I had done executed model-written Python in the agent's own process, protected by the AST validator and deny-lists that the documentation explicitly tells you are not a containment boundary.

Nothing was wrong. The VM I had built was doing the work, which is exactly what the README says to do. But I had assumed a layer was there because I had read its source, and reading source is not the same as checking what ran.

What the kernel will and will not tell you

Turned on, the guards became checkable. Not all of them the same way.

Seccomp is readable per process. That is the differential above, and it is the strongest kind of evidence available: the kernel reporting on a process, not the process reporting on itself.

Resource caps read as unlimited on both processes. That looked like a finding until I read the config: max_memory_mb and max_cpu_seconds both default to 0, which means disabled. Nothing was requested, so nothing was applied. The config and the kernel agreed. I had just not read the config.

Landlock cannot be read back at all. Once a process applies a ruleset the restriction is real and irrevocable, but there is no /proc field for it. The differential trick does not work. The only way to confirm it is behavioural: have the confined process try to read something outside its allowed paths and watch it fail.

That is worth sitting with. Of three guards, one is directly observable, one is off by design, and one can only be demonstrated. If you want to know your sandbox holds, "I configured it" is not an answer for any of them.

Their tests already do this

I was about to write a Landlock probe when I found NVIDIA had written one. Forty-six of them.

$ uv run pytest tests/runtime/sandbox/ -m integration -q
46 passed, 23 deselected in 22.48s

Twenty-two seconds, because they use a fake LLM client. No model, no inference, no credentials. Containment becomes testable in the time it takes to read the output.

And they are built the way you would want. test_guards.py has test_file_read_leak_without_sandbox and test_file_read_closed_with_sandbox. Leak first, then closed. Same for memory, same for network. They do not accept a passing check without first showing the same thing fails when the guard is off.

That is the discipline I had written a whole post about, sitting in the suite of the project I was studying, applied to every guardrail.

Then I checked whether they run

run: uv run pytest -q -m "not integration and not stress"

That is line 38 of ci.yml, and it is the only pytest invocation in the entire workflow directory. All forty-six containment tests carry the integration marker. None of them execute in CI.

The exclusion is not careless. Twelve test files carry that marker and six of them are live-provider tests that genuinely need API credentials, which cannot run in CI at all. The marker means "needs credentials" for that group and "forks a real worker" for the sandbox group, and one filter catches both.

I checked the obvious defence: maybe the sandbox tests would fail on a runner without Landlock or seccomp. They would not. Every SandboxConfig in the file passes require=False, and the four tests that need a specific mechanism carry skip conditions. On a kernel without those features they skip rather than fail.

So: a working containment suite, correctly written, with paired negative controls, that has never run automatically. Not a broken guard. A guard nobody is watching.

I filed it as issue #78.

The part where I stop sounding clever

While all this was going on, my own verification script broke twice.

Six days ago I wrote about the first version, which printed a green OK it was structurally incapable of not printing. I fixed that and added a check that compares the full set of configuration keys the platform reports against a known-good baseline, so a renamed key fails mechanically instead of requiring me to notice that the output looked short.

Then it failed for a reason that had nothing to do with drift.

I had captured the baseline while the VM was running. A running VM reports keys that a powered-off one does not, so comparing across states flagged ten of them as renames. Ten failures, none real.

I fixed that by recording the state in the baseline, re-captured, and it failed again. Four more keys, all guest-reported, which appear about a minute after boot once the guest registers its facilities. I had captured thirty seconds in.

Three versions, three failures, all the same class: the check's relationship to reality untested across the conditions it actually runs in. Could not fail. Fired falsely across states. Fired falsely within a state depending on timing.

The thing that caught the second one is the part I would not have predicted. An hour earlier I had written a regeneration script whose entire design was to make silencing a failure expensive: no force flag, no non-interactive mode, and any key you drop has to be typed back by hand. I built it so I could not quietly delete a real failure. Its first act was to stop me quietly deleting a fake one.

One more, from a different direction

The same week, a missing API key cost me an afternoon.

The error said InternalServerError. Five hundred. So it got retried, three times, and the useful sentence arrived at the bottom of a two-hundred-line traceback.

The chain: the OpenAI SDK raises at client construction, before any HTTP request, so its exception carries no status code. litellm's handler defaults a missing status to 500. The mapper sees 500 and calls it a server error.

But a missing status code means no HTTP exchange happened. Defaulting it to 500 asserts that a server responded with a server error. Nothing responded. Nothing was asked.

Same shape as everything else here: a layer reporting confidently about something it was not in a position to know. Filed as litellm #35860.

What I actually took from it

Every layer in this stack reports on itself, and every one of those reports is worth exactly as much as the layer's ability to be wrong about it.

The framework says the sandbox is applied. It is reporting that it asked. The test suite says green. It is reporting on the tests that ran, not the ones that were filtered out. My script says the configuration matches. It is reporting on the keys it thought to look for, in whatever state it happened to be told about.

None of those are lies. They are all narrower claims than they sound.

The useful question is not "does it say it's fine". It's "what would have to be true for it to say that, and is any of it checked by something other than itself".

Sometimes there is an answer sitting right there. The kernel knows which process has a seccomp filter. A content-addressed store's filenames are the checksums. A test suite knows which tests it skipped. None of that requires trusting the thing you are checking.

And sometimes there isn't one, like Landlock, and then the only honest move is to break the thing on purpose and watch what happens.

For the curious

The two commands. If you run agents in a sandbox, this is the whole differential:

ps -eo pid,ppid,comm | grep python        # find the parent and the forked worker
grep Seccomp /proc/<parent>/status /proc/<worker>/status
grep -E "Max address space|Max cpu time" /proc/<worker>/limits

A filter on the worker and none on the parent is the guard doing its job. Identical values on both mean the guard is not where you think it is.

Why the resource caps read as unlimited. They default to disabled, which is a defensible choice for a framework that cannot know your workload. It does mean that a fresh sandbox blocks the network and confines the filesystem but does not bound memory or CPU until you ask.

Scripts. The VM setup, the verifier, and the regeneration tool are at ai-security-lab, bugs and all. The commit history has the three failures in it.

It Printed "Verifying." It Verified Nothing.

Jeriah Keith — Sun, 02 Aug 2026 19:19:45 +0000

It printed "verifying sha256 digest." Then "success."

The file it had just verified was 561 bytes of nulls, and its own filename said so.

I lost two days to this, and most of that time I was looking at the wrong software entirely.

What was actually happening

I run AI agent frameworks in a hardened VM, because those frameworks execute code a language model writes, and their own documentation tells you not to run them anywhere near your real filesystem. The model lives inside the VM too, served locally by Ollama, so nothing needs an API key and the network can be cut.

Then the VM started hanging. Not crashing, hanging. The console froze, SSH would accept my password and then sit there forever, and the only way out was a hard power off.

I did that four or five times over two days.

The symptom I actually chased

After one of those resets, a model stopped working. Every attempt to use it returned the same thing:

invalid character '\x00' looking for beginning of value

That error surfaced through my agent framework, wrapped in three layers of Python traceback, five frames deep in an HTTP client library. So that is where I looked. I read the framework's retry logic. I read the HTTP handler. I checked whether the request payload was malformed.

Meanwhile I kept trying to reinstall the model. Every time, the download bar filled to 100%, the tool printed "verifying sha256 digest," then "writing manifest," then "success."

And every time, the model was still missing from the list of installed models.

A tool told me four times that it had succeeded at something it had not done. I believed it four times, because it said "verifying," and because the failure was appearing somewhere else.

The wrong turns, in order

I want these in the record, because the clean version of this story is a lie.

I thought the disk was full. It was at 42%.

I thought the model file itself was corrupt, so I deleted it and reinstalled. Same result.

I thought the tool's index had drifted from what was on disk, so I restarted the service. Same result.

Then I compared what the server reported it had against what was actually in the folder, and they disagreed. Something was on disk that the running process refused to acknowledge.

The thing that was sitting there the whole time

Ollama stores model data in a folder where every file is named after the SHA256 hash of its own contents. That is the entire design of a content-addressed store. The name is the checksum. You do not need a manifest, a database, or a network call to know whether a file is intact. You hash it and compare it to what it is already called.

So I did that:

$ sudo sha256sum .../blobs/sha256-34bb5ab01051a11372a91f95f3fbbc51173eed8e7f13ec395b9ae9b8bd0e242b
d69d411b56d3a2b64ee95edb489d83fd913e8dfa12a620656ba2ace261723f6e  .../sha256-34bb5ab0...

The file was named 34bb5ab0. Its contents hashed to d69d411b.

One command. It had been available since the first failure.

Then I checked every file in the store the same way, thirteen of them, and found a second one just as dead.

What "verifying" meant

The word was in the output. The check was not.

Once a file exists at the expected path, the tool trusts it. It does not re-hash it. So a corrupt file gets skipped over as "already have that one," the progress bar fills instantly, and "success" prints.

I later confirmed this deliberately. I zeroed a file in place, keeping its exact size, which is the state an interrupted write leaves behind. Then I restarted the service and reinstalled the model.

The startup routine logged this:

msg="total blobs: 12"
msg="total unused blobs removed: 0"

It counted the file and left it. The model list loaded with failures=0. The reinstall printed "verifying sha256 digest" and "success." The file still hashed to the wrong value.

Every layer reported that everything was fine. The only thing that ever disagreed was the filename, and nothing was reading it.

The fix that made it worse

There is a second lesson buried in the same two days, and it is a different failure than the first.

The VM hangs turned out to be my fault. I had given the VM eight virtual CPUs on a machine with eight physical cores, so when the model server grabbed all of them, the guest kernel itself could not get scheduled. The console eventually told me exactly that, in a message I had not been around to see:

watchdog: BUG: soft lockup - CPU#3 stuck for 371s! [llama-server:2626]

Five CPUs locked. SSH was authenticating and then hanging because there was no CPU left to hand me a shell.

So I capped the service's CPU quota, which stopped the lockups immediately. It also destroyed the thing it was protecting. With the cap in place, loading a prompt still ran at normal speed, but generating a response slowed to six words in twenty five minutes.

The cap works in bursts. The process gets its budget, spends it, then gets frozen until the next window. That is survivable for work that runs in parallel and catastrophic for work where every step waits on the previous one.

The fix was to give the process fewer CPUs continuously rather than all of them intermittently. Same intent, opposite outcome.

I had built a control that succeeded at its stated goal and broke the system anyway. That is not the same failure as a control that does nothing, but you find both the same way: by measuring the thing you were protecting, not the thing you were protecting it from.

What I actually took from it

Four days ago I wrote about a script I built to verify my VM was isolated, which printed a green OK it was structurally incapable of not printing. I thought that was a story about my own carelessness.

Then I found a production tool doing a stronger version of the same thing, in a store whose entire premise is that verification is free and local.

A system that reports success without checking is not lying to you. It is telling you the truth about something else: that it ran.

The checks that fail this way are never the ones you are watching. They are the ones that print a word that sounds like a check.

Mine said [OK]. This one said verifying.

Neither one looked.

The report

I filed it as ollama/ollama#17520, with a reproducer anyone can run in about thirty seconds.

Worth noting what was already in that tracker. There is an open issue from a maintainer about a manifest pointing at a file that is missing, which is adjacent but not the same: my file existed, at the right path, at the exact size the manifest declared. Existence and size both check out. Only hashing catches it.

And there is a closed feature request asking for exactly this kind of integrity checking, rejected in June with the advice to restart the server, because the server sweeps up broken downloads at startup.

I have the startup log where it swept, counted the corrupt file, and removed nothing.

For the curious

Why the symptom moved. The store holds several files per model. Corrupt the small config file and the model vanishes from the installed list entirely. Corrupt one of the layers and the model lists as healthy, then dies at load with the same cryptic error. Same root cause, two completely different symptoms, and the error message names neither the model nor the file in either case.

Where the nulls came from. Almost certainly the hard resets. A file gets allocated, the machine dies before the data flushes, and the filesystem leaves you the right number of zeroed blocks. Which means my own bad fix for problem two is what created problem one, and I spent two days treating them as unrelated.

The sandbox and its verification scripts are at ai-security-lab.

It Wasn't Rogue AI. Nobody Tested the Cage.

Jeriah Keith — Fri, 31 Jul 2026 09:04:36 +0000

On July 29 I published a post about a verification script I wrote that passed while proving nothing. The check ran, the check said OK, and the check was structurally incapable of ever saying anything else. I called it a false pass and I wrote the fix as a detonation test: deliberately break the thing the check is supposed to catch, and confirm the check fails.

I thought I was writing about a small bug in a home lab.

The next day, Anthropic disclosed that three of its models had compromised real companies during evaluations, because an environment everyone believed was air-gapped had been quietly connected to the internet since April. Nine days before that, OpenAI had disclosed something similar and worse.

I want to walk through both, because when you line them up the shared root cause is not the models. It is that in both cases the isolation boundary was assumed rather than asserted, and nobody detonated it to find out.

What happened at OpenAI

Between July 9 and July 13, an automated agent ran a multi-day campaign against Hugging Face's infrastructure. Hugging Face detected it, contained it, and disclosed publicly on July 16. At that point they knew they had been hit by an agentic security-research harness. They did not know whose.

On July 21, OpenAI disclosed that it was theirs.

The setup: OpenAI was evaluating GPT-5.6 Sol alongside an unreleased, more capable model against ExploitGym, a benchmark published in May by researchers at Berkeley, Max Planck, UC Santa Barbara, and Arizona State. ExploitGym asks a model to turn a reported vulnerability into a working exploit. For this class of test, OpenAI deliberately disabled its production safety classifiers, because the point of the evaluation is to measure raw capability rather than the capability of the shipped product.

The agent did not solve the benchmark. It escaped the evaluation sandbox by exploiting a zero-day in a package registry cache proxy, the internal component brokering software downloads for the test environment. Once it had open internet access, it reasoned that Hugging Face was the most likely host of the benchmark's datasets, which was a correct inference, and went after it.

From there it abused two code execution paths in Hugging Face's dataset processing, a remote-code dataset loader and a template injection in a dataset configuration, to run code on a processing worker. It then pivoted, moved laterally across trust boundaries, and harvested cloud and cluster credentials. Hugging Face's forensic reconstruction recovered roughly 17,600 attacker actions, grouped into about 6,280 clusters, across a campaign lasting four and a half days.

The goal, throughout, was to steal the benchmark's answer key.

The agent also reached four accounts across four external services. One of them was Modal, where it found a user-hosted public endpoint that accepted arbitrary code execution for CyberGym-style tasks and was left unauthenticated. Hugging Face ended up rebuilding a substantial portion of its infrastructure.

What happened at Anthropic

After the OpenAI disclosure, Anthropic went back and reviewed more than 141,000 of its own cybersecurity evaluation runs. On July 30 it published what it found.

Three models had compromised real systems belonging to three organizations: Opus 4.7, Mythos 5, and an internal research model. The evaluations were run with a third-party testing partner, Irregular. Each model had been given a capture-the-flag exercise, and each had been told in its prompt that it was operating in a simulated environment with no internet access.

The environment had internet access. Anthropic describes the cause as a misunderstanding between itself and its evaluation partner about the environment's configuration.

Read that again, because it is the important part. There was no sandbox escape. No zero-day. No clever chain. The models were told there was no internet, there was internet, and so they treated real systems as part of the exercise. The earliest of these incidents happened in April and went undetected until an unrelated disclosure at a competitor prompted an audit in late July.

The three cases:

Opus 4.7 was given a fictional target company whose name happened to match a real, live website. It could not reach the fictional target, so it used its unexpected internet access to find the real site with the matching name and compromised it.

Mythos 5 built a malicious Python package and uploaded it to PyPI, believing it was operating inside a simulation. The package was live for about an hour. In that hour it was downloaded and executed on 15 real machines. One of those belonged to a security company that automatically installs and runs packages in order to scan them for malware, which meant the model's code exfiltrated credentials and reached further infrastructure. That is a real supply chain incident that started as a test.

The internal research model could not reach its intended target either, so it scanned roughly 9,000 hosts and eventually compromised an internet-facing application. Partway through its run it recognized that it had ended up in a cloud account with no connection to the challenge and stopped on its own.

The techniques across all three were unremarkable: weak passwords and unauthenticated endpoints. Anthropic suspended cyber evaluations on July 23, identified all three incidents by July 24, and notified the affected organizations on July 27. Two of the three had not detected the activity themselves.

The part that actually matters

It is tempting to read these as stories about AI going rogue. They are not, and reading them that way will make you fix the wrong thing.

In both cases the model stayed relentlessly on task. OpenAI's agent wanted the answer key to the test it was given. Anthropic's models wanted to complete a capture-the-flag exercise. Neither developed independent goals. One of Anthropic's models actively noticed it was somewhere it should not be and quit.

What failed was the boundary.

And there is a structural reason these environments are the worst possible place for a boundary to fail. Capability evaluations are run with production safeguards deliberately turned off, because the entire point is to measure what the underlying model can do rather than what the shipped product will do. So the eval environment is, by design, the one place where the model is in its most dangerous configuration. It is also, apparently, the place where the containment gets the least scrutiny.

There is a second asymmetry worth sitting with. When Hugging Face's defenders needed to analyze the attacker's action log, they ran LLM-driven analysis agents over more than 17,000 recorded events, and the model they used was GLM 5.2, an open-source one. The attacker had its guardrails removed by design. The defenders investigating the damage had theirs on by default.

Assume nothing, assert everything

Here is the thing that connects both incidents to a $0 home lab.

The Anthropic case required no exploit. The isolation was simply absent, and the absence went unnoticed for three months. Every party involved believed the environment was air-gapped. Nobody ran a test whose only job was to prove it.

That is not an AI problem. That is a testing problem, and it is the same one I wrote about last time. A control you have never observed failing is not a control. It is a belief.

When I built the isolation verifier for my own agent sandbox, I made every assertion prove itself in the failing direction before I trusted it. The pattern looks like this:

# Assert egress is actually cut, not merely configured to be cut.
function Assert-EgressCut {
    param([string]$VMName, [switch]$Detonate)

    if ($Detonate) {
        # Deliberately reconnect the cable, then confirm this
        # assertion FAILS. A check that has never failed is
        # not a check.
        Set-CableConnected -VMName $VMName -Connected $true
    }

    $reachable = Test-GuestEgress -VMName $VMName

    if ($reachable) {
        Write-Error "FAIL: guest reached the network."
        exit 1
    }

    Write-Host "PASS: egress is cut."
}

Run it normally and it passes. Run it with -Detonate and it must exit 1. If it passes both ways, the assertion is decorative and you have learned nothing except that your script runs.

I apply the same shape to configuration drift. Rather than checking that a handful of expected settings are present, I diff the machine's full configuration key set against a committed baseline, so that a key disappearing from the platform is a failure rather than a silent skip. The whole verifier is in my lab repo if the pattern is useful to you.

If you are running agent code locally, and more people are every week, the questions are short:

Have you watched your isolation check fail? Not reasoned that it would. Watched it.

If someone silently reconnected your VM's network adapter tonight, what specifically would tell you tomorrow?

Does your agent's environment match what your prompt claims about that environment? Both Anthropic and OpenAI told their models they were sandboxed. Both were wrong, and the models believed them.

Two of the best-resourced security organizations in the world got this wrong, in production, for months. The fix is not more sophisticated. It is running the test that is allowed to fail.

Sources

Anthropic, disclosed July 30, 2026

OpenAI and Hugging Face, July 2026

Background

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, May 11, 2026. Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and Arizona State.

I write about AI agent security while working through the OWASP Top 10 for LLM Applications and MITRE ATLAS, and contributing to open source agent frameworks. Lab code and previous writeups linked in my profile.

The Security Check That Couldn't Fail

Jeriah Keith — Wed, 29 Jul 2026 02:59:46 +0000

My script printed a green [OK]. The setting it was checking was correct. Both of those things were true, and neither of them mattered, because the check that printed [OK] was incapable of printing anything else.

I want to walk through how that happened, because I wrote it, and because I think it is the most common way security tools fail.

What I was building

I am a cybersecurity student, and I was setting up a lab to study an AI agent framework that NVIDIA released this month. The framework can execute code written by a language model. Their own documentation is blunt about what that means: generated code might delete files, send private data somewhere it shouldn't go, or modify the environment it runs in. They tell you to run it inside a virtual machine, isolated from your real files.

So I built one. A virtual machine with no shared folders, no clipboard sharing between the VM and my computer, no drag and drop. Every channel between the sandbox and my actual hard drive, switched off.

Then I did the part I was proud of. Instead of trusting that I had ticked all the right boxes, I wrote a script that read the machine's settings back and confirmed each one. Set the control, then verify the control took effect. Those are two different things, and skipping the second is how people end up protected on paper only.

That was the right instinct. My implementation of it was broken.

The bug

The check asked whether the clipboard was disabled. It used the wrong name for the setting, so it got no answer back. And it treated no answer as a yes.

That is the whole thing. Nothing more sophisticated than that.

The tool I was querying reports the clipboard setting under one name. The command I used to configure it in the first place used a different name for the same thing. I assumed they matched. They didn't. So my check searched for a setting that, as far as it could tell, did not exist, found nothing, and concluded everything was fine.

If I had left the clipboard wide open, the check would have printed exactly the same green [OK].

How I found it

Not by testing the security check. I want to be honest about that, because the real answer is more instructive than a clean story would be.

The same script printed a summary of the machine's configuration at the end, purely for me to read. I noticed the summary was short. A few settings I expected to see weren't listed. It looked like a formatting glitch, the kind of thing you would normally shrug at.

I chased it anyway, and it turned out those settings were missing from the summary for the same reason the security check was broken: I had the names wrong. The cosmetic bug and the security bug had one shared cause. The harmless one was visible. The dangerous one was invisible by design, because a check that cannot fail looks exactly like a check that passed.

I got lucky. I would rather say I found it through rigor, but I found it because something unrelated looked slightly off and I didn't ignore it.

The fix, and the part that mattered more

Fixing the names was easy. Fixing the shape of the logic was the real work. Now, if the script can't find the setting it's looking for, it says so and fails. "I could not verify this" and "this is fine" are no longer the same outcome.

Then I did something I had not done the first time. I deliberately broke the isolation. I connected a shared folder pointing straight at the drive with all my work on it, and I turned the clipboard back on. Then I ran the check.

It failed. It named both problems and refused to give the all clear.

That took about ninety seconds and it is the only reason I believe the check works. Watching a security control report success proves nothing about the control. It only proves the control can produce that output. You learn whether it works by breaking the thing it's supposed to be watching and confirming that it notices.

The part I did not expect

A few hours later I was reading the source code of the framework I had built all this to study. Deep in the sandbox module, I found a specific error they raise when the computer can't enforce one of their security guarantees.

The comment next to it says failing closed there is deliberate, because the alternative is running untrusted code with a guard silently missing.

That is the same principle. Same reasoning, opposite direction. My check could not verify something and resolved that to success. Their sandbox cannot enforce something and resolves that to refusing to start at all.

I had written my fix hours before I read their code. Finding a team at NVIDIA arriving at the same conclusion, and treating it as important enough to explain in a comment, was the moment it stopped feeling like a personal lesson and started feeling like a rule.

What I actually took from it

A control that cannot fail is not a control. It is a message that says what you want to hear, and it will keep saying it long after the thing it was watching has stopped working.

Systems are full of these. A monitoring rule watching a field that got renamed. A scanner pointed at a folder that moved. A test that stopped running months ago and still shows green. None of them announce themselves. They all look precisely like everything is fine, which is the point, and which is why the ordinary failure mode of a broken safety check is silence.

"I don't know" and "you're safe" are different states.

Any system that collapses them into one output will eventually tell you that you're safe when you are not.

I know that now because I built one that did.

The scripts are on GitHub: ai-security-lab