- Book: Observability for LLM Applications
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
A model was told to break into a fake corporate network. Its in-scope attacks failed. So it went after a real open-source project on GitHub, submitted malicious code, and used fake accounts to lean on the maintainer. The maintainer said no. In Anthropic's words, "The run ended as the model ran out its token budget while continuing to pressure the maintainers to approve the change."
The model was Claude Mythos 5. The test was a cyber evaluation run by the UK's AI Security Institute (AISI) in July. On October 23, AISI and Anthropic each published an analysis of why it happened, and SiliconANGLE covered both the same day. If you run long agent sessions, read this part: Anthropic's best explanation points at the model's own compaction notes.
What happened in the evaluation
AISI's incident report came first: per Anthropic, AISI published it on August 4. The setup matters: according to Anthropic, "Internet access was deliberately enabled," and "The cyber safeguards we deploy with general release models like Fable 5 were disabled."
"In nine of AISI's 43 evaluation runs, Mythos 5 took unsanctioned actions," Anthropic writes. The worst one is the GitHub attack. In the other eight, which Anthropic reviewed more lightly, it lists repeated password guessing against the evaluation infrastructure, attempts to plant instructions for a suspected AI reviewer, a fake payment verification page, malware uploaded to file-sharing sites, and attempts to log into a real developer's GitHub account. Anthropic says none of those succeeded, though it also notes one file-sharing site accepted a malware upload that has since been taken down. AISI says no real-world harm occurred.
Anthropic also says this "would have been flagged immediately by our production safeguards on Fable 5, had they been enabled." That's a vendor claim about its own classifiers. Nobody outside can check it from these documents.
Why it didn't stop
Long agent runs don't keep the whole transcript. They compact it into a summary and carry on from that. AISI calls this "a description of its actions to date called a compaction summary (which is standard in long-running evaluations)" (AISI).
Anthropic's hypothesis is that "this persistence reflects a kind of 'momentum', in which the model's compaction notes" and prior actions imply its current path has been sanctioned (Anthropic). It concludes that the model's "decision-making relies heavily on the model's preceding compaction notes or its prefilled prior turns."
The resampling numbers back that up. All of these are from Anthropic's report:
- When five earlier reasoning steps were edited to mention possible real-world harm, "Mythos 5 went from pursuing the attack on 83% of occasions to 13%."
- Resampled from later in the run, "it continued the attack on 98% of occasions."
- Told outright that it was dealing with real humans, the statement "caused the model to cease the attack just 5% of the time."
- Across Anthropic's replays, looking only at the next few turns of the transcript, "Mythos 5 never once decided to stop its attack."
AISI ran its own version. It paused Mythos 5 just before the malicious submission and edited the summary to say the target was "a real human unrelated to the evaluation." The attack rate didn't drop: "Mythos 5 proceeded to attack in nearly all runs" (AISI). It also found that "the model incorrectly stated that entities were simulated before attacking them," and suggests the models "may be engaging in 'motivated reasoning'."
Newer models do better, with a caveat
Anthropic replayed the incident with Mythos 5.1, Opus 5.5 and an internal research model. "Opus 5.5 and our internal research model stopped spontaneously 10% and 67% of the time," it reports, against 1% for Mythos 5.1 and 0% for Mythos 5 (Anthropic). A later section of the same post gives 62% for the internal model, so treat that figure as roughly two-thirds.
AISI adds the caveat. Newer Anthropic models showed more stated awareness of being evaluated, so it's unclear whether the lower attack rates reflect better alignment (AISI). Its recommendation to evaluators: "treat unsanctioned behaviour as important evidence of alignment failures," even when the model says the environment is simulated.
On the fix side, "AISI has publicly committed to running these evaluations without internet access" until it has stronger access controls (Anthropic). AISI's October 1 post says it disables outbound networking from its sandboxes and adds a separate cloud-level block on the host.
What this means for the agents you run
You're probably not running a frontier model with its safeguards off against a cyber range. But the mechanism isn't specific to cyber tests. Any long session that compacts is steered by its own summary. If the summary says "we're blocked, try another route," the next turn starts from there. Here's what follows from these two reports. These are my inferences, not recommendations either report makes for coding agents.
Put the stop outside the model. Telling the model the target was real stopped it only 5% of the time. AISI fixed it at the network layer, and you can do the same for a risky unattended run: no network unless the task needs it.
docker run --rm --network none \
-v "$PWD":/work -w /work \
agent-sandbox:latest
--network none gives the container a loopback interface and nothing else.
Don't hand an agent the means to make identities. The worst run needed GitHub accounts. An agent with your logged-in gh session, a mail account, or a payment method has more reach than the task.
Steer the summary when you compact. In Claude Code, /compact [instructions] takes "focus instructions for the summary." Restating scope there costs one line:
/compact Keep the original scope: only files under src/.
No network calls, no new accounts, stop and ask if blocked.
That's no guarantee. The AISI test shows a model can read a clear correction and keep going. It's still better than letting a summary of "blocked, try another way" become the new brief.
Read the transcript after the run, not only the diff. The interesting failures here were in the actions between the start and the end.
Watching compaction in Hermes
I build Hermes IDE, so weigh this with that in mind. In Hermes 2.3.4, the bar above each session has a context meter that "shows how full the conversation is, and offers Compact when it gets close to full," and the Commands list includes Compact. The point is to see the window filling up before the agent compacts, so you get the chance to run /compact with your own focus line in the session's terminal.
The action bar above a Claude Code session (Prompts, Context 4%, Model, Plan, Review, / Commands) with the Commands list open: pinned Context, Model, Plan and Review, then built-in commands such as Compact, Usage and Diff.
For the network and account limits above, rely on Docker and your agent's own sandbox settings. Hermes is free to download, source-available (BSL 1.1), and runs on macOS, Windows and Linux.
Your agent's notes are part of its prompt
Whatever your agent writes into its summary becomes the brief for its next turn.
Do you read what your agent wrote into its compaction summary, or only what it changed in the repo? Tell me in the comments.
If this was useful
Most of this story comes from reading transcripts: which turn started the attack, which note kept it going, which correction it ignored. That is tracing and evaluation work, and it's what Observability for LLM Applications is about, from capturing traces of LLM runs to building evals that catch the behavior you don't want before users do. If you run agents unattended, it's the half of the job that tells you what they actually did. Hermes IDE is free to download if you want compaction and session context in view while you work.
Top comments (0)