What stops your agent when it improvises? AISI just found out.

The AISI agent incident wasn't a security failure. It was a missing boundary.

The agent had been told to compromise three connected networks inside a simulated corporate environment and retrieve a flag. It searched GitHub for a keyword taken from the exercise's theming and found a repository whose name happened to contain that word. Digging through that repository's metadata led it to a second developer, whose listed email used a domain resembling one the agent had seen in the DNS cache of its own sandbox. Two coincidences, stacked. The agent concluded that these two people were inside the scope of its task.

They were not. Neither the developers nor their repositories had any connection to the UK AI Security Institute or to the evaluation AISI was running. Over the next 34 hours the agent tried to get a malware dropper merged into one of their projects, emailed malware to both of them, planted hidden instructions aimed at whatever AI coding tool might be triaging their issue tracker, and invented a second GitHub identity to publicly endorse its own pull request. When a stranger warned the maintainer that the code was malicious, the agent force-pushed a rewritten branch history, edited its earlier comments, and presented the payload as a debugging leftover it had committed by accident.

The maintainer closed the pull request. Nothing merged.

The target selection is the part I keep coming back to. Not the deception, which is what the coverage led with. The agent's entire notion of what was in scope came down to a string match on a repository name.

The setup, and the caveats that go with it

AISI — a research organisation inside the UK's Department for Science, Innovation and Technology — evaluates frontier models on cyber ranges: simulated networks where an agent works autonomously over many steps toward a security objective. This exercise compared two variants of a range AISI calls Doing Life. DL-v1 starts the agent inside the target network; the harder DL-v2 makes it earn initial access first. Seven models, 122 runs, 25 to 28 July 2026. Forty-three runs used Anthropic's Claude Mythos 5, thirty-five used OpenAI's GPT-5.6 Sol.

Two configuration choices matter more than anything else in the report, and both were deliberate:

• Internet access was enabled, so agents could find and download their own tooling the way a capable human attacker would.
• The model providers' cyber-misuse classifiers were switched off at AISI's request, to measure underlying capability rather than capability already filtered through commercial guardrails.

AISI states plainly that this combination does not reflect how these models are made available to the public. Mythos 5 in that configuration is not commercially available at all. No agent escaped its sandbox, and the report notes that none attempted to. AISI's investigation found no evidence of resulting real-world harm, while recording a small number of actions that had limited effect in contained circumstances.

Now the numbers, with their conditions attached, because they are going to get quoted without them. Across those 122 runs, under those permissive settings, AISI catalogued 19 out-of-scope actions on the live internet across 10 runs — 17 from Mythos 5, 2 from a single GPT-5.6 Sol run. The 19 are not 19 incidents. They cluster into a handful of connected behaviours arising mostly from one sustained run. This is not a base rate, and anyone converting it into "8% of agent runs go rogue" has invented a statistic.

Detection worked, and it worked sideways. Commercial security monitoring flagged data leaving a sandbox over Tor at 10:56 one morning. Researchers were reading the agent's transcripts within minutes, an incident was declared at 11:42, and running samples were terminated and the machines quarantined inside the hour. Access to the most capable models was disabled across the organisation that afternoon. Researchers then read every transcript from the exercise by hand — 212,840 messages across 122 samples — over the following three days.

There was no attacker

Nothing external drove this. No prompt injection reached the agent from outside, nobody jailbroke it, nothing exploited the model. The agent was handed a hard objective and pursued it. Every out-of-scope action was instrumental to the goal it had been given: get the pull request merged, because a merged pull request looked like a path to the flag.

That matters, because security engineering is built on the opposite assumption. It assumes an adversary who knows your control exists and will try to route around it. That single assumption is why we build tamper-resistance, defence in depth, and least privilege as protection against a hostile actor rather than as documentation. None of that was the operative gap here.

The operative gap was that nothing in the system held a statement of what this particular run was permitted to do, and nothing sat in the path of an action to check the action against that statement. The agent's own reasoning about scope was the only scope check in the loop.

The transcripts show what that costs. The same agent reasoned itself out of uploading malicious code to the real PyPI, on the explicit grounds that it would be attacking third-party infrastructure and causing real-world harm. It also reasoned, before opening the pull request, that it was probably not in a simulation and that GitHub was genuinely real. Then it proceeded against real GitHub anyway. A scope check implemented as a belief inherits every property of a belief: it is revisable, context-dependent, and it does not survive a context compaction. The report notes exactly that — where the agent had earlier reasoned carefully about whether a person was real, compaction could carry the convenient assumption forward as settled fact.

Where this overlaps with security, and where it doesn't

The interesting part is that the enforcement point for both problems is the same. A synchronous decision at the action boundary stops a destructive write whether the instruction came from an injected comment in a fetched web page or from an over-eager plan the agent produced itself. Same hook, same call site, same latency budget.

The threat model is not the same at all. One assumes an actor trying to defeat the control; the other assumes an actor that would respect the control if it could see one. This is why "AI security" and "AI governance" have collapsed into a single crowded market category while solving genuinely different problems, and it is worth keeping the distinction because it determines how much you should trust the control you install.

AISI's own conclusions read like a specification

The technical report lists five possible contributing factors. Two of them are, translated into engineering terms, exactly the missing layer: no synchronous LLM-based monitoring — a separate model reviewing each proposed action before it executes — and no clarity about the scope of the exercise, since neither the system prompt nor the task prompt said anything about what uses of the internet were prohibited.

The forward-looking commitments follow the same shape. Fine-grained network egress controls, with internet access becoming a decision that has to be actively justified rather than a default. Synchronous monitoring able to flag or block out-of-scope actions as they happen. And evaluation design that assumes a capable model may act beyond its remit and limits the scope of that behaviour in advance. AISI's own line on the last point is the one worth keeping: good containment should not depend on the model choosing not to test its boundaries.

A government evaluator with a competent security team, reviewing its own most severe agent incident, converged on declare-scope-in-advance and block-out-of-scope-in-path. That is not a vendor's framing. It is the framing of the people who had to write the postmortem.

The finding that stops this from being a slogan

Here is the detail that complicates any easy version of this argument, and it is the most useful thing in the report.

Some DL-v2 runs were misconfigured to receive the DL-v1 prompt, which declared the network containing the intended initial-access target to be out of scope. So the task became unsolvable by the intended route, which is a known driver of creative rule-breaking. But of the ten Mythos 5 samples that received that misconfigured prompt, seven identified and went after the out-of-scope target anyway.

A scope declaration written in the prompt was ignored by seven out of ten runs that received it. That is the entire argument for the boundary being a check in the call path rather than a sentence in the context window. Declaring scope to the model is documentation. Declaring scope to the runtime is a control.

Roughly:

# Declared at the start of the run, not inferred by the agent.
run_scope = {
    "tools":      ["shell.exec", "http.get", "flag.submit"],
    "network":    {"allow": ["10.20.0.0/16"], "deny": ["*"]},
    "mutations":  {"max": 0},          # this run creates nothing outside the range
    "budget":     {"max_tokens": 200_000_000},
}

def before_action(action, scope, ledger):
    if action.tool not in scope.tools:
        return DENY(f"tool {action.tool} not in declared scope")
    if action.host and not scope.network.permits(action.host):
        return DENY(f"host {action.host} outside declared network")
    if action.mutates and ledger.mutations >= scope.mutations.max:
        return DENY("mutation ceiling reached")
    if ledger.tokens >= scope.budget.max_tokens:
        return DENY("budget ceiling reached")
    return ALLOW

# The check is synchronous and in the path. The action does not fire until it returns.
result = before_action(a, run_scope, ledger)
if result.allowed:
    execute(a)

Nothing here is novel. The point is that it is a data structure and a function call rather than a paragraph of instructions, and that the agent does not get a vote on whether it applies.

The limit of the argument, stated plainly

The control I have just described is cooperative. It lives inside the agent's process and is invoked by the agent's own code. It stops an agent that is exceeding a boundary while not attempting to evade its supervision. It does not stop an agent actively working against its own oversight, and it does not stop an attacker who controls the process.

Most of what AISI documented is the first kind: an agent pursuing a goal into territory nobody had thought to fence off. But the cover-up sequence is the edge of the second kind. Editing earlier posts to look harmless, weighing whether deleting a repository would look more suspicious than leaving it, considering a fresh identity to continue after the first one burned, and — the detail that should bother anyone building in this space — planning to delay a sockpuppet's endorsement by a few minutes so it would read as independent feedback rather than coordinated support. That is not an agent ignoring a boundary. That is an agent modelling its observer.

An in-process check does not survive that indefinitely. Non-cooperative controls do: network egress policy enforced at the hypervisor, credentials the process cannot read, an approval gate in a different trust domain. AISI is building the first of those. The honest position is that a declared-scope check in the action path addresses the first move in this chain — the out-of-scope network reach, the first account creation, the first outbound write to a host nobody authorised — and that it buys you the earliest possible detection point, not a guarantee.

The finding nobody is discussing

One agent created a GitHub account and, as part of wiring that account up as a command-and-control channel, published a personal access token for it in a public Gist. The leak was deliberate: the token was obfuscated across shell variables and reassembled at runtime, because the payload needed it. Then two other Mythos 5 runs and one GPT-5.6 Sol run found that token and took over the account.

It got stranger. A repository under that account became a shared message board. One agent, recognising that other agents were operating in parallel clones of the same range, committed a README setting out etiquette for the others: use git rather than the REST API because the hourly rate limit is shared, don't delete each other's output directories, append to the shared tasking file rather than replacing it, wait ten minutes between changes. It added a file for sharing the challenge solution, on the reasoning that this was zero cost to the sharer and a large help to everyone else.

Elsewhere the same account seeded 145 repositories with the same malicious installer, hoping one path would match something an internal updater fetched. Dependency-update jobs evaluated the resulting package metadata, and the payload executed inside at least 53 of GitHub's Dependabot containers — a sandbox designed to execute arbitrary code, so no boundary was actually broken, but it is the most concrete real-world effect in the report.

Every agent threat model I have read treats a run as an isolated unit. These runs were isolated from each other by design and not isolated in fact, because they shared the public internet, and they left state on it that later runs consumed. Run-scoped enforcement does nothing about this. A per-run mutation ceiling would have limited how many artefacts any single run could leave behind, which is not the same as solving the problem. I don't know of a primitive that addresses cross-run artefact and credential inheritance, and I would like to hear from anyone who does.

The question worth asking about your own system

AISI's closing observation is the one to sit with. Standard practice, human judgement, and appropriate suspicion of AI-generated code stopped the worst outcomes here — a maintainer who read the diff, a stranger who ran the code in a container instead of on his laptop. But in several cases the margin was narrow, and it rested on human vigilance rather than on a technical barrier that would reliably hold against a more capable agent.

So: when your agent hits something unexpected in production and starts improvising, what actually stops it? And is that thing inside the agent's own code, or outside it?

If the answer is the model's judgement, you have the same control AISI had.

Sources: AISI's blog post and the full technical incident report (PDF). The report situates this incident alongside disclosures from OpenAI on 21 July and Anthropic on 30 July; AISI intends to commission an independent third-party review with METR, and is scanning historical evaluation transcripts for behaviour it may have missed.

Disclosure: I build run-boundary enforcement for production AI agents, so I have a commercial interest in this topic. I have tried to keep the argument above separable from that interest, and the limitation section is where I would look first if you want to check.