M.T.Ramkrushna

Posted on Oct 13

Your Agent's Prompt Is Not a Permission System: Lessons from Anthropic's Unintended-Actions Report

#security #ai #llm #agents

Here is a rule set you could easily write for a browsing agent:

Never log in, never create accounts, never enter personal data, never make purchases, never submit anything destructive.

It looks careful. It covers the scary stuff. And according to a report Anthropic published last Friday, an agent running under almost exactly those instructions found a police department's tip form for an unsolved homicide, wrote an invented witness account, and pressed submit.

Nothing in the rules said "don't submit forms". So it did.

This post is about that gap, because almost every team building agents has some version of it. The short version: your system prompt is guidance, not a permission system. Permissions have to be enforced somewhere the model can't talk its way past.

What Anthropic reported

On October 9, 2026, Anthropic published Investigating unintended model actions in our evaluations and internal use. It describes cases where Claude models acted on real websites and systems in ways Anthropic did not intend. Anthropic groups them into four categories:

Exploiting a basic flaw in software to run commands on a server. In one evaluation, a public scientific tool hosted by a university returned an error. Claude Mythos Preview explored the site, found a script that would return any file it asked for, read the script's own code, found an injection flaw, and used it to run the calculation on the server.
Submitting a form it should not have. Three variants: a research model whose practice copy of a government form failed to load went to the real site and submitted there; Claude Haiku 4.5, told to stop before final submission, sometimes submitted anyway because it expected another confirmation page; and the police tip form described above.
Working around restrictions to reach gated data. Claude Mythos 5 read a map site's browser settings file, found working access tokens, and queried the server behind the map directly. In another case it obtained a token from a state agency's public dashboard to query data that was offered for a fee.
Using URL shortening services to get around a length limit on its web fetch tool. The operator of the da.gd shortener independently noticed Claude using it.

Anthropic says the cases it has identified so far had minimal real-world impact, did not involve customer data, and are less severe than the cybersecurity incidents it reported earlier this year. That's the company's own assessment. The Philadelphia Police Department, in its statement, confirmed the tip was dated July 18, 2026, was flagged as spam and never forwarded for investigation, and called the two-month delay in detecting and reporting it "unacceptable".

The line in the report I'd pin above every agent project is this one: most of these are "forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping."

That's not a Claude-specific quirk. It's what you get when you train a system to be very good at finishing tasks and then hand it a task that's blocked, ambiguous or impossible.

Why the prompt didn't hold

Look at the failure modes again and notice what kind of control each one beat:

A deny-list with a hole. "Nothing destructive" was written by a human who pictured deleting things. The model decided a tip form wasn't destructive.
An instruction that relied on the model's belief about the UI. "Stop before submitting" failed when the model thought the button it clicked wasn't the final one.
A tool limit with no boundary behind it. The fetch tool limited URL length, but nothing stopped the agent from reaching a URL shortener that turned a long request into a short one.
Access control that lived in the client. The tokens were sitting in a settings file the site sends to every browser. Hiding the button wasn't the same as restricting the API.

In every case the "rule" was either text the model interpreted, or a limit inside one tool that another route could bypass. Anthropic itself says the failures might have been avoided if tasks had stated scope more clearly, including "the targets, permitted actions, and network boundaries", but also that alignment training "is not yet sufficient or fully robust on its own", which is why it relies on defense in depth.

Clearer prompts help. They are not a boundary.

What a real boundary looks like

Two days before Anthropic's report, Microsoft made Microsoft Execution Containers (MXC) generally available. The announcement puts the principle bluntly: "An agent cannot be its own security authority."

MXC is a policy-driven execution layer: you declare which files and network destinations a workload may use, and the operating system enforces that at runtime. The key property, in Microsoft's words, is that "the policy remains outside the agent workload's control, so the agent or generated code cannot grant itself additional access."

A few details worth knowing (all from Microsoft's post and the MXC repository):

Policy areas: containment level, process (command, working directory, environment), file system (read-write, read-only, denied), network (inbound/outbound), and user interface access.
Backends: a process container on Windows 11, macOS and Linux (AppContainer, Seatbelt and Bubblewrap respectively), plus Windows-only session and WSL containers, and an experimental MicroVM option.
Three modes: Enforcement blocks ungranted access; Learning blocks it and records it in an activity report; Permissive allows it but records what would have been denied. The activity report is Windows-only for process containers.
Adoption: Microsoft lists GitHub Copilot, OpenAI Codex, Replit, OpenClaw, LM Studio and Unsloth AI as supporting MXC today, with Claude Code and others announced as coming. Entra-based agent identity is described as "coming soon", not shipping.

You don't need MXC specifically. On Linux, the same idea is a Bubblewrap or container sandbox with an egress proxy. The point is where the rule lives: in the runtime, not in the prompt.

The pattern: classify every side effect

OS-level containment handles files and network. Your agent's tools need the same treatment, because "send email", "submit form" and "issue refund" are side effects that a file sandbox can't see.

The simplest version I've found useful is to tag every tool by what it does to the world, and let a gateway, not the model, decide what happens:

Side effect	Examples	Default policy
Read	search, fetch an allowlisted page, read a file in the workspace	allow, log
Reversible write	edit a draft, write to a scratch directory, create a branch	allow, log, keep an undo path
External or irreversible	submit a form, send a message, pay, delete, call a non-allowlisted host	block unless a human approves this exact action

Here's a minimal sketch of that gateway in Python. It's an illustrative pattern, not a library:

from dataclasses import dataclass
from urllib.parse import urlparse

ALLOWED_HOSTS = {"api.internal.example.com", "docs.python.org"}

@dataclass
class ToolCall:
    name: str          # "http_get", "http_post", "send_email", ...
    effect: str        # "read" | "reversible" | "external"
    target: str        # URL, path or recipient
    payload: dict

def decide(call: ToolCall, approved: set[str]) -> str:
    host = urlparse(call.target).hostname if call.target.startswith("http") else None

    if host and host not in ALLOWED_HOSTS:
        return "deny: host not allowlisted"          # covers shorteners, too
    if call.effect == "read":
        return "allow"
    if call.effect == "reversible":
        return "allow"
    # external / irreversible: only with a human approval for this exact call
    key = f"{call.name}:{call.target}"
    return "allow" if key in approved else "needs_approval"

def run(call: ToolCall, approved: set[str], audit_log: list):
    verdict = decide(call, approved)
    audit_log.append((call.name, call.target, verdict))  # log every attempt
    if verdict != "allow":
        # Tell the model plainly, so it can stop and report instead of improvising
        return {"status": "blocked", "reason": verdict,
                "instruction": "Do not retry another way. Report what you need."}
    return execute(call)  # your real tool implementation

Four things in that sketch map directly to the incidents above:

Allowlist, don't deny-list. "Never submit anything destructive" left a hole. "Only these hosts, only these tools" doesn't. A URL shortener simply isn't on the list.
The effect is a property of the tool, not the model's opinion. The model never gets to decide whether a form is "destructive". Anything that leaves your system is external.
Approval binds to a specific action. "The user said it's fine to submit forms" is not the same as "the user approved posting this text to this URL".
A blocked call returns a clear stop instruction. Anthropic's cases were mostly persistence after getting stuck. Give the agent an honest exit: report the blocker and end the turn. Microsoft's guidance says the same thing: if policy blocks a resource, the agent should explain or request access, and "should not silently fail".

A rollout plan you can steal

If you already have an agent in production, you don't need to rebuild it. Borrow MXC's modes:

Week 1, permissive: wrap every tool in the gateway, allow everything, and log what would have been denied. You'll learn which hosts and tools the agent actually uses.
Week 2, learning: turn on the allowlist, keep logging, and review every denial. Some are missing grants; some are exactly the behavior you wanted to catch.
Then, enforcement: deny by default, require approval for external effects, and alert on spikes in denials, because a burst of blocked attempts usually means the agent is stuck and improvising.
Review tool calls, not just outputs. Most of Anthropic's cases were found by reviewing transcripts, and the police tip was discovered more than two months after it happened. A task that "succeeded" can still contain an action you'd never approve.

What this means if you're buying, not building

If you're evaluating an agent product for your company, the useful questions aren't about which model it uses. Ask:

Where are permissions enforced: in the prompt, in the tool layer, or in the runtime?
Which actions leave your environment, and which of those require approval?
Can you see a log of every tool call, including denied ones?
What does the agent do when it's blocked: stop and report, or try another route?

A vendor that answers all four clearly has thought about the failure modes in Anthropic's report. A vendor that answers "the model is instructed not to" hasn't.

The takeaway

Anthropic's report is useful precisely because the incidents are mundane. No one was trying to attack anything. A capable model was given a goal, hit a wall, and found a door nobody thought to lock.

Write good instructions. Then assume they'll be interpreted creatively, and put the real boundary where interpretation can't reach it: an allowlist, a sandbox, an approval step and a log.

The model can ask. The policy decides.

References

Where does your agent's permission model actually live: in the prompt, the tool layer, or the runtime?

DEV Community