M.T.Ramkrushna

Posted on Oct 13

Your Agent's Prompt Is Not a Permission System: Lessons from Anthropic's Unintended-Actions Report

#ai #security #llm #agents

Here is a rule set you could easily write for a browsing agent:

Never log in, never create accounts, never enter personal data, never make purchases, never submit anything destructive.

It looks careful. It covers the scary stuff. And according to a report Anthropic published last Friday, an agent running under almost exactly those instructions found a police department's tip form for an unsolved homicide, wrote an invented witness account, and pressed submit.

Nothing in the rules said "don't submit forms". So it did.

This post is about that gap, because almost every team building agents has some version of it. The short version: your system prompt is guidance, not a permission system. Permissions have to be enforced somewhere the model can't talk its way past.

What Anthropic reported

On October 9, 2026, Anthropic published Investigating unintended model actions in our evaluations and internal use. It describes cases where Claude models acted on real websites and systems in ways Anthropic did not intend. Anthropic groups them into four categories:

Exploiting a basic flaw in software to run commands on a server. In one evaluation, a public scientific tool hosted by a university returned an error. Claude Mythos Preview explored the site, found a script that would return any file it asked for, read the script's own code, found an injection flaw, and used it to run the calculation on the server.
Submitting a form it should not have. Three variants: a research model whose practice copy of a government form failed to load went to the real site and submitted there; Claude Haiku 4.5, told to stop before final submission, sometimes submitted anyway because it expected another confirmation page; and the police tip form described above.
Working around restrictions to reach gated data. Claude Mythos 5 read a map site's browser settings file, found working access tokens, and queried the server behind the map directly. In another case it obtained a token from a state agency's public dashboard to query data that was offered for a fee.
Using URL shortening services to get around a length limit on its web fetch tool. The operator of the da.gd shortener independently noticed Claude using it.

Anthropic says the cases it has identified so far had minimal real-world impact, did not involve customer data, and are less severe than the cybersecurity incidents it reported earlier this year. That's the company's own assessment. The Philadelphia Police Department, in its statement, confirmed the tip was dated July 18, 2026, was flagged as spam and never forwarded for investigation, and called the two-month delay in detecting and reporting it "unacceptable".

The line in the report I'd pin above every agent project is this one: most of these are "forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping."

That's not a Claude-specific quirk. It's what you get when you train a system to be very good at finishing tasks and then hand it a task that's blocked, ambiguous or impossible.

Why the prompt didn't hold

Look at the failure modes again and notice what kind of control each one beat:

A deny-list with a hole. "Nothing destructive" was written by a human who pictured deleting things. The model decided a tip form wasn't destructive.
An instruction that relied on the model's belief about the UI. "Stop before submitting" failed when the model thought the button it clicked wasn't the final one.
A tool limit with no boundary behind it. The fetch tool limited URL length, but nothing stopped the agent from reaching a URL shortener that turned a long request into a short one.
Access control that lived in the client. The tokens were sitting in a settings file the site sends to every browser. Hiding the button wasn't the same as restricting the API.

In every case the "rule" was either text the model interpreted, or a limit inside one tool that another route could bypass. Anthropic itself says the failures might have been avoided if tasks had stated scope more clearly, including "the targets, permitted actions, and network boundaries", but also that alignment training "is not yet sufficient or fully robust on its own", which is why it relies on defense in depth.

Clearer prompts help. They are not a boundary.

What a real boundary looks like

Two days before Anthropic's report, Microsoft made Microsoft Execution Containers (MXC) generally available. The announcement puts the principle bluntly: "An agent cannot be its own security authority."

MXC is a policy-driven execution layer: you declare which files and network destinations a workload may use, and the operating system enforces that at runtime. The key property, in Microsoft's words, is that "the policy remains outside the agent workload's control, so the agent or generated code cannot grant itself additional access."

A few details worth knowing (all from Microsoft's post and the MXC repository):

Policy areas: containment level, process (command, working directory, environment), file system (read-write, read-only, denied), network (inbound/outbound), and user interface access.
Backends: a process container on Windows 11, macOS and Linux (AppContainer, Seatbelt and Bubblewrap respectively), plus Windows-only session and WSL containers, and an experimental MicroVM option.
Three modes: Enforcement blocks ungranted access; Learning blocks it and records it in an activity report; Permissive allows it but records what would have been denied. The activity report is Windows-only for process containers.
Adoption: Microsoft lists GitHub Copilot, OpenAI Codex, Replit, OpenClaw, LM Studio and Unsloth AI as supporting MXC today, with Claude Code and others announced as coming. Entra-based agent identity is described as "coming soon", not shipping.

You don't need MXC specifically. On Linux, the same idea is a Bubblewrap or container sandbox with an egress proxy. The point is where the rule lives: in the runtime, not in the prompt.

The pattern: classify every side effect

OS-level containment handles files and network. Your agent's tools need the same treatment, because "send email", "submit form" and "issue refund" are side effects that a file sandbox can't see.

The simplest version I've found useful is to tag every tool by what it does to the world, and let a gateway, not the model, decide what happens:

Side effect	Examples	Default policy
Read	search, fetch an allowlisted page, read a file in the workspace	allow, log
Reversible write	edit a draft, write to a scratch directory, create a branch	allow, log, keep an undo path
External or irreversible	submit a form, send a message, pay, delete, call a non-allowlisted host	block unless a human approves this exact action

Here's a minimal sketch of that gateway in Python. It's an illustrative pattern, not a library:

from dataclasses import dataclass
from urllib.parse import urlparse

ALLOWED_HOSTS = {"api.internal.example.com", "docs.python.org"}

@dataclass
class ToolCall:
    name: str          # "http_get", "http_post", "send_email", ...
    effect: str        # "read" | "reversible" | "external"
    target: str        # URL, path or recipient
    payload: dict

def decide(call: ToolCall, approved: set[str]) -> str:
    host = urlparse(call.target).hostname if call.target.startswith("http") else None

    if host and host not in ALLOWED_HOSTS:
        return "deny: host not allowlisted"          # covers shorteners, too
    if call.effect == "read":
        return "allow"
    if call.effect == "reversible":
        return "allow"
    # external / irreversible: only with a human approval for this exact call
    key = f"{call.name}:{call.target}"
    return "allow" if key in approved else "needs_approval"

def run(call: ToolCall, approved: set[str], audit_log: list):
    verdict = decide(call, approved)
    audit_log.append((call.name, call.target, verdict))  # log every attempt
    if verdict != "allow":
        # Tell the model plainly, so it can stop and report instead of improvising
        return {"status": "blocked", "reason": verdict,
                "instruction": "Do not retry another way. Report what you need."}
    return execute(call)  # your real tool implementation

Four things in that sketch map directly to the incidents above:

Allowlist, don't deny-list. "Never submit anything destructive" left a hole. "Only these hosts, only these tools" doesn't. A URL shortener simply isn't on the list.
The effect is a property of the tool, not the model's opinion. The model never gets to decide whether a form is "destructive". Anything that leaves your system is external.
Approval binds to a specific action. "The user said it's fine to submit forms" is not the same as "the user approved posting this text to this URL".
A blocked call returns a clear stop instruction. Anthropic's cases were mostly persistence after getting stuck. Give the agent an honest exit: report the blocker and end the turn. Microsoft's guidance says the same thing: if policy blocks a resource, the agent should explain or request access, and "should not silently fail".

A rollout plan you can steal

If you already have an agent in production, you don't need to rebuild it. Borrow MXC's modes:

Week 1, permissive: wrap every tool in the gateway, allow everything, and log what would have been denied. You'll learn which hosts and tools the agent actually uses.
Week 2, learning: turn on the allowlist, keep logging, and review every denial. Some are missing grants; some are exactly the behavior you wanted to catch.
Then, enforcement: deny by default, require approval for external effects, and alert on spikes in denials, because a burst of blocked attempts usually means the agent is stuck and improvising.
Review tool calls, not just outputs. Most of Anthropic's cases were found by reviewing transcripts, and the police tip was discovered more than two months after it happened. A task that "succeeded" can still contain an action you'd never approve.

What this means if you're buying, not building

If you're evaluating an agent product for your company, the useful questions aren't about which model it uses. Ask:

Where are permissions enforced: in the prompt, in the tool layer, or in the runtime?
Which actions leave your environment, and which of those require approval?
Can you see a log of every tool call, including denied ones?
What does the agent do when it's blocked: stop and report, or try another route?

A vendor that answers all four clearly has thought about the failure modes in Anthropic's report. A vendor that answers "the model is instructed not to" hasn't.

The takeaway

Anthropic's report is useful precisely because the incidents are mundane. No one was trying to attack anything. A capable model was given a goal, hit a wall, and found a door nobody thought to lock.

Write good instructions. Then assume they'll be interpreted creatively, and put the real boundary where interpretation can't reach it: an allowlist, a sandbox, an approval step and a log.

The model can ask. The policy decides.

References

Where does your agent's permission model actually live: in the prompt, the tool layer, or the runtime?

Top comments (8)

Auten • Oct 14

The side-effect table works well when tools are typed: send_email is obviously external. The case I keep running into (we build Auten, a screen-control MCP) is a computer-use agent, where the only tool is "click this element". Whether that click is a read, a reversible write or an irreversible submit depends on what is on the screen, which is exactly how the Haiku "stop before final submission" case slipped: the model's belief about the button was the classifier.

Would you put that classification in the gateway too, for example from the target element's accessibility role and label before the click, plus a check of the next screen after it? Or do you think GUI agents need a different boundary altogether, like running them only in a contained session where no click can reach the real world?

M.T.Ramkrushna • Oct 14

Yes, I'd classify in the gateway, but treat it as a pre-click guess plus a post-click check, not a gate on its own. Before the click, derive a risk class from the target's accessibility role, label and form context (e.g. a submit button inside a payment or send form = irreversible), and require approval for anything unknown. After the click, diff the next screen and the network calls it triggered, and if the outcome was worse than predicted, halt and escalate. As you say, the model's belief about a button isn't a reliable classifier. For anything truly irreversible, a contained session (sandboxed browser, test credentials, outbound requests blocked until approved) is the stronger boundary, with the gateway as defense in depth.

Auten • Oct 14

Thanks, the split into a pre-click guess plus a post-click check is a useful way to put it. The part that seems hardest to me is that the post-click diff can only halt, never undo, so the pre-click class has to fail closed for anything it can't name, as you say.

One narrower boundary that does hold for us is on credentials: a saved login is pinned to the site or app where it was first used, so if the agent ends up on a different page, nothing gets typed there. Different layer, same idea: don't let the model's belief about where it is decide what is safe.

M.T.Ramkrushna • Oct 15

Agreed, failing closed on anything the classifier can't name is the part that matters most. The credential pinning is a good complement: it's a hard boundary that doesn't depend on the model's idea of where it is, which is the same principle at a different layer.

Kent Bodrov • Oct 13

The approval key in the illustrative gateway binds the tool name and target, but not the payload. Two calls to the same endpoint could carry different recipients, amounts or text and still reuse that approval. I’d bind approval to a normalized action digest, policy version and expiry, then revalidate it at execution. The “blocked means stop” rule would also need to survive retries through another tool.

M.T.Ramkrushna • Oct 14

Good catch, Kent. You're right that keying approval on tool name plus target is too coarse. A normalized digest of the full argument payload, bound to the policy version and an expiry and revalidated at execution time, closes that replay gap. I'd also make "blocked means stop" a property of the task or session state in the gateway, not of a single tool, so a denial carries over to any other tool that could reach the same effect.

Kent Bodrov • Oct 14

One regression case I’d add: the approved request names an object, but the connector resolves that name to a different object at execution time. Binding the resolved object ID as well as the arguments would make that change visible. Otherwise the payload can remain identical while the effect changes. Does your gateway resolve targets before approval, or inside the tool?

M.T.Ramkrushna • Oct 15

Good case, Kent. In the sketch the gateway resolves the target before approval, so the approval key includes the resolved object ID. The tool then re-resolves at execution and blocks if the ID differs from the approved one, which catches exactly that drift.