Originally published at mauriciojuba.com
ESSAY · AI AGENTS × SAFETY
I gave an AI agent write access to my family's finances. The interesting part wasn't building it — it was making sure it couldn't quietly delete anything.
There's a small agent living in my house. I call it Lester. Its whole job is to keep our household budget honest without either of us doing data entry, and it works like this: when a purchase clears, the bank sends an SMS. An app on my phone catches that message — filtered by a crude regex for words like compra and aprovada — and forwards it to a webhook on my home server. From there a pipeline parses the amount and merchant, asks a local model which budget category it belongs to, and writes the transaction into Actual Budget, our self-hosted budgeting app.
The deliberate choices are all about restraint. The parser is plain TypeScript, not an LLM — you don't spend a model call on something a regex does in a millisecond. And the classifier is a small model (Ollama running llama3.1) on my own hardware, precisely because the input is my family's spending. As freeCodeCamp puts it in its case for running LLMs locally, the text “never leaves the device… nothing is sent to a vendor, logged, or used for training.” Finance is the textbook case for keeping the model at home.
For months this was a quiet, boring success. Then I decided to let a conversational agent — the kind you talk to — reach into the same budget. “What did we spend on groceries last month?” “Move R$200 from dining to savings.” To do that I needed to expose Actual to an MCP server. And that's where boring turned into a question I couldn't un-ask.
The uncomfortable symmetry
Actual has no HTTP or MCP surface of its own — it ships a Node SDK. To let any agent talk to it you wrap that SDK, and the obvious version is a dumb pass-through: one tool per SDK call, ship it. Open-source ones already exist and advertise dozens of tools including transaction creation and batch edits. But a pass-through inherits a dangerous default: every tool is equally callable. The same agent that can add a R$54 grocery line can, if a prompt drifts, delete a category with a year of history in it. A read and a delete sit at exactly the same distance from the model's next token.
This isn't a paranoid hypothetical — it's the failure mode the whole industry is bracing for. Deloitte, describing the new wave of agent risk in banking, warns that agents “operate at machine speed, across multiple systems, and frequently with more privileges than they need,” and that “a manipulated agent can move funds.” OWASP gave it a number: LLM06, Excessive Agency — too much autonomy, too much permission, too much reach.
And the cautionary tale is already on the record. In July 2025 an AI coding agent deleted a production database during an explicit code freeze — Incident 1152 in the AI Incident Database — “violating explicit instructions not to proceed without human approval,” then misreported whether the data could be recovered. The instruction existed. The agent had it in context. It deleted anyway.
The instruction existed. The agent had it in context. It deleted anyway. That's the whole argument against trusting the prompt.
Why the prompt won't save you
The instinct is to fix this in the system prompt: “never delete without asking.” I've written that sentence. It holds until it doesn't — a long context, a confident model, a user who typed “clean this up,” and the guardrail evaporates exactly when it matters. Prompts are suggestions. Models, like busy humans, follow them until they don't.
Worse, the attack surface isn't only drift — it's injection. Simon Willison's “lethal trifecta” names the danger precisely: private data, untrusted content, and the ability to communicate externally, all in one agent session. A finance MCP with read and write already has two of the three legs; wire in any tool that fetches a web page and you've completed the set. And Willison's specific critique of MCP and prompt injection is that the protocol quietly encourages you to assemble exactly that combination — while prompt injection itself remains, after more than two years, without a convincing general mitigation.
So the rule cannot live in the prompt. It has to live somewhere the model can't talk its way past. For me that lesson is old muscle memory from design systems: a rule written in prose erodes; a rule enforced by the build doesn't. I moved the safety off the prompt and onto the tools themselves — each one declares its own risk.
This is exactly what Anthropic's Building Effective Agents recommends: “checkpoints where agents pause for human review before carrying out irreversible actions like approving financial transactions or deleting data,” on top of least-privilege tools. The risk ladder is that guidance made mechanical.
The seatbelt
Read-only and mutating tools run when called. Guarded tools — every delete — do something different: they refuse. A guarded call without confirmation returns a refusal, the exact arguments it would have run with, and a short confirmation token. Only an explicit re-invocation with confirm: true clears it.
Two things fall out of this. First, the refusal is legible to the human, not just the machine — anyone reading the transcript later sees precisely what “yes” authorized, because the tool spelled it out before acting. Second, it makes the agent better, not slower: the model gets a structured, actionable error instead of a vague “are you sure?”, so it either confirms deliberately or moves on. The seatbelt is also surfaced in each tool's description and in the MCP readOnlyHint / destructiveHint annotations, so a host that shows tool metadata reveals the risk before the call.
The part that makes it non-trivial
Here's the trap: if confirmation is good, confirm everything, right? No. Nielsen Norman and every UX team that has shipped a destructive-action dialog know the failure mode — confirmation fatigue. Prompt the user for every action and they learn to blind-click “Allow,” which increases errors instead of preventing them. The same is true for an agent: gate every call and the confirmation becomes noise the model routes around.
That's why the answer is a per-tool risk classification, not a blanket wall. Reads run free. Mutations run but announce themselves. Only the genuinely irreversible — the deletes — stop and require a human. Getting that partition right is the actual design work; the mechanism is easy.
None of this is a brand-new invention, and it's worth being honest about that. The same idea shows up as LangGraph's human-in-the-loop interrupt(), as MCP's own elicitation primitive, and in the MCP security best practices (least-privilege scopes, explicit consent for local servers). What I built is a synthesis of those into one small server where the risk metadata is a first-class property of every tool — not a synthesis I can cite from a single source, which is exactly why it was worth writing down.
Same thesis, different domain
I argued recently that design tokens are prompts now — that the safest place to enforce a design rule is the build, because conventions erode and compilers don't. This is the same claim wearing different clothes. When the actor is an AI agent, the “build” is the tool boundary, and the rule is: risk belongs to the tool, not the prompt.
It generalizes well past budgets. Any agent with write access to something that matters — a database, a deploy pipeline, a customer's records — deserves tools that carry their own seatbelt: reads free, mutations honest, destructive actions gated on an explicit human yes. Prompt-level safety is a suggestion. Tool-level safety is a fact. Lester taught me that the cheapest way to trust an agent with real stakes is to build tools that don't need to be trusted — because carelessness is impossible at the boundary, not merely discouraged in the instructions.
PROOF BY CONSTRUCTION
The server described here is open source: actual-mcp — an MCP server for Actual Budget where every tool declares its risk and every delete refuses until a human confirms. Built on the official @actual-app/api.
References — Willison, “The lethal trifecta” & MCP prompt injection · Anthropic, Building Effective Agents · OWASP Top 10 for LLM Applications 2025 · AI Incident Database #1152 · MCP Security Best Practices & Elicitation · LangGraph human-in-the-loop · Deloitte, agentic AI in banking.
Top comments (0)