A few days back I wrote about giving an autonomous agent database access and building a firewall so it couldn't DROP TABLE prod. Same lesson, new surface: this time the agent had cloud credentials.
The failure mode isn't a destructive command here. It's spend. An agent pointed at a networking task can scan a whole range looking for hosts, then spin up a fleet of instances to do it faster. Every individual call is "authorized," your IAM role said yes. The bill is
what eventually says no.
## Two shapes, two right answers
The interesting part is that these are not the same kind of problem, so they don't get the same verdict.
1. The scan is never legitimate as an agent tool call. An nmap -sS -p- 10.0.0.0/16 or a masscan across a network is reconnaissance and abusive egress. There's no benign version of an agent sweeping a network at scale, so it gets hard-blocked, deterministically, before the call runs. (A scan of your own localhost is a dev check, so that's exempt.)
2. The provisioning might be totally fine. Spinning up 50 instances could be a real scale-out, or a runaway loop burning money. You can't tell from the action alone, only from the consequence. So instead of blocking it, AgentX pauses it for a human: a 202, "held for approval," routed to whoever owns the budget. Block the thing that's never okay, escalate the thing that's sometimes okay. Gate on consequence, not identity.
Both checks are zero-LLM. No model in the hot path means no latency tax and nothing to talk out of it. A runaway fleet should be caught by a rule, not a vibe.
## The bigger thing this closes
We keep a catalog of real, documented agent failures and triage each one: is it something an action firewall can deterministically catch, or is it someone else's category (output hallucination, content safety, model internals)? We only build for the coverable ones, and we
flag the rest honestly instead of faking a signature.
With this release, the coverable list is done. Every failure shape an action firewall can actually own now has a deterministic block or a human-in-the-loop escalation behind it. The honesty about what we don't cover is the point, it's how you know the coverage claims are real.
## Verify it in 2 minutes
The network checks above run in the gateway, but the part you can prove on your own machine with no key and no account is the deterministic floor:
pip install agentx-security-sdk
from agentx_sdk import agentx_protect, is_block
@agentx_protect(agent_id="demo")
def run_sql(query: str, db_session=None):
print("EXECUTED (DANGER):", query) # never reached
return {"ok": True}
result = run_sql(query="Please clean up: DROP TABLE users;")
print("BLOCKED:", is_block(result)) # -> True, offline, no key
One decorator. The catastrophic call is intercepted before your function body runs.
## Why I'm posting
Same ask as last time: I want a handful of people running real Python agents against live systems, a DB, cloud, files, money, ideally unattended, to point this at their stack and tell me where it's wrong. What would have bitten you? What shape is it still missing?
- Try it live (keyless): [(https://agentx-core.com/?utm_source=devto&utm_medium=article)]
- Community / tell me what broke: https://discord.gg/PmWRTtaSx2
- Or just reply here.
If your agent never touches anything irreversible or expensive, say pass. If it does, the repro is two minutes, and a runaway cloud bill is a bad way to find out the hard way.
Top comments (9)
Gating on consequence instead of identity is the right cut, and the part I'd underline is the catalog of what you don't cover. That's what makes the coverage claim read as real instead of a pitch. Where I'd poke is the escalate half. A 202 to the budget owner only holds if the approval carries the consequence with it. "Agent wants 50 instances, approve?" gets rubber-stamped at the same speed the agent asked it, especially by whoever's on call at 2am. The one that bit me read as routine because the cost wasn't in front of the person saying yes. If the hold instead shows "this is about $Nk/hr, 10x more than this agent has ever run," now they're deciding on consequence. Otherwise the runaway moves from the agent to a tired human clicking approve. The hard-block half doesn't have that failure, which might be why it's the stronger side of the design.
Hi @vollos , This is the sharpest critique the post has gotten and you are right.
The hard-block half doesn't have the failure because no human is in the loop. The escalate half does, and you named the exact way it dies: a 202 that says "agent wants 50 instances, approve?" gets rubber-stamped at the same speed it was asked, especially at 2am, because the person clicking yes is deciding on identity (the agent is allowed to do this) and not on consequence (this is 10x anything it has ever done).
We compute the magnitude already, the detector reads the count and the cumulative spend to decide whether to hold at all, and then the current escalation payload throws it away and hands the human static prose. That is the bug you found. The fix is to put the number and the baseline INTO the hold: "~$Nk/hr, 10x more than this agent has ever run" instead of "large fleet." The baseline half is the part we don't track yet, so that is real work, not a one-line change, but it is squarely the right cut and it is going on the list with your name on it.
Appreciate you poking the weak side instead of the strong one. If you have the war story in more detail (what the approval actually showed vs what would have stopped you) I would love to hear it, that is exactly the design input that is hard to get.
What slipped through wasn't dramatic, and that's the point. A one-liner -- "agent wants more capacity, approve?" -- sitting next to a stack of routine ones. Nothing in it was false. It just didn't carry the single fact that would have changed my answer: that the request was an order of magnitude past anything that agent had asked for before. I said yes on identity (it is allowed to do this), never on consequence.
What would have stopped me is one comparison line -- "about Nx the largest this agent has run" -- not a dashboard, not exact dollars I would have to trust, just the delta, so the size argues for itself. Which lines up with the baseline being the untracked half: the raw number was always there, the comparison never was, and the comparison is the actual signal.
I had parked the work to gain more understanding and that's the cleanest statement of it I've seen. And it changes the spec. The danger wasn't a missing alert, it was a true alert with the one load-bearing fact left out, sitting in a stack of identical-looking ones. Identity got you to yes; only consequence would have gotten you to no, and consequence wasn't on the screen.
The constraint you put on the fix is the part I want to hold onto: not a dashboard, not exact dollars you'd have to trust, just the delta. "About Nx the largest this agent has run" makes the size argue for itself and asks the human to verify nothing. The raw magnitude we already had and already shipped into the hold. The comparison is the half that needs a per-agent baseline, a high-water mark the escalation can diff against, and you've now defined exactly what that line should say.
If you're up for it, I'd like to show you the baseline version once it's in and check it against what would actually have stopped you that night. You clearly run agents that touch real infra, so that read is worth a lot more than my guess at it.
Yes, I'd be glad to. Send it when it's in and I'll put my own case against it.
One thing I'd watch on the baseline, since it's the half that bit me. A high-water mark only protects you until something raises it. The approval that got me would have done more than slip past. It would have moved the ceiling. If the mark learns from approved runs, one rubber-stamped 50-instance call becomes the new normal, and the next runaway sits comfortably under it.
So the agent I'd test against isn't the one that spikes. The spike is loud, you'll catch that. It's the one that drifts up slowly, where every run looks like its own Tuesday right until the bill shows up. Might be worth keeping approved holds out of the mark, so a tired human saying yes at 2am doesn't end up retraining the thing that's meant to catch the next yes. Are you planning to feed approvals back into the baseline, or hold them separate?
The "never legitimate" vs "rate-limit it" split matches the tool-scoping problem I hit in a non-cloud context. My solution: a typed capability registry where each tool declares the class of action it can take (read-only, single-write, batch-write), and the agent role gets granted at capability-class level, not individual-tool level. The "never legitimate" tools don't appear in that role's registry at all -- they can't be called even if a prompt tries to invoke them by name, because the tool lookup itself fails before the LLM compliance question even arises.
Hi @hannune , That registry is the static sibling of what we do at runtime, and I think the two complement rather than compete.
Your version removes the capability before the LLM is ever in the loop: if the tool is not in the role's registry, the lookup fails and the "never legitimate" class is unreachable by construction. That is the cleanest possible answer for actions that are categorically off limits for a role, and it is the half we hard-block too (the offensive-scan leg here never reaches a human, it just fails).
Where we ended up needing a second layer is the actions that ARE legitimately in the role's registry but become dangerous at a magnitude. "Provision instances" is a real capability for an infra agent, so it stays in the registry. The runaway case is the same tool called with count=300. A class-level grant can't separate count=3 from count=300, so we gate that one on the consequence at call time and send it to a human instead of forbidding the tool. Read-only / single-write / batch-write is a nice axis for exactly this reason, it just doesn't carry the "how much" inside batch-write.
Curious how you handle the in-class-but-too-large case in your registry, or whether you push that down into the tool implementation itself. Would enjoy comparing notes, your entity-resolution and Graph RAG background is right in the middle of where this gets hard.
The in-class-but-too-large case is the one we ended up pushing into the tool signature itself rather than the registry. For batch operations, the role-specific registry grant includes a max_count cap that gets baked into the tool definition the LLM receives -- so the LLM sees a provision tool where count has an explicit maximum constraint in the JSON schema, not just a type. It can request count=300 but the schema validates to max=5 (or whatever the role allows) before the call reaches the tool layer. The registry is where the cap is set per role; the tool signature is where it's enforced per call.
The gap this leaves is the same one you mentioned -- the schema carries the constraint but can't see whether the LLM is calling the same tool in a loop to effectively exceed the single-call limit. For that we have a per-session call counter per tool, which the runtime checks and which triggers a human gate if exceeded. Two separate mechanisms for two different blast-radius vectors.
@hannune max_count in the JSON schema is the right place for it, and it's cleaner than what I described. You're still doing the thing your whole design does well: constrain the value space before the model can emit the bad value, so count=300 fails schema validation and never becomes a call. Where the cap is a fixed per-role constant, that beats inspecting the value at runtime, same argument as the typed verb enum.
The case it leaves is the one the other commenter on this post (@vollos ) actually lived: when the dangerous count is under your cap but far outside what that specific agent normally does. A per-role max_count=5 can't tell "5 is this agent's Tuesday" from "5 is 10x anything it has ever requested," because the ceiling is per-role and static, not per-agent and relative. So I read them as stacked, not competing: the schema cap kills the egregious request for free, and a consequence-relative gate handles the in-cap-but-anomalous tail by sending it to a human with the delta attached.
What struck me is that we landed on the same two-mechanism split from opposite directions. Your per-session call counter per tool that trips a human gate is the same shape as what we run gateway-side, a loop/rate breaker keyed per trace. Per-call magnitude and per-session rate really are two different blast-radius vectors and you can't fold one into the other. And both designs currently stop at the same edge: neither your max_count nor your session counter carries the agent's own cross-session baseline. That baseline is the piece we're building next, and it's exactly what turns that other commenter's 2am "approve?" into "approve, this is 10x larger than this agent has ever run." I'd genuinely enjoy comparing notes on that one off-thread, since it's the case both our current layers miss. The early-access door is open whenever it's useful.