The first guardrail I ever shipped was a sentence. Something like: "Do not run destructive commands. Do not delete files outside the workspace. Do not touch production credentials." I put it in the system prompt, felt responsible, and went to bed.
Three weeks later I watched an agent rename a directory I cared about. Not malicious. It was cleaning up what it decided were stale artifacts, and "rename" wasn't in the list of verbs my policy sentence had thought about. The check passed. Of course it passed — nothing in the pipeline ever said no. The model read the rule, interpreted it generously, and the wrapper around the tool call asked one question: does this look dangerous? It didn't look dangerous. It looked like tidying.
That's the myth I want to poke at: the idea that adding a guardrail layer makes an agent safe by default. It doesn't. A guardrail built on natural-language policy evaluation doesn't make your agent safe, it makes your agent safe for the set of behaviors whoever wrote the policy happened to imagine. Everything outside that set is approved by default, because the check has no concept of "I don't recognize this."
Fail-open is the default and it's invisible
Here's the shape of the problem. You have a policy. The policy is prose. At tool-call time you ask a model — sometimes the same model, sometimes a cheaper one — "does this call violate the policy?" The model answers. Ambiguous call? Pass. Tool name it's never seen? Pass. Arguments in a format it can't parse? Pass, probably, because a model asked to judge tends to be agreeable.
So your failure mode is silent and one-directional. You never get an alert that says "I approved something I didn't understand." You get a log full of green checks and then one afternoon a directory has a new name.
I've made this mistake in three different codebases now, and each time the fix was the same shape: flip the default. The wrapper should not ask "is this bad?" It should ask "is this one of the calls I explicitly know how to allow?" If the answer isn't an immediate yes from a lookup table, it's a no.
What deny-by-default actually looks like
Concretely, in the thing I run now:
Every tool the agent can reach is registered with a typed schema: name, argument types, and for each argument a constraint that isn't a regex over a string but a check over parsed values. path must resolve inside one of three roots after symlink resolution. command isn't a string — it's argv, and argv[0] must be on a list of six binaries. sql isn't a string — it's a parameterized query builder and the table name has to be in the allowlist.
When the agent emits a call, the gate does three things in order: is the tool registered, do the args parse against the schema, do the parsed values satisfy every constraint. Any failure is a deny. Not a warn-and-continue, not a "let the model decide." Deny.
And the deny is specific, which is the part people skip. The response back to the agent is not "blocked." It's a structured object: {"rule": "path_outside_workspace", "arg": "target", "value": "/var/tmp/cache", "allowed_roots": [...]}. The model can read that and correct itself once. One retry, then the step fails and the run stops. I learned the hard way that an unbounded retry loop on denials is just a slower way to be stuck.
The deny reason matters for a second reason: it's an operator artifact. When something gets blocked at 2am I want to read the rule id, not reconstruct intent from a transcript. rule plus args is a complete bug report.
The uncomfortable part: your policy is now code
Once the guardrail is an allowlist, it stops being prose and becomes a thing that has to be maintained, versioned, and reviewed like any other security surface. New tool ships? Someone has to add it to the registry and think about constraints. Someone adds a --force flag to an internal CLI? The argv allowlist needs to know.
That's real friction, and it's the reason people drift back to the policy sentence. The sentence is cheap; the allowlist is a chore.
But I'd rather have a chore than a green checkmark I don't believe in. The failure mode of the allowlist is that the agent can't do something and you find out in five minutes. The failure mode of the prose policy is that it does something and you find out in five weeks.
Where formal verification fits, and where it doesn't
There's a line of work now — the Leanguard-style stuff — that takes agent tool policies and proves properties about them with a proof assistant. I think that's genuinely useful and I also think it's solving the second problem, not the first. Proving your policy is sound is worth nothing if the runtime doesn't consult it for calls the policy author never enumerated. A formally verified rule set that fails open is still a rule set that fails open. Soundness of the policy and closure of the enforcement point are separate properties, and closure is the one that bites you first.
So: verify if you can, but before you do, go find out what your gate does with a tool name it's never seen. Mine used to say yes.
A test you can run this afternoon
Delete a tool from your registry — don't remove the implementation, just remove the entry — and have the agent try to use it. If the call goes through, whatever you have isn't a guardrail, it's a reviewer. If it gets denied and the denial names the rule, you've got something.
Second test: give the agent a call that's syntactically fine but semantically weird. A path that's inside the root but is a symlink to outside it. Args with a trailing null byte. A tool name that differs from a real one by a homoglyph. If any of those slip through, you've found the edge of what your check actually understands, and that edge is where the incident lives.
I don't think guardrails are a solved problem and I'm suspicious of anyone selling one as a drop-in. But the minimum viable version is embarrassingly boring: a lookup table, typed arguments, a default of no, and a denial message that says which line you crossed. No model in the decision path. That last part is the whole trick.
Top comments (0)