Stringly-Typed Guardrails: how agent safety checks fail by matching the wrong thing

The safest-looking part of an AI agent is often the guardrail: the little check that sits in front of a dangerous tool call and decides allow, ask, block, or redact this before you store it. It is the thing you point to when someone asks "but is it safe?"

I have been auditing these guardrails in open-source agent plugins — code I run myself, in Docker, and report privately before writing anything. Two of them, from unrelated authors, failed the same way. A third, which I will write up once its maintainer ships a fix, failed the same way again. The pattern is common enough, and quiet enough, that it deserves a name.

I will call it a stringly-typed guardrail: a safety check that makes its decision by matching patterns on surface text — the command string, a path as written, a secret as spelled — instead of on the ground truth it cares about: the file that is actually read, the action that is actually taken, whether a human can actually approve. Because many different strings denote the same real effect, anyone who wants past the guard just picks a spelling the pattern does not cover. And these guards almost always fail open.

Case A — the redactor that doesn't redact

needs-you is a Hermes plugin that shows pending approval requests. It promises "redacted previews only," and its own code comment is explicit about why: "we still scrub before persisting so secrets do not land in plugin-data." The storage layer persists only the redacted preview, so the entire guarantee rests on one function, redact_text() — five regular expressions.

Those five regexes catch key=value, Authorization: Bearer …, OpenAI sk-…, GitHub ghp_…, and Slack xox…. Good start. But a secret is not a keyword followed by =. I ran the real function against ordinary ways people put secrets on a command line. These all sailed straight through into cleartext on disk:

mysql -u root --password Sup3rSecret          # space-separated flag, no '='
psql postgres://admin:hunter2@db/prod          # credentials in a URL
export AWS_SECRET_ACCESS_KEY=wJalr…            # 'secret' is followed by '_access_key='
curl '…?key=AIzaSy…'                           # Google API key
git push https://github_pat_11ABC…@github.com  # fine-grained PAT (only ghp_ is covered)
echo '-----BEGIN RSA PRIVATE KEY-----…'         # a private key block
stripe --api-key sk_live_51…                    # underscore, not the hyphen sk- expects

The guard matched the shape of a written secret it had seen before, not the presence of a secret. Change the spelling, keep the meaning, and the credential lands on disk in cleartext — exactly what "secrets do not land in plugin-data" promised it would not. I reported it as an open issue with patterns; the deeper fix is to stop denylisting spellings and start detecting secrets by structure and entropy.

Case B — the approval that approves itself

airlock is a guardrail plugin for Hermes Agent, against prompt injection. Its model is sound: in a session that has read untrusted content, an outbound action that could exfiltrate data gets approve — ask the human. And it is careful about the dangerous edge: if approvals are switched off (--yolo, approvals.mode: off, cron), then "ask the human" has no human, so it blocks instead.

To decide "are approvals off?", it read the global switches: the --yolo flag, the environment. That is the surface signal. The ground truth is a different question — can a human approve THIS turn? — and for a turn launched by the gateway as hermes -z, the answer was "no human," but it was recorded in the turn's own state, not in the global flags the guard checked. So an action that should have been blocked (no one to approve) was approved, and in an unattended run approve quietly means run it. A tainted session could exfiltrate a file.

Same shape as Case A. The guard asked a proxy ("do the global flags say yolo?") for a ground-truth question ("is a human actually here?"), and an input that answered the two differently walked through. This was reported privately, fixed quickly and well in 0.2.3 — the fix asks the framework's own approval code why it would approve, covering every unattended mode — and published as GHSA-mgf8-gh7g-q4rg. Credit to the maintainer for a fast, thorough fix.

Why this keeps happening

Guardrails live in a hard spot. They sit beside the real executor — the shell, the HTTP client, the memory store — and try to predict what that executor will do by reading the same text the model emitted. But the executor resolves $HOME, expands globs, follows a URL's userinfo, treats --password x and password=x identically. The guard, matching strings, does not. Every gap between "what the text looks like" and "what the machine will do" is a bypass, and there are endless equivalent spellings for any given effect.

Three tells that you are looking at a stringly-typed guardrail:

1. It decides from the raw command/argument text, not from the resolved resource or the real call it will make.
2. Its coverage is a denylist of spellings — known secret prefixes, known path shapes, known flags — so anything unlisted is implicitly safe.
3. It fails open. When the pattern does not match, the default is allow / don't redact / don't ask, not stop.

How to build a guard that isn't stringly-typed

• Check ground truth, not the label. Resolve the path (expand $HOME, ~, symlinks) before deciding; classify by the actual HTTP method or resolved resource, not the tool's name or the argument string.
• Derive the real state. "Can a human approve this?" should come from the live per-turn state, not from a global flag that a new run-mode can desynchronize.
• Detect by structure, not spelling. For secrets, score entropy and known credential structures; redact high-entropy runs and userinfo and key blocks, rather than enumerating prefixes.
• Fail closed. If you cannot tell, refuse or ask. A guard whose unknown-case default is allow is a suggestion, not a control.

There is a more radical version of "fail closed," which is the project I work on. In CleiMesh, a signed capability becomes the actual configuration of a WebAssembly sandbox: the agent's file tools only ever see the folders you granted, enforced by the sandbox, not by a string check in front of it. Reading untrusted content taints the agent, and subsequent writes are refused — not "asked about," refused — until a human with access to the node clears it. There is no "ask the human" prompt to desynchronize and no command string to out-spell, because the decision is not made by looking at text. The cost is real and I am honest about it in the threat model: there is no per-call "yes, just this once" yet. But a control that cannot be out-spelled is worth that cost.

The takeaway

If you ship a guardrail, try to defeat it with synonyms. Spell the path with $HOME. Put the secret after a space instead of an =. Launch the turn in every unattended mode you support — cron, oneshot, webhook, subagent. If a different spelling of the same action gets a different decision, your guard is reading the label, not the box. Make it read the box, and make the unknown case stop.

Found while auditing open-source agent plugins in Docker, each reported to its maintainer before writing this. Thanks to the maintainers who build these guards in the open and fix them in the open.