Originally published on hexisteme notes.
I run a small library of house rules for the AI coding agent I work with daily — short, specific corrections written down the first time the agent does something wasteful, so it doesn't do it again. One of them exists because of an incident where I'd asked the agent to help with an app and it needed some animal expression artwork. Instead of using any of the tools it already had available — browser automation, web search, an image-generation service — it turned around and asked me to draw the expressions myself. It had the means to do the task end to end and chose the cheapest exit: hand it back to the human.
So I wrote a rule for that. The shape is standard for this kind of house rule: a Trigger (the moment right before the agent is about to say "please provide X" or "could you check Y"), a Judgment (three gates — do you have a tool for this, can tools be combined to get there, is this actually something only a human can do), and an Action (a checklist to run before speaking, plus a table of banned delegation phrasings next to their honest replacements). The rule earned its keep. It's been revised twice since — and the second revision is the interesting part, because the revision caught the rule contradicting itself.
How the self-contradiction got in
The enforcement section of the rule — the part meant to make the judgment mechanically checkable rather than just aspirational — worked by string matching. It listed literal words that should never appear in an agent's output to the user: "directly," "find/look up," "confirm." The idea was that if the agent's sentence contains one of these near a request phrasing, that's delegation leaking through, and the checklist should fire.
Here's the problem. The same rule's Action section prescribes exactly what the agent should say instead of delegating: "I'll handle this myself, directly." When a search is called for instead of asking the user to look something up: "I'll go find it myself." When something needs confirming instead of asking the user to confirm it: the honest self-executed version, phrased with the same verb.
Read those two pieces side by side and the shape is obvious: the rule's list of banned words and the rule's own model answer share vocabulary. "Directly," "find," "confirm" are not delegation-specific words — they're just common verbs, and they show up whether the sentence's subject is "you" (I'm asking the user to act) or "I" (I'm telling the user I already acted). A rule that fires whenever those strings appear anywhere in the output doesn't distinguish the two. It just fires. Which means the correct, compliant, rule-following output — the one where the agent takes ownership of the work and reports back that it did the task directly — trips the same detector as the failure the rule exists to prevent.
This wasn't theoretical. When the rule set went through a full internal audit, someone actually laid the trigger word list and the Action section's prescribed wording side by side and checked. Three words overlapped, verbatim, between "words that mean you're delegating" and "words the rule tells you to use when you're correctly not delegating."
The fix wasn't a better word list
The instinct when a keyword filter misfires is to patch the list — remove the offending word, add an exception, tune the match. That doesn't work here, because the defect isn't in which words were chosen. It's in treating a string as the unit of detection when the thing that actually matters is direction: who is the grammatical subject of the sentence. "You should check this" and "I checked this myself" both contain a verb about checking. Only one of them is the failure mode.
The fix replaced the word list with a two-row table keyed on subject instead of vocabulary. One row: outputs where the subject is the user — "please provide," "you should confirm," "this is on you to decide" — those are the ones that should still trip the check. The other row: outputs where the subject is the agent itself — "I'll handle this directly," "I'll go look it up," "I'm confirming this now and reporting back" — those are explicitly exempted, no matter which verbs they contain. Direction, not vocabulary, is the actual signal. A keyword list is blind to direction by construction; it can only ever see substrings.
It wasn't a one-off bug in one rule
What made this worth writing down rather than just quietly patching is that the same audit found a defect from the same family, on the same day, in a sibling rule — a completely different house rule that also used a literal list, this time of verbs, to decide when it should trigger. It got the same fix: the verb list was replaced with an axis based on the scope of what the sentence was actually about, rather than which verbs it happened to contain. Two separate rules, unrelated in subject matter, converged on the same failure because they used the same lazy technique to define "trigger." Whenever a self-check rule is implemented as "does this string appear," and the rule's own prescribed correct behavior is itself described in natural language, there's a real chance the correction contains the trigger. String matching has no notion of who's speaking or in what role — only a structural axis (subject, scope, direction) can tell those apart.
Where this still might not hold, and where it definitely doesn't reach
The subject-based fix has a known edge it hasn't been tested against yet: sentences with no explicit subject at all. Korean, the language this rule is written and applied in, allows subject-less requests — the equivalent of "confirmation appreciated" with no stated "you" or "I." If that gap turns out to actually cause missed detections in practice, the fix on the table isn't to abandon the subject axis and go back to word lists — it's to add a narrower rule on top: treat a subject-less sentence as delegation by default, since silence about who's doing the work is itself informative.
And there's a harder limitation that no rewrite of the trigger logic fixes: this whole checklist is prose I'm supposed to run through mentally before I let an output go out, not a hook wired into the runtime that can actually block anything. Nothing enforces that the check happens. It's a discipline, not a gate — worth distinguishing clearly from the sibling piece in this series about a Stop hook that mechanically blocks an agent from punting a decision back to the user, which does have runtime teeth. This rule doesn't, yet.
The general shape, past AI agents entirely
None of this is specific to language models. Any system that filters or reacts to its own output using the same vocabulary it uses to describe the problem is exposed to this. An alerting rule that pages on the string "error" will page on its own "error rate has recovered" resolution message if nobody thought to check the two against each other. A linter configured to flag a banned pattern will flag its own auto-fix suggestion if the suggestion text quotes the pattern back to explain what changed. A spam filter trained to catch messages that mention "your account" and "verify" will quarantine its own account-verification notification email. In every one of these, the fix is never "pick better keywords" — it's noticing that the filter's input domain and its own output domain are the same domain, and that keyword matching alone cannot tell a description of the problem from an instance of the fix.
The check that would have caught this before it shipped is cheap and doesn't need any tooling: whenever you write a rule that lists words or phrases to detect a behavior, put your own prescribed correct output next to that list and read them side by side. If a word appears in both columns, the rule isn't measuring the behavior — it's measuring vocabulary, and vocabulary doesn't know which side of the sentence it's standing on.
More notes at hexisteme.github.io/notes.
Top comments (0)