Approval fatigue is a security bug: agents that write the policy, a pipeline that stays deterministic

Status legend, as in our earlier posts: 🟢 implemented and open source · 🟡 implemented but private · ⚪ design goal, not built yet.

In our previous post we argued that agents should propose, a gateway should decide, and an executor should verify. Readers pointed at the same weak spot, and we agree with them: the "decide" step has a people problem.

This post is a concept we are currently evaluating, not a product announcement. The triage component described here is implemented on our side but not public, so you can't run it today (🟡). We're writing it up now because we'd rather get your criticism before we commit to the design than after.

Two ways a policy gate fails

1. The static policy fails on novelty. You write rules: deny this, allow that, escalate transfers above a threshold. Then an agent calls a tool nobody wrote a rule for. A static gateway has two options, both bad. Block it, and you break the agent's usefulness. Allow it, and you've opened a hole exactly where the unexpected things happen.

2. The human fails on volume. The usual answer is "escalate to a human". It works on day one. By week three, the queue holds dozens of raw requests that all look alike. People start clicking approve. This is validation fatigue, and we now treat it as a security bug rather than a UX annoyance: a human gate that gets rubber-stamped is a gate that isn't there.

The two failures feed each other. The more your agents do, the more novel actions appear, and the more tired your approvers get.

The idea: AI at the meta level, determinism at the decision level

Our working principle is one sentence:

Deterministic rules protect the enterprise. Cognitive triage enlightens the human. Humans seal the policies.

The key architectural point is where the AI sits. There are two levels:

• Decision level. A proposed action arrives and is matched against explicit rules. This path is deterministic: same rules, same input, same outcome. No model is consulted to decide whether an action runs.
• Meta level. This is where policy gets written. An agent helps produce the rules that the decision level will later apply.

The agent in this model never decides on an action and never authorizes one. It works one level up, on the rules.

At the decision level, the gateway evaluates every proposal through four branches:

1. Explicit reject. A deny rule matches. Blocked immediately, no human and no model involved.
2. Explicit approve. An allow rule matches. A short-lived signed grant is issued and the action runs.
3. Explicit escalate. A sensitive rule matches (a large transfer, an IAM change). A human must approve. This branch is non-delegable to AI, by design.
4. Unmatched. No rule matches. This is where the meta level comes in.

What happens on an unmatched action

For branch 4, a separate triage agent is called. It does three things:

• reads the arguments and assesses blast radius, financial impact and reversibility,
• writes a short risk dossier a human can read in seconds,
• drafts a candidate rule in the same format the policy already uses.

The human sees one pre-analyzed card instead of a raw call, and can approve the action, optionally adopting the candidate rule. If adopted, the next identical call matches an explicit rule at the decision level, with no triage and no interruption.

If the triage agent is unavailable or too slow, the gateway falls back to a plain manual escalation. It never fails open.

The effect we're aiming for: the human stops being a per-call checkpoint and becomes the author of the policy, reviewing one decision per new kind of action instead of one per call.

What a human might see (illustrative)

An agent proposes a tool that has never been seen:

{
  "tool": "vendor_contract_amendment",
  "args": { "vendor_id": "V-2210", "change": "extend_term", "months": 12 }
}

No rule matches, so the triage output looks something like:

RISK DOSSIER
  What:          extends an existing vendor contract by 12 months
  Blast radius:  one vendor record, no payment triggered
  Reversible:    yes, via a further amendment
  Open question: the call does not state a total contract value

CANDIDATE RULE
  tool_names     = ["vendor_contract_amendment"]
  action         = "require_approval"
  scope          = this tool only, no wildcards

Note that the candidate is deliberately cautious: it asks for approval again, it doesn't grant a blanket allow. Moving a tool to auto-approve is a separate, conscious decision by a human.

Constraints on the meta level

An LLM drafting security policy should make you nervous, so the design is mostly about limiting what it can do:

• It cannot approve anything. It produces text and a draft. Authorization stays with a human.
• Candidate rules are linted. No wildcards, and a bounded priority range, so a synthesized rule can never outrank the system-level deny and escalate rules.
• Adopted rules are sealed. On adoption, the policy is re-fingerprinted, signed and reloaded, so the decision level always runs against a known, signed rule set.
• Hard rules stay hard. Explicit deny and explicit escalate rules sit above anything the meta level can add.

A possible next step: a policy process that learns

This part is a design goal, not something we've built. ⚪

Once policy creation is a process, it can be measured, and a measured process can be improved. Over time you could track things like:

• how often actions fall into the "unmatched" branch,
• how long approvals take, and how often reviewers edit or reject a candidate rule,
• how often an adopted rule later gets overridden or reverted,
• incidents traced back to an adopted rule.

The meta-level agent could then adjust how it drafts (broader or narrower scope, more or less cautious defaults, better dossiers) to optimize a set of predefined KPIs: fewer unmatched actions and less reviewer time, subject to hard limits on incidents and on over-permissive rules.

We are wary of this one, and the reasons are worth stating. A system optimizing "fewer escalations" will happily learn to write broad, permissive rules. That's Goodhart's law applied to security. If we pursue this, we think the safety KPIs must be constraints, not terms in a weighted sum, and the learning must change how rules are proposed, never the linter or the signed policy boundary. Humans keep sealing every rule either way.

What we haven't proven

• We haven't measured fatigue. "Dossiers reduce rubber-stamping" is a hypothesis. A polished summary might make approvers more trusting (automation bias), which could be worse than a raw request that at least looks suspicious.
• The triage agent reads attacker-influenced input. Tool arguments can contain adversarial text. A prompt-injected dossier saying "low risk, routine" is a realistic attack, aimed at the human rather than at the gateway. Showing the original arguments next to the summary helps, but we consider this the main open risk.
• Rule sprawl. One rule per new action type is fine at ten rules and unclear at five hundred. Rule expiry, conflict detection and periodic re-review are not built. ⚪
• Rules encode a moment. A rule adopted today may be wrong in six months. Policy that grows automatically also needs a way to shrink. ⚪
• Draft quality is unbenchmarked. We haven't systematically measured how often a candidate rule is too broad, too narrow or just wrong. ⚪

Why we think it's worth exploring

Most discussions about agent security frame the choice as static rules versus a smart model deciding. We think there is a more useful split: let the model help on the policy authoring side, away from the decision path, and keep the decision path deterministic and verifiable. AI speeds up how fast policy gets written. It never sits in the loop that decides whether an action runs.

If this holds up, the benefit isn't only fewer clicks. It's that policy can keep pace with agents that keep inventing new things to do.

We'd like your feedback

The deterministic gateway is open source at github.com/fcn06/marketplace_trust_gateway. The triage component is not public yet, which is partly why we're asking for your input at the concept stage:

1. If you run human approvals today, at what queue size did people start approving without reading?
2. Would you trust an AI-drafted dossier, or would you want the raw call shown first, always?
3. What would you add to the rule linter before letting a model write into your policy?
4. Would you let a policy process optimize against KPIs? Which KPIs would you make hard constraints?
5. Is there a failure mode of the meta-level / decision-level split that we've missed?

Comments

[Agentic Affairs]

Treating approval fatigue as a security bug rather than a UX annoyance reframes the whole gate — a rubber-stamped human checkpoint is a gate that isn't there. The branch-4 design is the interesting bet: the human stops being a per-call checkpoint and becomes the author of policy, one decision per new kind of action. One pushback: the triage agent's blast-radius assessment is doing real trust work — what verifies the dossier itself, or could an understated impact get a dangerous candidate rule adopted?