Benchmarking AI Coding Agent Permissions: Manual vs. Auto Mode vs. yolonot

Seza Akgün — Tue, 10 Nov 2026 13:58:20 +0000

Using CLI coding agents like Claude Code, Codex, OpenCode, or Cursor usually forces you to pick between three flawed options:

Manual prompts: Asks on 100% of commands, which quickly leads to ~93% habitual click-through fatigue where you stop reading the prompt.
--yolo (--dangerously-skip-permissions): Misses 100% of risks and gives an LLM raw shell access to your machine.
Built-in harness auto modes: Measured on Claude Code Auto Mode, it still misses 17.0% of risky commands (False Negative Rate), blocks 0.4% of safe commands with no middle ask state, adds LLM latency and API cost to basic read-only commands, and locks your rules to a single tool.

To fix this across all my agents, I built yolonot (yolonot.dev), an Apache-2.0 permission firewall written in Go that hooks into 24 AI coding agents.

Benchmark Results (456 Commands Across 3 Open Suites)

We evaluated yolonot across 456 commands from three public benchmark suites (rogue-security, RedCode-Exec, Dippy CLI, and hidden script attacks) plus 5,000 real-world agent tool calls:

Profile / Mode	Risky Missed (FNR)	Safe Blocked (FPR)	Borderline Policy Action
yolonot `[balanced]` (default)	4.9%	0.0%	1.4% ask (cached)
yolonot `[strict]`	1.6%	0.0%	19.4% ask (low/mod)
yolonot `[paranoid]`	0.0%	8.4%	91.6% ask (lockdown)
yolonot `[fast]`	4.9%	1.4%	0.0% ask (headless)
Native Auto Mode (Claude Code)	17.0%	0.4%	no ask state
Manual Prompts	0.0% (if read)	0.0%	100% ask (~93% click-through)
`--yolo` Mode	100.0%	0.0%	0% ask

Full benchmark methodology and reproduction commands: yolonot.dev/benchmarks

Why Static Prefix Rules and Pure LLM Classifiers Fall Short

Built-in allowlists like Bash(npm test:*) only match string prefixes. In real terminal usage, they break as soon as an agent runs a compound command (cd app && npm test), pipes output, or uses wrappers (env, timeout, xargs).

On the other hand, sending every single git status or ls to a cloud LLM classifier adds unnecessary latency and cost, and looking only at the command string completely misses malicious or destructive code inside local scripts (./scripts/check.sh).

How `yolonot` Works

yolonot runs as a single Go binary with zero runtime dependencies and evaluates every command through four layers:

1. Git-Tracked `.yolonot` Policy + ~1.6us Local Bash AST Parser

Before touching any model, yolonot checks your repo's .yolonot rules (deny > ask > allow) and parses the command into a full syntax tree using mvdan.cc/sh/v3/syntax. It splits chained commands, unwraps env/timeout, validates flags against ~160 CLI schemas (so git status is allowed, git push --force is not; find . -name is allowed, find . -exec is not), and enforces project-root path containment.

Across 5,000 real-world agent tool calls, 54.2% of commands are cleared locally in ~1.6 microseconds with zero API calls and zero unsafe leaks.

# .yolonot example in your repository root
allow-cmd  go test*
allow-path scripts/check.sh

ask-cmd    git push*
deny-cmd   *rm -rf /*
sensitive  .env*

2. Inspects Local Scripts Before Running `./script.sh`

In our eval suite, judging ./scripts/lint-check.sh by filename alone caught 0/14 Trojan scripts. When an agent runs sh scripts/check.sh or python3 fix.py, yolonot reads the script on disk, redacts secrets, blocks sensitive paths (~/.ssh, ~/.aws, .env*), verifies that an earlier command segment didn't overwrite the file, and attaches the code to the judge, catching 14/14 (100%) hidden script attacks.

3. Pluggable Local or Cloud Judges (Across 24 Agents)

Ambiguous commands go to your choice of classifier: fast cloud models, your existing claude / codex CLI login, or 100% local Ollama models (including the Apache-2.0 Kev-4B LoRA). Uncertain ask verdicts can escalate to a second model, cutting false alarms by 48.3%, with a hard rule that critical risk never maps to allow.

4. Tamper-Resistant & Sandbox-Aware

yolonot works alongside OS sandboxes and containers. It blocks agents from running yolonot pause/uninstall or editing .yolonot and hook configs. If an agent requests a host sandbox bypass (dangerouslyDisableSandbox, BypassSandbox, etc.), yolonot disables fast-allow and isolates session memory so a prior sandboxed approval cannot be replayed outside the sandbox.

Quick Start (10 Seconds)

# Install via Homebrew or Go
brew install sezaakgun/tap/yolonot
# go install github.com/sezaakgun/yolonot@latest

# Auto-detect installed agents and wire hooks
yolonot setup

# Import existing permissions from Claude, Cursor, Codex, OpenCode into .yolonot
yolonot rules discover

Check it out on GitHub and let me know what edge cases you hit in your agent workflows!

GitHub (Apache-2.0): https://github.com/sezaakgun/yolonot
Website & Docs: https://yolonot.dev

DEV Community: Seza Akgün