Humans in the Loop Miss a Third of Dangerous AI Coding Agent Requests

#ai #claudecode #anthropic #developertools

Originally published at https://tekmag.thsite.top/humans-in-the-loop-miss-a-third-of-dangerous-ai-coding-agent-requests/

A browser-based game testing AI coding agent permissions has collected over 400,000 human decisions, revealing that people approve roughly a third of dangerous commands. The data exposes how approval fatigue, familiar command patterns, and missing context combine to undermine the human-in-the-loop safety model that many AI coding Tools rely on.

When AI coding agents like Claude Code, GitHub Copilot Workspace, or Devin propose a command, most developers have been trained to review and approve each one — a practice detailed in our earlier coverage of securing AI coding tools. The assumption is that a human eye will catch malicious or destructive requests before they execute. A new dataset from Scale X suggests this assumption is flawed.

Alex Wauters, a Belgian software developer and former Staff Engineer at Uber, built a browser game called LLM Game (llmgame.scalex.dev) that simulates AI coding agent permission prompts. Players see real-world commands their tool might request, then choose approve or deny. Over 40,000 runs and 409,000 individual decisions, the average player missed about one in three threats. That miss rate isn't a quirk of bad players. It tracks directly onto how developers actually use permission prompts in production tools today.

Key Takeaways

Mean accuracy across 409,000 decisions was 66.3%, meaning roughly 33.7% of malicious commands were approved
npm run analyze had the highest miss rate at 64.7%, suggesting familiar script names lower suspicion
Approval fatigue, end-of-session pressure, and missing changed-file context all degrade judgment
Commands wrapped in package.json scripts are significantly harder to evaluate than raw shell commands, a pattern we explored in our guide to npm security best practices.
The ~93% Claude Code approval rate reported by secondary outlets could not be independently verified from an Anthropic source
Auto Mode in Claude Code reportedly catches ~83% of overeager agent behaviors, but ~17% still pass through

The dataset behind the numbers

Wauters' game presents players with realistic permission requests an AI coding agent might make during a development workflow. Commands range from obviously destructive, like deleting the entire repository, to stealthier variants that hide malicious payloads inside seemingly routine scripts. Players have roughly 60 seconds per decision, simulating the time pressure developers face when working through long permission queues.

The dataset covers four threat categories: obviously destructive commands, persistent mutation attempts, exfiltration and code execution payloads, and scope violations. Each category shows a different miss rate, with scope violations and exfiltration falling just below the overall average at 35.0% and 33.4% respectively. Obvious destruction is the easiest to catch at 11.7%, but that still means roughly one in eight players approve a command that would wipe their codebase.

Why npm run analyze trips people up

The single highest miss rate in the dataset belongs to npm run analyze at 64.7%. That command looks like a standard linting or type-checking step. In reality, the script could execute arbitrary code, exfiltrate environment variables, or modify files outside the project directory. The same pattern holds for npm run setup (48.0% miss rate) and npm run deploy (44.9%).

When these commands are pooled together, the npm-run family shows a 51.1% miss rate, compared to 33.4% for other exfiltration-style attacks. Familiar command names create a false sense of safety. Players see something they recognize from their daily workflow and approve without reading what the script actually does.

What the game reveals about approval fatigue

The time limit is deliberate. Real permission queues in tools like Claude Code or Copilot Workspace can stretch into hundreds of prompts per session. Each decision takes a fraction of a second, and after the fiftieth approval, cognitive load accumulates. The game measures this degradation directly: later decisions show higher miss rates than earlier ones.

Context gaps compound the problem. Players often see a command without knowing which files were changed, what the agent was trying to accomplish, or whether the request aligns with the task at hand. Without that context, the only signal is the command text itself. And command text, as the npm run analyze data shows, can be deliberately misleading.

Auto Mode and the gap between human and machine judgment

Secondary coverage has cited data suggesting Claude Code users approve approximately 93% of permission prompts. That figure appeared in reporting by The Register and on Hacker News but could not be verified from an accessible Anthropic source during this research run. The claim should be treated as unconfirmed until a primary source is published.

What is verifiable is the Auto Mode statistic. The Register reported that Claude Code Auto Mode detects roughly 83% of overeager agent behaviors, leaving about 17% that still pass through. That 17% represents a significant attack surface. Even with automated oversight, a meaningful fraction of dangerous commands reaches execution.

The gap between human and machine judgment matters here, echoing findings from our analysis of AI agent security risks in production environments. Auto Mode is stricter than human reviewers on average, but it is not infallible. Human reviewers, when they actually stop to evaluate each prompt carefully, perform worse than the tool designed to assist them. That inversion is the core finding of the dataset.

What this means for developers using AI coding tools

The data does not suggest abandoning permission prompts. It suggests redesigning them. The current model assumes a human can reliably evaluate thousands of short decisions under time pressure. The game shows that assumption is wrong.

Several practical adjustments stand out, building on the developer workflow automation principles we've covered before. First, tools should surface changed-file context alongside each permission request. A command like npm run analyze looks harmless in isolation. Paired with a diff showing the script was modified to include a curl command, the risk becomes obvious. Second, batching similar commands and asking for group approval reduces fatigue without sacrificing oversight. Third, defaulting to deny for unfamiliar scripts, especially those executed through package.json, forces a second look at commands that look routine.

The npm-run miss rate is the clearest signal that current permission designs are too permissive for script-wrapped commands. Tools that treat script execution as higher-risk than raw shell commands would close a known gap. So would any change that reduces the volume of decisions developers need to make in a single session.

Conclusion

The 40,000-run dataset from Scale X provides concrete evidence that human-in-the-loop safety degrades under realistic conditions. Approval fatigue, command familiarity, and missing context combine to produce a miss rate of roughly one in three. The data is specific enough to act on. Tools should surface more context per decision, treat script execution differently from raw commands, and reduce the total volume of approvals required. Until then, developers should assume that a significant fraction of permission prompts will be approved without careful review, and design their workflows accordingly.

Frequently Asked Questions

What is the LLM Game and who built it?

The LLM Game is a browser-based permission-choice game built by Alex Wauters, a Belgian software developer and former Staff Engineer at Uber. It simulates AI coding agent command approvals to study how humans evaluate permission prompts under time pressure.

How many decisions were collected in the dataset?

The dataset contains over 409,000 approve/deny decisions across more than 40,000 game runs. The mean accuracy was 66.3%, with a miss rate of approximately 33.7% on malicious commands.

Why do npm run commands have higher miss rates?

Commands like npm run analyze appear familiar and routine, so players approve them without examining the underlying script. The pooled miss rate for the npm-run family was 51.1%, compared to 33.4% for other exfiltration-style attacks.

Is the 93% Claude Code approval rate confirmed?

No. The ~93% approval rate cited in secondary coverage could not be verified from an accessible Anthropic primary source. It remains unconfirmed. The Auto Mode detection rate of ~83% is corroborated by The Register but also lacks a directly accessible primary source.

What can developers do to improve permission review?

Surf changed-file context alongside each prompt, batch similar commands for group approval, default to deny for unfamiliar scripts, and reduce the total volume of decisions per session through smarter tool design.