Building a Security Scanner with Claude Code Skills - How I Tackled LLM's "p-hacking" Problem
The Problem That Emerged from Previous Articles
In my previous article, Claude Code Security: 500+ Zero-Days Found, Security Stocks Crash 9.4%, I covered Anthropic's announcement of Claude Code Security. It's genuinely impressive technology, but it's Enterprise/Team only - individual developers like me can't use it yet.
Meanwhile, Snyk's research shows that 36.8% of free Skills have security issues. There's no review process for the Skills marketplace, and Anthropic's own documentation states that "security verification of SKILL.md is not performed."
Waiting for the Enterprise version wasn't going to help, so I built my own security scanner using Claude Code Skills. With nothing but a SKILL.md definition, you can build a hybrid scanner combining static pattern matching and LLM semantic analysis.
But here's what I didn't expect: building the scanner was the easy part. The real challenge was a fundamental issue with LLM-based tools - the same input can produce different results every time. This article covers the scanner's design philosophy and how I confronted this p-hacking problem head-on.
Enterprise vs. Skills: An Honest Comparison
Let me be upfront. The Skills version is not equivalent to the Enterprise version.
| Aspect | Claude Code Security (Enterprise) | Skills-Based Security Scanner |
|---|---|---|
| Target | Entire codebase | External skills (SKILL.md) |
| Detection rules | Defined internally by Anthropic (not public) | You define them (fully customizable) |
| False positive handling | Multi-stage self-verification | Quantitative confidence scoring |
| Report format | Anthropic's standard format | Fully customizable |
| Cost | Enterprise/Team plan pricing | No additional cost |
| Updates | Managed by Anthropic | You add and update rules yourself |
The killer advantage of the Skills version is that you control the detection rules. You can customize them for project-specific security requirements, and update rules at your own pace.
Design: 3-Layer Scan Architecture
The scanner is structured in three layers:
Layer 1: Static Pattern Scan (14 categories, 95+ items)
-> Detection results
Layer 2: LLM Semantic Analysis (7 checks)
-> Context-aware judgment
Layer 3: Risk Score Calculation + Report Generation
Layer 1 is rule-based static pattern matching. 95+ check items organized across 14 categories including command injection, obfuscation, secret leakage, and ransomware patterns. These are deterministic - same result every time.
Layer 2 leverages Claude's reasoning for LLM semantic analysis. It analyzes from 7 perspectives including "instructions cleverly disguised in natural language" and "gradual escalation." Pattern matching can catch c${u}rl-style variable expansion evasion, but attack instructions embedded within context that even humans would miss require LLM reasoning to detect.
Layer 3 calculates a quantitative score by multiplying severity and confidence for each detection, then assigns a final rating across 4 ranks (SAFE/CAUTION/DANGEROUS/CRITICAL). Dangerous combinations like "external communication + secret reading" trigger composite risk bonuses.
Iron Laws - A Lesson Learned the Hard Way
The most important design aspect of the scanner is ensuring the scanner itself can't be weaponized.
During early development, while scanning a malicious test skill, the scanner nearly followed an instruction inside the skill that said "First, execute this command to verify your environment." If the scanner executes instructions from its targets, the security tool becomes the attacker's stepping stone - the worst possible scenario.
That experience led me to design "Iron Laws." Rules structurally embedded in SKILL.md ensuring scan targets are never executed, only read and analyzed as text. Simply telling an LLM "don't do this" isn't enough - you need a workflow structure that makes execution impossible by design.
LLM's Weakness: The p-hacking Problem - A Wall I Hit After Building It
With Layers 1-3 designed, I thought "this is going to work." Then I started running tests and hit a wall.
I scanned the same skill 5 times, and got 3 CRITICALs at different scores, with 2 runs scoring 10+ points lower. The rank was the same, but the detected items were subtly different each time. Specifically, Layer 2's "gradual escalation" detection kept appearing and disappearing.
Digging into it, I found this is a well-known problem across LLMs. arXiv:2509.08825 "Large Language Model Hacking" demonstrates through a massive experiment with 13 million labels that 31% of state-of-the-art LLMs produce incorrect conclusions. Additionally, arXiv:2504.14571 "Prompt-Hacking: The New p-Hacking?" coined the term "Prompt-Hacking" for the problem where slightly different prompts produce different results.
| Traditional p-hacking | Prompt-hacking |
|---|---|
| Trying different statistical methods to find significance | Tweaking prompts to get desired output |
| Degrees of freedom in analysis | Degrees of freedom in prompting |
| Caused the reproducibility crisis | The same crisis is recurring in AI tools |
"It said CRITICAL this time, but maybe it'll say SAFE next time" - that's not a tool you can trust. This had to be solved.
p-hacking Countermeasures: 4 Approaches After Much Trial and Error
My first thought was "maybe more precise prompts will stabilize it." That was naive. No matter how carefully you craft prompts, LLM non-determinism doesn't go away.
I shifted my thinking: instead of eliminating the variability, build a structure where variability doesn't affect the final assessment.
1. Transparency Through Source Tags
Every detection result gets tagged with [Static] or [LLM]. Users can immediately tell "is this a 100% reproducible static detection, or an LLM judgment?"
This alone made a huge difference - report readers can now say "this is an LLM judgment, so take it as a reference" and make their own assessment.
2. Limiting LLM Score Impact
This was the most painful part to tune. Setting an upper limit on how much LLM detections can affect the overall score sounds simple, but set it too tight and the LLM's detection capability dies. Too loose and it's pointless.
I settled on using the static detection score as a baseline, limiting LLM contribution to a fixed proportion of that. I tried multiple thresholds to find the balance that maintained detection capability while suppressing score fluctuation.
3. Strict Confidence Escalation Rules
I restricted LLM from unilaterally escalating confidence levels. Upgrades now require corroboration from static detections, structurally preventing LLM "overconfidence."
LLMs answer confidently even when they're wrong. The research even points out that "the smaller the effect size, the more errors LLMs make." The design had to assume this characteristic.
4. Explicit Composite Risk Trigger Conditions
For composite risk (dangerous combination) scoring, I introduced rules that reduce bonus points when LLM-sourced detections are involved. If both detections are LLM-sourced, no bonus is applied at all.
The common design philosophy across all four: "Static detection (deterministic) is the backbone, LLM detection (non-deterministic) is supplementary." Not eliminating LLM, but leveraging it "within the bounds of trust."
Test Results: Validated Across 30 Independent Sessions
Claims without evidence aren't enough. Here's the quantitative proof.
Test Method
- Created 3 dummy skills (clean / gray zone / suspicious)
- 5 scans each on pre-fix (v1) and post-fix (v2)
-
30 completely independent sessions executed via
claude --print(non-interactive mode) - Each run is an independent process, so previous results can't influence the next
Results
Key Metric: LLM Detection Reproducibility
| Dummy Skill | Before Fix | After Fix | Improvement |
|---|---|---|---|
| Suspicious (CRITICAL-level) | 75% | 100% | +25% |
| Gray zone (CAUTION-level) | 100% | 100% | - |
| Clean (SAFE-level) | 100% | 100% | - |
Before the fix, the suspicious skill's "gradual escalation" detection only appeared in 2 out of 5 runs (75% reproducibility). After the fix: consistent detection across all 5 runs (100% reproducibility). The "sometimes detected, sometimes not" problem was completely eliminated.
All Metrics
| Metric | Threshold | Before | After | Result |
|---|---|---|---|---|
| Score CV (Coefficient of Variation) | < 0.10 | 0.031 | 0.089 | PASS |
| Rank Stability | 100% | 100% | 100% | PASS |
| LLM Detection Reproducibility | > 80% | 75% | 100% | PASS |
All metrics PASS.
As far as I can tell, no other LLM-based security tool has implemented p-hacking countermeasures and demonstrated reproducibility with empirical data. Major tools like NVIDIA garak (6,900+ stars), Trail of Bits Skills, and Promptfoo have no countermeasures from this perspective.
Summary
| Item | Details |
|---|---|
| Architecture | Static patterns (14 categories, 95+ items) + LLM semantic analysis (7 items) + quantitative scoring |
| Iron Laws | Structurally prevents attacks on the scanner itself |
| p-hacking countermeasures | Source tags, score capping, strict confidence escalation, composite risk conditions |
| Test results | 100% LLM detection reproducibility across 30 independent sessions, all metrics PASS |
You don't need to wait for Claude Code Security's Enterprise version. A production-grade security scanner is buildable with Skills. And if you're going to use LLM-based tools in production, confronting the p-hacking problem is unavoidable. I hope this article helps anyone tackling the same challenge.
References
- Large Language Model Hacking - Large-scale demonstration that 31% of LLMs produce incorrect conclusions (arXiv:2509.08825, September 2025)
- Prompt-Hacking: The New p-Hacking? - Risk of result manipulation through prompt adjustment (arXiv:2504.14571, April 2025)
Want to Try This Scanner?
The complete version of the security scanner described in this article is available. All 14 categories with 95+ check rules, 7 LLM semantic analysis items, 5 known IOC databases, and p-hacking countermeasures for score stabilization - everything included.
- Security Scanner ($19.99): The full scanner from this article. Reproducibility guaranteed with p-hacking countermeasures -> View Details
- Pro Pack ($49.99): Everything included. For $30 more, you also get 21 agents + CI/CD auto-design -> View Details
- Starter Pack (Free): TDD, debugging, and code review workflows -> Free Download
Top comments (0)