I Built a Claude Code Skill That Refuses to Report Vulnerabilities It Can't Prove

TL;DR: I previously built a Claude Code skill that runs a full-stack security audit in one command — a pattern library that tells the AI where to look. Running it in real engagements taught me that "where to look" and "how to conclude" are different skills. So I built a second skill, security-assessment, that governs how conclusions are reached: every claim starts as a falsifiable hypothesis with the falsifier written before evidence is collected, evidence is graded E1–E6, anything undecidable goes into an UNKNOWN registry instead of being guessed at, and a CLI publication gate (sa validate) mechanically refuses to publish findings that don't meet the bar. The output is fewer findings — and findings a client can act on.

The two ways AI security reviews fail

Pattern detection is easy to judge: if eval() is there, it's there. Conclusions are where AI-assisted reviews break, and they break in exactly two ways:

• False positives. A pattern gets reported as a vulnerability without checking reachability, exploitability, or whether a control already exists somewhere else.
• Silent gaps. A component nobody actually examined gets waved through as "probably fine."

Both are worse than finding nothing. The first destroys trust in the whole report; the second hands the client false confidence. Everything in this skill exists to prevent those two failure modes.

Scan vs. assessment: different questions

A scan asks: "Does this pattern appear?"

An assessment asks: "Given how this system actually works, what could an attacker achieve — and can I prove it?"

The second question is what a client is paying for: a short list of findings backed by evidence, not a long list of maybes. A client can run a scanner themselves. The valuable work is promoting scanner output to provable findings — or honestly demoting it.

So the two skills are complements, not competitors: security-audit is the library of where to look; security-assessment is the engine for how to conclude. The assessment skill references the audit skill as an optional dependency.

Mechanism 1: Write the falsifier before you look for evidence

A checklist can't be refuted. "eval() exists, therefore report" is how false positives are mass-produced.

Instead, each claim starts as one falsifiable sentence about attacker capability, and — this is the crucial part — the falsifier is written before any evidence is collected:

H-007
  statement:  A user in tenant A can fetch tenant B's invoices by swapping the ID
  falsifier:  A tenant predicate is applied at the query layer on every path
              that reaches this handler
  status:     OPEN | REFUTED | SUPPORTED | INCONCLUSIVE

Then you search for disproof first. Enumerate every place the control could live — middleware, guards, ORM default scopes, database row-level policies — and check each one. If you find it, the hypothesis is REFUTED: cheap, correct, and it never becomes report noise. If you don't find it, you haven't merely "not noticed" a control; you've deliberately established its absence. That's a far stronger claim.

This has saved me from embarrassing reports in practice. In one assessment, an apparently dangerous public endpoint — it looked like it rewrote production data — turned out to be guarded so it only ran outside production environments. A naive scan would have filed that as Critical. A webhook with properly implemented signature verification was the same story: looking for the falsifier first is how you discover you're actually protected.

Mechanism 2: The UNKNOWN registry — separate "safe" from "not checked"

Some things static analysis cannot settle. With the deployed IAM configuration, can anyone actually invoke this public function? Is the key that leaked into git history still valid?

The skill refuses to guess. Those go into an UNKNOWN registry, and the tooling enforces the consequence: while an UNKNOWN sits on a finding's exploitation path, that finding cannot be promoted to CONFIRMED.

Every report therefore carries a "what was not assessed" section: not "this area is safe," but "this area could not be examined, and here is the exact command that would settle it." An honest not assessed beats a confident guess — that's the core philosophy of the whole design. In PASSIVE mode (static analysis only, no traffic to the target system), deployed IAM and leaked-key validity are structurally undecidable, and saying so, with resolution steps attached, is a useful deliverable rather than an admission of failure.

Mechanism 3: Evidence grades, triage, and a gate that says no

Every candidate — from a tool, a subagent, or my own reading — must pass the same pipeline before it may become a finding:

candidate → reachability → existing controls → exploitability → impact → classify

Evidence is graded:

Grade	Meaning
E1	Observed directly in an authorized environment
E2	Full code path traced, every intervening control checked
E3	Sink located, one link inferred rather than read
E4	Configuration read from the authoritative source
E5	Tool output, unverified
E6	Inference or framework convention

E5/E6 alone can never support a finding. If Semgrep says "Critical," that is not severity evidence — and sa validate will reject a report that treats it as such. Promoting "the tool said so" into a finding is precisely what a client is paying you not to do; they can run the scanner themselves.

The publication gate also refuses findings that: have no evidence at all; claim Critical without E1/E2 evidence; claim CONFIRMED while an UNKNOWN is open on the path; record no controls_checked (no account of where a defence was looked for); or are Critical/High without a human-review flag. It also fails the run on open investigation threads, unexamined attack surfaces, and credential-shaped strings anywhere in the workspace.

Severity is computed, not asserted:

score = 0.35·T + 0.25·E + 0.25·B + 0.15·X     (each factor 1–4)
        Critical ≥ 3.50 · High ≥ 2.80 · Medium ≥ 2.00 · Low ≥ 1.40

T technical impact, E exploitability, B business impact, X exposure. CVSS alone is rejected: it cannot know whether the endpoint is internet-facing, or whether the data behind it is a payroll table or a marketing blog. Every finding stores its four factors, so any rating can be re-derived or challenged; a manual override is allowed by one band, with the reason recorded in the report.

Multi-agent collection, single-agent promotion

A real assessment runs as an orchestrator plus specialized subagents:

[main] Orchestrator (design · triage · promotion)
  ├── [sub A] Backend (authorization, injection)
  ├── [sub B] Frontend (XSS, token storage)
  ├── [sub C] Secrets, supply chain, CI/CD
  └── [sub D] Privacy, compliance

Subagents produce hypotheses, evidence, and candidates — nothing more. Only the orchestrator promotes a candidate to a finding. There's a structural reason: a control in one layer often neutralizes a weakness in another, and that cross-layer context is exactly where naive per-layer reviews generate false positives. Only the side that integrates can see it.

And the orchestrator does not take subagent reports at face value. One subagent flagged what looked like personal data left in source comments (High concern). The orchestrator opened the actual file: the values were keyboard-mash strings like akdjhjf... and ZZZ... — test data. Recorded as FALSE-POSITIVE and dropped from the report.

The most dangerous LLM failure mode is describing plausible-but-nonexistent code with total confidence. So the skill mandates, in writing: re-read the actual file before every claim. Not the subagent's summary — the artifact itself: the git diff, the file contents, the observed behavior.

Why a skill plus a stateful CLI, not one long prompt

The skill ships with a state-holding CLI (sa). Three reasons:

• Anti-drift. Long assessments fail by chasing one interesting thread and forgetting nine others. The investigation frame lives on an explicit stack — sa stack dump shows exactly where you are.
• Resumable and delegable. State lives in one directory on disk, so work survives across sessions and can be handed to subagents mid-flight.
• The gate is mechanical. sa validate enforces the evidence rules whether or not anyone remembers to be disciplined that day.

Client data never enters the skill directory. Each engagement gets an isolated workspace; only generalized lessons flow back into the skill, through a sanitizer that blocks customer names, domains, IPs, account IDs, and credential shapes — and even those require human approval.

What changed in practice

• Fewer findings, higher quality. Not "30 findings, 20 of them noise," but "6 real ones." A client will actually fix 6.
• "Not assessed" became something I can write without flinching. With the resolution command attached, it's a deliverable, not a gap.
• Findings turn into prevention. Every confirmed finding class maps to a CI/lint guardrail that would stop the next instance. A report is finished when it says not only what is broken, but what stops the next one.
• False positives became a catalog. Demoted candidates are recorded with reasons, so the next assessment doesn't re-litigate the same pattern.

Where this is NOT the right tool

• You want a fast pattern sweep. Use the security-audit skill instead — that's what it's for. This skill is slower by design.
• You want it to replace a security engineer. It doesn't. Severity ("how bad if real") and AI confidence ("how sure am I that it's real") are reported as separate numbers, because conflating them is how a real Critical gets ignored. Human review is mandatory-flagged for Critical/High, authorization, authentication, business logic, cloud IAM, and cryptography.
• You need active exploitation. The default posture is PASSIVE: read code, config, and IaC; no traffic to the target. Destructive techniques are never automatic and require written, per-technique authorization.

Install

/plugin marketplace add toshipon/claude-code-security-assessment-skill
/plugin install security-assessment@toshipon-security

Optionally, add the companion pattern library (the skill works without it, but detection is sharper with it):

git clone https://github.com/toshipon/claude-code-security-audit-skill.git ~/.claude/skills/security-audit

Then ask Claude for an assessment, or invoke /security-assessment directly.

The honest summary of the whole design: don't stop at "let the AI do a security review." Design the process until the AI can only output findings it can prove — and say so, explicitly, about everything else.

---

I'm toshipon — an SRE and indie hacker in Japan, founder of BUILD GEEKS. This is part 2 of my Claude Code security skills series (part 1: the security-audit skill). If you're shipping AI-built software and want this kind of evidence-first review on it, DM me on X.