Benchmarking My MCP Security Scanner: 14 of 14 Findings, Zero False Positives

A scanner without numbers is a demo. A scanner with made-up numbers is worse. So I hand-labeled a small benchmark suite and ran the scanner against it, 1000 times per fixture.

The setup is deliberately boring:

• 5 synthetic MCP tool manifests: one clean (3 tools), one openly risky (4 tools), one realistic mixed shop (8 tools), one payments flavored (6 tools), and one leaky agent (3 tools) built to exercise the two newest rules. 24 tools total.
• Ground truth per tool: which rule IDs should fire. MCP-001 unsafe command capability, MCP-002 excessive scope, MCP-003 secret-like value, MCP-004 untrusted input into a sensitive operation, MCP-005 credential forwarded in metadata, MCP-006 write tool without tenant scoping. These are benchmark labels, not mcpscan's own rule IDs (MCP001 to MCP023).
• 1000 timed scans per fixture, single thread, Python 3.13, no warmup tricks.

The results:

fixture	tools	expected findings	detected	false positives	median scan
safe_notes	3	0	0	0	28 us
risky_ops	4	6	6	0	63 us
mixed_shop	8	3	3	0	94 us
payment_tools	6	3	3	0	73 us
leaky_agent	3	2	2	0	89 us

Aggregate: 14 of 14 planted findings detected, 0 false positives, median 82 microseconds per manifest.

Two honest notes. First, this measures the rule engine on synthetic manifests, not detection quality on real world servers; synthetic ground truth is easy to trust too much. Second, the clean fixture matters as much as the risky ones. Zero findings there is the actual test, because a scanner that cries wolf gets ignored, and an ignored scanner is worse than none.

The harness, fixtures, and raw results ship with the repo under benchmark/, so anyone can rerun it: clone, one command, verdict. Same zero dependency habit as the test suites.

Repo: https://github.com/glatinone/mcp-security-preflight

Related on this blog

• My Scanner Passed Until I Built a Harness That Lied to It on Purpose
• My MCP Security Scanner Missed 2026's Worst MCP RCE: Here Is the One-Rule Fix
• A happy-path MCP demo proves almost nothing about tenant isolation

---

If you want to run checks like these on an MCP server or a .claude/ directory before you install it, I maintain mcpscan, an open-source scanner for tool poisoning, risky permissions and leaked secrets. Issues and false-positive reports are welcome.