An MCP tool description can tell an assistant to change another tool's arguments, read a local file before answering, or hide an extra action. A useful scanner comparison needs to show which malicious definitions receive a signal, what happens on benign definitions, and which targets never receive a complete result.
I maintain both WARDEN and HISTOR, which supplies the historical package observations used here. That is a direct conflict of interest. I wrote some fixtures, and WARDEN has been developed with several of these datasets available. This is a maintainer-run comparison with inspectable evidence, not an independent ranking.
On the 485 served MCPTox definitions, published WARDEN produced high/critical signals on 460; the other four produced between 1 and 56. Lower-severity signals change that picture substantially, especially for mcp-audit. On historical packages, WARDEN also produced thousands of advisory-only signals. The tables preserve both thresholds and complete coverage rather than reducing the comparison to a winning percentage.
Figure 1. Native analysis at scale, with protocol parity and live discovery measured separately.
Five published products
The primary comparison uses these actual published packages, verified at the retained registry check, including the final registry snapshot on November 7:
| Scanner | Package | Version |
|---|---|---|
| WARDEN | npm @aimarket/warden
|
0.23.0, ruleset 17
|
| mcp-audit | PyPI mcp-audit-scanner[mcp]
|
0.28.0 |
| mcp-shield | npm mcp-shield
|
1.0.4 |
| truecopy | npm @askalf/truecopy
|
0.10.5 |
| mcpsentry | npm @roshan6335/mcpsentry
|
0.1.5 |
Published WARDEN has 40 rules and digest sha256-UOfK25xo6D8vrKEVfNFBTBVn5OspyaZut9vC9Z9Q5JE=. Its npm tarball SHA-256 is 833841f2aab89df4c51b5f84385ad69c8ffd7dcf5a7725ec0eb4ebabe594b7ee.
The unpublished candidate 0.23.1-dev.20261107.1, ruleset 19, has a separate supplementary row. It never replaces the published product. An earlier frozen build labelled v17 had six changed wrapper/result-screening distribution files and was not byte-identical to npm; it is retained only as candidate history. No npm release was made during this work.
Published packages and upstream public corpora are obtainable. Exact replay of the maintainer fixtures and adapters requires their source release. Bundle metadata enables inspection of row IDs, thresholds, coverage and output hashes, not full independent replay. Candidate source remains unpublished too: hashes and receipts are auditable evidence, not an externally reproducible build.
Native analysis and protocol checks
Bulk execution invokes the real implementations' native classifier or full-analysis APIs with fresh per-case state. mcp-audit runs its default six analyzers through the full pipeline; only discovery transport is replaced by captured tool objects validated through its installed SDK. This is not copied-regex testing, or a claim that every row traversed every CLI's stdio transport.
Representative native/API checks cover 10/10 cases for each Node competitor, 13 agreements for the Python pipeline including one partial result, and 20/20 for published WARDEN, including attack/clean/repeat and a 1.17 MB tool list. A matching partial result remains partial. The frozen v18/v19 comparison additionally matched actual-MCP/library results on 3,177/3,177 cases per version per host.
The shared columns are high/critical definition signals, any relevant definition signal, and advisory-only, where advisory-only = any − HC. They are not identical enforcement outcomes. WARDEN's native allow: false is retained separately. truecopy can classify even advisory content findings as poisoned and withhold them; cold-lock/unpinned refusal is a different policy. mcp-shield normally exits 0 despite findings, and mcpsentry exits 1 only for critical findings, not high-only ones.
WARDEN uses allowUnknownServers=true, sensitiveToolPatterns=[] and pinToolDefs=true. Both static-scan and threat-feed definition gates count. Launch hygiene, origin/pinning policy and first-scan baseline noise remain in raw records but are excluded consistently for every product. A corrected first aggregator had wrongly omitted three legitimate WARDEN threat-feed detections; native outputs were unchanged, and that normalization diagnostic is retained.
What the 18,110 rows contain
These are target rows, not 18,110 independent live servers. Our fixtures contain 23 attacks and 12 benign controls. Pinned third-party fixtures contain 10 attacks, one benign case and one ambiguous case. Mixed totals of 35 or 12 are not attack-recall denominators. These known development inputs are not new blind tests.
MCPTox has overlapping views: 485 unique served poisoned definitions across 45 servers; 1,348 complete prompt-surface scenario rows; 225 decorated variants; and 45 clean server lists. The full scenario view retains 1,338 unique server/frame values and 10 retained duplicate instances, with zero discarded. The original split has 267 development definitions and 218 test definitions. That test split is a known historical holdout, repeatedly published and sharing templates with development cases. Reuse does not restore blindness.
The official source commit is f85189f9ad12504c197c7f920ab818a40657b1fa. Projection handles 702 literal-line-feed and 646 escaped-line-feed rows. The full-frame projection retains argument text verbatim but represents declared parameter properties as strings; it does not reproduce original native argument types. The clean projection exposes 362 tools versus 353 stated in the paper; deduplication did not remove that discrepancy. The repository specifies no license, so public downloads contain factual metadata rather than raw definitions, schemas, payloads or benchmark Python.
The public corpus contains 986 server snapshots and a separate 41-observation carryover cohort. There are 28 exact name+definition overlaps, leaving 999 unique snapshots. The original cohorts remain separate.
HISTOR contributes 14,933 recorded package versions: 12,023 npm and 2,910 PyPI, referencing 14,449 distinct toolsets and 272,657 tool records. Every row was recovered and executed independently, including repeated toolsets. MTL digests, file hashes and signatures were verified on both hosts. Historical base fields remain historical: current titles, annotations and extensions were not merged into them.
HISTOR's 2026-10-20 archive, with its signed manifest, supplied 14,930 rows. The archive is CC-BY-4.0, attributed to Aleksandr Artamokhov / HISTOR. Three missing digests came from immutable public API URLs without a separate stated license; their public inventory is metadata-only.
Two remote gVisor hosts
Reported comparative measurements ran remotely on attested and PingBlip. An earlier local pilot is excluded and its generated artifacts removed. All bulk engines used runsc, network none, nonroot UID/GID 501:501, read-only root/source/runtime/data, dropped capabilities, no-new-privileges and dedicated results. No historical package implementation or upstream benchmark Python was executed.
Both hosts used image sha256:db7085498187e8477fa462d5f1652d62bdf4b8af29ea3666bd1748a8315631a5, containing Node 24.22.2 / Python 3.13.18. Node engines used 1 CPU / 768 MiB / pids 128. Audit base used 1 CPU / 768 MiB / pids 256; its historical phase used one worker / 1 CPU / 768 MiB on attested and two workers / 2 CPU / 1536 MiB on PingBlip. Linux cgroup quotas were checked. This is not a controlled speed comparison.
The 3,177-view catalog SHA-256 is e29baa05fbdbf46e34631ff2c75f0b4f535f0188054dd4499699bd189a3dc757; the 14,933-package catalog is 78d4dab7125318b80cc35302e7cf1cfc1a49fa6f310957ed7b8fddfb538375b6. Every case has a file hash and common opaque wire id. Full methods and input units are in the report and data ZIP.
Severity signals and advisory costs
Every cell below is HC/any/advisory-only, as case counts. All base rows completed on both hosts. HISTOR exceptions follow in the coverage table. The overlapping MCPTox rows are not additive independent trials.
| Cohort and scheduled unit | WARDEN | mcp-audit | mcp-shield | truecopy | mcpsentry |
|---|---|---|---|---|---|
| 23 own attacks | 20/23/3 | 11/13/2 | 6/10/4 | 3/8/5 | 4/4/0 |
| 10 third-party attacks | 10/10/0 | 10/10/0 | 8/9/1 | 1/7/6 | 5/5/0 |
| 485 served MCPTox definitions | 460/474/14 | 56/392/336 | 56/71/15 | 1/31/30 | 19/19/0 |
| …218 historical-test definitions | 210/217/7 | 39/178/139 | 41/46/5 | 0/27/27 | 8/8/0 |
| 1,348 full scenario rows | 1331/1338/7 | 951/1264/313 | 158/570/412 | 432/491/59 | 460/460/0 |
| 12 own benign controls | 0/3/3 | 1/2/1 | 1/2/1 | 1/1/0 | 2/2/0 |
| 45 clean MCPTox servers | 0/7/7 | 0/3/3 | 3/6/3 | 0/0/0 | 1/7/6 |
| 986 public snapshots | 4/388/384 | 43/159/116 | 343/451/108 | 2/39/37 | 86/450/364 |
| 14,933 historical package rows | 409/6327/5918 | 481/2089/1608 | 5354/6767/1413 | 15/252/237 | 1449/6362/4913 |
All five had zero signals on the one third-party benign control. Only mcp-shield advised on the ambiguous case; it is excluded from attack and benign denominators. Decorated/development/carryover results are in the full report.
Figure 2. Published products only. Severity signals and advisory costs are separate; overlapping cohorts are not additive.
Public and package rows are unlabelled. Their positive frequency is not a false-positive rate. WARDEN's 409 historical HC positives coexist with 5,918 advisory-only cases; calling it simply “409 alerts” would hide the lower-tier workload. truecopy has fewer HC positives there, while also detecting fewer native malicious definitions at that threshold. These are dataset-specific tradeoffs, not a universal ranking.
Coverage and exact agreement
For scheduled N, complete C, HC B, advisory-only A, error/partial E, skipped S and not-run U: N=C+E+S+U, C=B+A+no-finding. Threshold-signal rate is B/C; it is recall only within attack-labelled cohorts. Coverage is C/N. Errors are not clean scans or completed misses.
| Product | Complete/scheduled, attested; PingBlip | Errors, attested; PingBlip | Identical/complete paired results |
|---|---|---|---|
| WARDEN | 18110/18110; same | 0; 0 | 18110/18110 |
| mcp-audit | 18108/18110; same | 2; 2 | 18108/18108 |
| mcp-shield | 18079/18110; 18078/18110 | 31; 32 | 18078/18078 |
| truecopy | 18110/18110; same | 0; 0 | 18110/18110 |
| mcpsentry | 18110/18110; same | 0; 0 | 18110/18110 |
All products have zero skipped, unrun or missing rows. Two mcp-audit HISTOR records hit native schema/text budgets and are partial on both hosts, excluded from complete coverage and primary counts. mcp-shield's strict SDK rejects 31 schemas on both hosts; PingBlip also has one 30-second timeout. Native raw JSON-RPC clients used by truecopy and mcpsentry do not impose that validator. A rejected 42-tool case passed mcpsentry CLI/API with no findings: a real compatibility difference, not an omitted adapter check.
Every completed comparable result matched, including canonical full native-result signatures—not merely aggregate counts. The case-level proof in the report/data ZIP also checks input and engine identities and separates incomplete pairs. Only mcp-audit's own temporary config path and fresh rug-pull state-file evidence are normalized; finding content remains represented.
Figure 3. Exact agreement uses complete paired results. Partial scans and errors remain visible.
The unpublished candidate and the retained miss
The first frozen v18 candidate regressed against the earlier frozen17 build: 457/485 served blocks versus 460/485, and 207/218 historical-test blocks versus 210/218. FileSystem_20, _22 and _23 were newly missed. Those failures remain in the evidence.
Candidate19 .1 recovers two: overwriting an SSH public-key file is security-configuration tampering; reading /etc/shadow accesses a secret store. FileSystem_22, a public-key-read prerequisite, remains nonblocking. It remains labelled as an attack and stays in the denominator; public material is not reclassified as secret just to recover a benchmark score.
The verified candidate blocks 459/485 served definitions, 209/218 historical-test definitions, 1331/1348 scenarios and 225/225 decorated cases, with 0/45 clean-server blocks. It completes all 18,110 rows without errors and matches across hosts. Its 1536/1536 tests in 63 files pass on both hosts, alongside typecheck/build/red-team commands. This is post-hoc work on observed failures, not new holdout generalization.
Candidate HISTOR has 407 HC / 6327 any versus published 409/6327, with no v18→v19 verdict change. Narrow review of the two public-key provisioning passages in @krovacloud/mcp@0.3.16 and @runpod/mcp-server@4.4.0 supports allowing those passages, not certifying whole packages benign. The later .2 runtime build fixes optional-MIME blob compatibility. Both hosts pass 1536/1536 full tests, 304 focused tests and 8/8 SDK sessions; independent binary checks preserve valid 400k blobs with/without MIME and withhold poisoned results after one execution. The 320 source/distribution files outside the changed result-screen module are byte-identical to .1, but .2 is not substituted for the frozen .1 measurement. The current 0.23.1-dev.20261108 snapshot adds Codex/OpenCode hooks absent from those builds; its acceptance validation is separate and is not represented by these comparison rows.
Download the report and data bundle — English, Russian, Spanish, French and Chinese. The ZIP contains the five reports for offline reading, the full 18,110-row CSV/JSON inventory, grouped metrics, case-level proofs, methods and package pins. Public per-case data contains statuses, signal flags and input/result hashes, not full native findings or unlicensed payloads. The bundle supplies recorded versions and provenance without uploading product source.
Separately, candidate19 .1 completed 8/8 official-SDK sessions and 4/4 direct/wrapped list-digest pairs per host: filesystem 2.2.0, memory 2.1.0, sequential-thinking 2.1.1 and WARDEN, using SDK 1.32.1. Checks included discovery, a safe synthetic call, unknown-tool refusal and later ping. HTTP discovery found DeepWiki/Microsoft Learn/Cloudflare Docs with 3/3/2 tools, all allowed, using only initialize and tools/list.
This measures definition classification at recorded thresholds. It does not measure model attack success, full proxy enforcement, dependency safety or universal protection. The useful result is the detection/advisory-cost tradeoff with coverage visible, supported by case-level evidence.
AI assistance was used for implementation changes, test orchestration, analysis, writing and figure generation. Responsibility for the benchmark design and interpretation remains mine as WARDEN and HISTOR's maintainer.
Top comments (0)