MCP registries list tens of thousands of servers. Most of what is written about their safety comes from reading them: the tool descriptions, the ...
For further actions, you may consider blocking this person and/or reporting abuse
Useful dataset, and the "Testing the tester" section is what makes me trust the rest of it.
One case I'd like to understand. Disclosure: I'm on the Auten team, and our npm package is a thin launcher. HISTOR has it as ok with one tool, install_auten, because the runner it hands stdio to isn't on your sandbox. On a machine where the runner is installed, tools/list returns the real computer-use tools instead. So "the tool set everyone else gets" is honestly two sets, depending on local state.
How does WARDEN treat that? Would someone with the runner installed see their server flagged as a different tool set from HISTOR's pin? And is there a way for a publisher to declare the expected set (in server.json, say), so it can be checked rather than only diffed? There are probably a fair number of launcher-style packages like this in the "needs something the machine lacks" bucket too.
Following up on the launcher case — I recovered HISTOR’s stored sandbox snapshot: one install_auten tool. I’ve also prepared a capture helper with instructions for comparing actual environments.
It runs the supplied launch command and collects initialize metadata and paginated tool lists. The collector makes no tool calls, creates no trust pins and uploads nothing itself. Starting the launcher still executes its code. The saved tools are client-normalized definitions, not a raw wire transcript.
Could you share two captures from the differing environments, ideally using the same pinned @autenai/mcp@0.1.5 launcher? For each, please include:
Thanks for digging the old snapshot back out, and for spelling out exactly what the collector does and doesn't do.
Honest answer: we won't run captures against the installed runner and post them in this thread for now. That's our call about publishing the full tool definitions, not a concern with your helper. The publisher-declaration route you suggested is the one we're working toward (a hosted tool-sets file), but it isn't live yet, so there's nothing on our side to check today.
Agreed that a digest alone shouldn't stand in for real snapshots.
Understood, thanks for clarifying. We’ll keep the recovered snapshot labelled as a historical sandbox observation; it cannot verify the currently installed runner. A hosted tool-sets file would be publisher-declared information, separate from a runtime capture. There’s nothing further to verify on your side while that file isn’t available.
Thanks, and fair point. For a launcher, "the tool set everyone else gets" really means "the set a machine without your runner gets", and that is what HISTOR holds for your package: the one install tool. We have changed that sentence in the post.
How WARDEN treats it:
scan --histordigests the tools/list your client actually got (MTL/1) and asks HISTOR whether it has seen that digest for the package. With the runner installed it has not, so the answer isdifferentand WARDEN shows an advisory finding. It never blocks. Without the runner the answer issame.Your comment also showed that finding was worded wrong for this case: it named only a version HISTOR had not reached or a package that is not the published one, which reads like tampering. WARDEN 0.15.0 names local state first (a program the package hands over to, a real key) and gives the tool count on both sides.
Declaring the expected set: you can now. Put the MTL/1 digest of the set your runner serves in your server.json, under
_meta→io.modelcontextprotocol.registry/publisher-provided→dev.modelmarket.histor/tool-sets, as{"mtl": "1", "sets": [{"digest": "sha256-…", "count": 23, "when": "the runner is installed"}]}, and publish that version to the registry. On a machine with the runner,npx -y @aimarket/warden scan --jsonprints the digest asmtlDigest. HISTOR reads the declaration at its next daily crawl and answers /check with it, marked as your declaration, not an observation; WARDEN 0.15.0 then reports that set as declared by the publisher, at low severity, instead of unseen. Its own checks still read every tool. Details: github.com/alexar76/histor/blob/ma...The "needs something the machine lacks" bucket: yes, launchers that exit when their program is missing are among those 236. HISTOR has no set for them, so WARDEN has nothing to compare and flags nothing, and a declaration works for them too.
Thanks for changing the wording and shipping the declaration path so fast. Naming local state first in the WARDEN finding is exactly what makes it readable for a launcher.
We'll look at adding a tool-sets declaration on our next registry publish. One follow-up: our runner is versioned separately from the npm launcher (and per OS), so the set it serves can change while server.json stays the same. Is the intended pattern to list each known digest in "sets" and republish when it changes, or does HISTOR expect one declared set per published package version?
The container-address reuse bug is a good reason to treat observer attribution as part of the test, not just infrastructure. I'd add a deliberate handoff case: one container leaves a delayed connection, the next receives the same address, and neither run may inherit the other's canary. Pairing that with your incomplete-on-missing-DNS rule would keep a clean verdict distinct from a run the observer couldn't fully account for. That distinction is more useful to an adopter than one overall 'answered' percentage.
Thanks — we added a synthetic observer test for the handoff you described, and it failed. It uses collector records, not live Docker address reuse. A's queued traffic could be attributed to B even with the start-time filter. We now check the canary's run tag: foreign-only records are excluded from B's evidence, and B is marked incomplete (
foreign-run-traffic), not clean.Your reporting point exposed another bug: new incomplete reasons replaced existing ones, so a missing DNS log could disappear. They now accumulate, and any such reason prevents a clean verdict. We wrote up both fixes and the test's limits here: dev.to/alexar76/a-period-bypassed-...
@alexar76 Right, and that's the number worth taking next: 182,157 definitions over 7,656 servers is about 24 tools each, and every definition is resent with each request. Counting the tokens of one typical manifest would turn that into a cost.
Good call, so we counted. HISTOR keeps the current tool list of every server it observes, so this is all of them rather than one typical manifest: 18,204 packages (plus 16,170 remote servers), each list counted the way a client sends it to a model ({name, description, input_schema} per tool), with the o200k tokenizer.
A median package's list is 1,272 tokens; the mean is 3,812, the 90th percentile 8,361, the 99th 39,612. Remote servers look the same, median 1,328. 1,447 packages are over 10,000 tokens and 31 over 100,000; the largest is 304,780, more than a 200k context window. (The set has grown since the post: today the mean is 18.5 tools and the median 8.)
In money, per 1,000 requests at $2 per million input tokens, a Sonnet-class list price: $2.54 for a median server, $7.62 for the mean, $16.72 for the 90th percentile, paid on every request before the user has said anything. Prompt caching brings it to about a twentieth when the client keeps the list unchanged at the front of the prompt, and loading schemas only when a tool is about to be called avoids most of it.
Method, the full table and the caveats (one tokenizer; providers' own tool-use text not counted; every listed server weighs the same, used or not): github.com/alexar76/histor/blob/ma...
Thanks for the nudge.
running a few mcp servers on my homelab made me think about this a lot. the sandbox test answers "what does it do on first start", but the part that keeps me up at night is the runtime behavior — a server that passes initialize can still hit a network endpoint or read a file it never declared, and only when a tool is actually called. i ended up writing a small monitor that tracks which local services are alive, what they connect to, and how much vram they burn, so at least i can see the drift between what a tool says it does and what it does. github.com/SikamikanikoBG/homelab-monitor — not mcp-specific, but the "who is actually running what on my box" question is the same one. the "testing the tester" section is the right instinct. a log that lies is worse than no log at all.
The crawl records the first start. Inside the sandbox the observer speaks initialize and tools/list. That is the start, not what a tool does on a later call.
A sandbox that reports wrong things is worse than none.
The placeholder key finding is wild - 1,039 servers told you their tools but didn't need a key. Makes you wonder how many are doing that on purpose vs cargo-culting the check.
1,039 started with a placeholder key. A key that works nowhere was enough for every one of them to tell us its tools. None needed a real key to do that; they just refused to start without one.
The run counts those servers. It cannot see whether that was on purpose or copied by habit.
The 41% cost startup time, not tokens: a server that never answers adds no tool list. The bill is the 59% that did, since every tool list they return rides along in each request.
5,259 of 12,915 never answered, so they returned no tool list. The 7,656 that answered did return one. Between them they listed 182,157 tool definitions. This run does not measure token cost.
Seeing 292 PyPI servers die on import after mcp 2.0.0 is the result I'd send every maintainer, since 207 of the 269 you retried came back with mcp<2. I also appreciated the breakdown of the 295 decoy file opens, because the ~/.netrc reads from requests would have alarmed me too. As the crawl continues, will you track how many of those 292 recover once maintainers add an upper bound on mcp?
Following up with the fresh-install count you asked for: we rechecked the reconstructed 292-package cohort on 24 October, without adding our own mcp<2 constraint.
Fourteen answered initialize and tools/list: twelve had newer releases, while two worked at the same version after we installed their declared MCP extra. Eleven of the twelve newer-release successes have a matching fresh old-version control that still fails; the remaining old release is no longer available from PyPI.
Seven confirmed pairs add an upper bound, three move to SDK 2, and one changes an already bounded range with a Pydantic-related old failure. Another 276 still fail on imports, APIs or dependencies; two need real authentication or a prepared project index that our synthetic environment did not provide.
I also need to correct my earlier reply: not all 292 were proven MCP 2 failures. Our old classifier included missing extras and MCP 1/Pydantic errors because it matched the FastMCP path anywhere in a traceback.
The follow-up explains the controls and limitations: dev.to/alexar76/we-rechecked-292-f...
Success here means answering initialize and listing tools; safety and authenticated functionality were not established.
The 292, the 207 of 269, and the 295 are the 13 October snapshot. The 292 are PyPI servers that depend on mcp with no upper bound, install 2.x, and die on import. We started 269 of them again with pip held to mcp<2, and 207 answered. 295 opened a decoy credential file.
The crawl continues. Each pass installs the latest version and records whether that run answered initialize and tools/list, and what it printed as it exited. That exit text is what placed a server in the 292. A later pass records the new run. The snapshot does not carry a separate count of how many of those 292 recover once a maintainer adds an upper bound.