We installed 12,915 MCP servers and started each one. 41% never answered.

MCP registries list tens of thousands of servers. Most of what is written about their safety comes from reading them: the tool descriptions, the README, the code. We wanted the other half. What happens when you actually install one and start it, the way a client does? Does it answer? What does it ask for? What does it touch while it starts?

So HISTOR, our public log of MCP servers, now installs the npm and PyPI servers it finds, starts each one in a sandbox and writes down what happened. The registries it follows list 47,586 npm and PyPI packages; it runs up to 8,000 per crawl. This post is the state on 13 October 2026, after 12,915 of them.

How we ran them

• The latest version, installed fresh. npm with --ignore-scripts (install scripts then run on their own, traced); PyPI wheels only, nothing built from source.
• gVisor (runsc), an unprivileged user, a read-only root, 512 MB and one CPU. 360 s to install, 150 s to start and answer.
• No route out. Every DNS name resolves to our traps, which record what reaches them and forward nothing.
• Decoys. ~/.ssh, ~/.aws, ~/.npmrc, ~/.netrc, shell history and a few more in HOME, a decoy .env in the working directory.
• The observer speaks MCP: initialize, tools/list, then up to 15 tools called once each with canary arguments. gVisor writes its own syscall trace, outside the sandbox, so a package cannot hide what it opened, started or dialled.
• What it asks for, as placeholders. A server that exits with MYTOOL_API_KEY is not set or a usage text naming serve is started again with MYTOOL_API_KEY=histor-envkey-<run> or serve. The names only ever come from the package's own error text or its registry entry. The values are ours and work nowhere.

59% answered

7,656 of 12,915 answered initialize and tools/list: 64% on npm, 52% on PyPI. Between them they listed 182,157 tool definitions, a median of 10 per server (one listed 3,631).

1,239 of those started only with something they asked for: 1,039 with a placeholder key, 177 with a start argument, 23 with both. A client config would have given them that, so we count them as working. But look at the 1,039 again. A key that works nowhere was enough for every one of them to tell us its tools. None needed a real key to do that; they just refused to start without one.

Why 5,259 never answered

We read what each package printed as it exited, last run first.

• Almost half, 2,476, are broken as published. 874 crash with an error of their own. 636 will not install: no wheel for some dependency, conflicting pins, the wrong platform. 490 hang, stay silent or speak something that is not MCP. 292 were written for MCP SDK 1.x (next section). 184 miss one of their own files, ship TypeScript that Node will not load from node_modules, or import a dependency they never declared.
• 793 ask for a key or a setting and stop, even with a placeholder. The usual reason: they check the key's format at startup (must start with sk_) or call their API before answering anything, and our trap is not their API.
• 924 print a usage text. Some want a start argument we pass next time (serve, mcp, --stdio). Many are command-line tools that mention MCP somewhere. With the 666 that declare no command at all, or exit 0 having done something else, up to one package in eight listed as an MCP server is not a stdio MCP server when installed the way its listing says.
• 164 start an HTTP server instead of answering on stdin. Their next run gets --transport=stdio.
• 236 need something the machine does not have: another program (git, Java, Bun, a desktop app), the internet at startup, or another operating system.

MCP SDK 2.0 broke 292 PyPI servers

mcp 2.0.0 (28 July 2026) renamed FastMCP and changed the low-level server's decorators. A PyPI server that depends on mcp with no upper bound installs 2.x today and dies on import: cannot import name 'FastMCP', or 'Server' object has no attribute 'list_tools'. We started 269 of them again with pip held to mcp<2, and 207 (77%) answered. Anyone who installs them today gets the crash, not the server.

What they did in the sandbox

11,891 runs carry a gVisor trace.

• 2,239 npm packages ran install scripts, about 30% of the npm packages we ran. 580 of them tried to reach the network while installing, mostly to fetch a prebuilt binary (prebuild-install, node-gyp's headers, esbuild). Here that fails. On your machine it runs with your credentials in reach.
• 1,484 reached for the network at startup, before any tool was called. The names they asked for most: pypi.org (613), github.com (478), nodejs.org (245), registry.npmjs.org (243). Mostly version checks and downloads.
• 295 opened a decoy credential file (not counting .env). Before anyone panics: Python's requests reads ~/.netrc before every HTTP request (141), npm and npx read ~/.npmrc (103), boto3's credential chain reads ~/.aws (39). Each of the other 17 (~/.kube, ~/.docker, ~/.ssh, shell history, gh's config) was opened by a server whose job is Kubernetes, Docker, SSH or "ops context". Opening a credential file is evidence, not a verdict. What matters is whether it fits the server's purpose and where the bytes go next.
• 73 runs sent the placeholder key over the network. Most went to an identity provider (Microsoft's login 17 times, Google OAuth 7) or to the vendor's own API. Each package's page shows where.
• Decoy content left three packages, each time when a tool forwarded the canary arguments we called it with to its own API, which is what tools do. A fourth record said a package had mailed decoy data to an extra recipient at startup. That one was ours; see the last item below.

Testing the tester

A sandbox that reports wrong things is worse than none, so here is what we got wrong on the way.

• 880 servers looked silent. They had printed their help to stdout, and we kept only stderr. Now we keep both and the exit code; 179 are really silent.
• About 400 "install failed" because they require Python 3.13 or 3.14 and our image had 3.12. pip then says from versions: none, which reads like a missing package. The sandbox now picks the image from Requires-Python.
• 1,917 Python servers were flagged as "reached out at startup". asyncio's event loop writes a byte to its own socketpair; our trace reader saw a send with no destination and listed it as network. gVisor does not print the file descriptors socketpair returns, so the rule is now: a send that names no destination, on a socket that socket() did not make and no connect() reached, after a socketpair, stays in the process. All of them have since been traced again, including servers that fail to start, whose pages kept the old reading until we noticed they were never re-read.
• 126 "persistence" writes were relative paths the old reader could not resolve, such as node-gyp's buildcheck.gypi. gVisor prints the directory next to each openat; we were not reading it.
• One missed case, found by our red team (MOMUS). After we moved to Debian images, one of its 72 behaviour cases went unseen under load: glibc's resolver gave up on a late DNS answer, the package never connected, and the run still said "complete". The resolver now waits longer, and a DNS query our logger never saw makes the run incomplete, never clean. Every change to the observer now has to pass those 72 cases again, or HISTOR's release gate stays red.
• A real package was shown mailing decoy data to an extra recipient. It had not. The traps know a run only by its container's address, and Docker hands a freed address to the next container. Our window opened before docker run, so it took in the last request of a red-team fixture that had held the address a moment earlier. The recipient carried the fixture's run tag, which is how we caught it. Evidence is now matched only from the container's own start to its removal, and by when a connection opened; every record holding another run's canary is being traced again.

If you publish an MCP server

• List your tools without a key. Ask for it when a tool is called. Do not check its format at startup.
• Default to stdio, or name the flag that turns it on in your error text.
• Pin mcp<2 or move to 2.x.
• Ship one command. If your package also has a CLI, put mcp in the server's name.
• Keep stdout for JSON-RPC. Logs, banners and help go to stderr.
• Set Requires-Python honestly and publish wheels.
• Do not download binaries at install or startup. Ship them as per-platform optional dependencies.

Check what you run

Every package's record is public on histor.modelmarket.dev: its tools, what it asked for, what it touched, which image it ran in. WARDEN reads the same log. npx -y @aimarket/warden@0.15.0 scan --histor asks HISTOR, for every server in your Claude, Cursor or VS Code config, whether you get the tool set HISTOR recorded, and shows what its package did in the sandbox. For a package, that is the set its sandbox got. A launcher whose program is installed on your machine lists another one; its publisher can now declare that set in the server's registry entry, and HISTOR shows it as the publisher's word.

Try it on a server of yours, in the browser: look up its history in HISTOR, scan its tools/list with WARDEN, or count what its tool list costs per model request. Free, no account. That last one answers a question in the comments below; the numbers for all 34,374 lists are in What an MCP server's tool list costs.

We build HISTOR and WARDEN. The numbers are a snapshot of 13 October 2026; the crawl continues, so the site will have moved on. No package is named here. The method and the observer's code are in alexar76/histor (docs/architecture.md, sandbox/histor_observe.py).

Edited 14 October 2026: the WARDEN sentence above said it checks "whether you get the tool set everyone else gets". For a launcher package that is not one set, as a reader pointed out in the comments; it now says what HISTOR compares with.

Comments

[Auten]

Useful dataset, and the "Testing the tester" section is what makes me trust the rest of it.

One case I'd like to understand. Disclosure: I'm on the Auten team, and our npm package is a thin launcher. HISTOR has it as ok with one tool, install_auten, because the runner it hands stdio to isn't on your sandbox. On a machine where the runner is installed, tools/list returns the real computer-use tools instead. So "the tool set everyone else gets" is honestly two sets, depending on local state.

How does WARDEN treat that? Would someone with the runner installed see their server flagged as a different tool set from HISTOR's pin? And is there a way for a publisher to declare the expected set (in server.json, say), so it can be checked rather than only diffed? There are probably a fair number of launcher-style packages like this in the "needs something the machine lacks" bucket too.

[Alex]

Following up on the launcher case — I recovered HISTOR’s stored sandbox snapshot: one install_auten tool. I’ve also prepared a capture helper with instructions for comparing actual environments.
It runs the supplied launch command and collects initialize metadata and paginated tool lists. The collector makes no tool calls, creates no trust pins and uploads nothing itself. Starting the launcher still executes its code. The saved tools are client-normalized definitions, not a raw wire transcript.
Could you share two captures from the differing environments, ideally using the same pinned @autenai/mcp@0.1.5 launcher? For each, please include:

• OS/architecture and runner release/build, or explicitly “runner absent”.
• The exact launch command, distinguishing the npm launcher from direct auten mcp-bridge.
• Which runner executable was selected; AUTEN_BIN and the standard installation location can take precedence over PATH. The useful files are tools.json, initialize.json, capture.json and SHA256SUMS. Please review them before sharing; no keys, environment values, complete client configs or personal SSE URLs are needed. No need to remove a working runner to manufacture the missing-runner case. One current working capture would already let us make a first comparison with the older sandbox baseline, with the version/time differences stated explicitly. The helper passed ten synthetic Linux tests. We haven’t run your real runner, so those tests don’t establish the second environment’s tool set. A publisher declaration can then be checked separately; its digest/count alone cannot replace the actual snapshots.

[Auten]

Thanks for digging the old snapshot back out, and for spelling out exactly what the collector does and doesn't do.

Honest answer: we won't run captures against the installed runner and post them in this thread for now. That's our call about publishing the full tool definitions, not a concern with your helper. The publisher-declaration route you suggested is the one we're working toward (a hosted tool-sets file), but it isn't live yet, so there's nothing on our side to check today.

Agreed that a digest alone shouldn't stand in for real snapshots.

[Alex]

Understood, thanks for clarifying. We’ll keep the recovered snapshot labelled as a historical sandbox observation; it cannot verify the currently installed runner. A hosted tool-sets file would be publisher-declared information, separate from a runtime capture. There’s nothing further to verify on your side while that file isn’t available.

[Alex]

Thanks, and fair point. For a launcher, "the tool set everyone else gets" really means "the set a machine without your runner gets", and that is what HISTOR holds for your package: the one install tool. We have changed that sentence in the post.

How WARDEN treats it: scan --histor digests the tools/list your client actually got (MTL/1) and asks HISTOR whether it has seen that digest for the package. With the runner installed it has not, so the answer is different and WARDEN shows an advisory finding. It never blocks. Without the runner the answer is same.

Your comment also showed that finding was worded wrong for this case: it named only a version HISTOR had not reached or a package that is not the published one, which reads like tampering. WARDEN 0.15.0 names local state first (a program the package hands over to, a real key) and gives the tool count on both sides.

Declaring the expected set: you can now. Put the MTL/1 digest of the set your runner serves in your server.json, under _meta → io.modelcontextprotocol.registry/publisher-provided → dev.modelmarket.histor/tool-sets, as {"mtl": "1", "sets": [{"digest": "sha256-…", "count": 23, "when": "the runner is installed"}]}, and publish that version to the registry. On a machine with the runner, npx -y @aimarket/warden scan --json prints the digest as mtlDigest. HISTOR reads the declaration at its next daily crawl and answers /check with it, marked as your declaration, not an observation; WARDEN 0.15.0 then reports that set as declared by the publisher, at low severity, instead of unseen. Its own checks still read every tool. Details: github.com/alexar76/histor/blob/ma...

The "needs something the machine lacks" bucket: yes, launchers that exit when their program is missing are among those 236. HISTOR has no set for them, so WARDEN has nothing to compare and flags nothing, and a declaration works for them too.

[Auten]

Thanks for changing the wording and shipping the declaration path so fast. Naming local state first in the WARDEN finding is exactly what makes it readable for a launcher.

We'll look at adding a tool-sets declaration on our next registry publish. One follow-up: our runner is versioned separately from the npm launcher (and per OS), so the set it serves can change while server.json stays the same. Is the intended pattern to list each known digest in "sets" and republish when it changes, or does HISTOR expect one declared set per published package version?

[Alex]

Good question, and it made us change something. A declaration belongs to your server's registry entry, not to an npm version: HISTOR applies it to the server whatever launcher version someone runs, so there is no "one set per package version". You list every digest you know, but you no longer have to republish server.json when the runner changes. Point the entry, once, at a file on your own domain, and HISTOR reads that file every hour.

1. On each OS and runner version: npx -y @aimarket/warden@0.16.0 declare --when "runner 1.4, macOS" --out mcp-tool-sets.json -- npx -y <your package>. It starts the launcher the way a client does, digests what tools/list returns and adds that set to the file (other sets kept, up to 64).
2. Put the file on the domain your registry namespace was verified with, or a subdomain, e.g. https://<your domain>/.well-known/mcp-tool-sets.json. HISTOR reads it from there and nowhere else, so nobody can declare sets in your name.
3. Once: declare --server-json --link <that url> prints the _meta block; publish that server.json version.
4. declare --check <that url> --name <your server name> tells you what HISTOR will take from the file.

With each runner release: step 1 again, upload the file. Keep older runners' digests listed while people still run them; a set you drop is marked withdrawn, and if your site is down HISTOR keeps the last good file instead of withdrawing anything. People who get a declared set see a low-severity finding that says it comes from your file, and WARDEN still checks every tool.

Details: github.com/alexar76/histor/blob/ma...

[Auten]

That answers it, thanks. A file on the namespace domain that HISTOR re-reads hourly fits a runner that versions separately from the launcher much better than republishing server.json each time, and keeping the last good file when the site is down is the right default.

The declare --check step is a nice touch too: being able to see what HISTOR will accept before anyone gets a finding makes this easy to try. We'll test it against our next runner release.

[Marcus Kim]

The container-address reuse bug is a good reason to treat observer attribution as part of the test, not just infrastructure. I'd add a deliberate handoff case: one container leaves a delayed connection, the next receives the same address, and neither run may inherit the other's canary. Pairing that with your incomplete-on-missing-DNS rule would keep a clean verdict distinct from a run the observer couldn't fully account for. That distinction is more useful to an adopter than one overall 'answered' percentage.

[Alex]

Thanks — we added a synthetic observer test for the handoff you described, and it failed. It uses collector records, not live Docker address reuse. A's queued traffic could be attributed to B even with the start-time filter. We now check the canary's run tag: foreign-only records are excluded from B's evidence, and B is marked incomplete (foreign-run-traffic), not clean.

Your reporting point exposed another bug: new incomplete reasons replaced existing ones, so a missing DNS log could disappear. They now accumulate, and any such reason prevents a clean verdict. We wrote up both fixes and the test's limits here: dev.to/alexar76/a-period-bypassed-...

[Paul Irolla]

@alexar76 Right, and that's the number worth taking next: 182,157 definitions over 7,656 servers is about 24 tools each, and every definition is resent with each request. Counting the tokens of one typical manifest would turn that into a cost.

[Alex]

Good call, so we counted. HISTOR keeps the current tool list of every server it observes, so this is all of them rather than one typical manifest: 18,204 packages (plus 16,170 remote servers), each list counted the way a client sends it to a model ({name, description, input_schema} per tool), with the o200k tokenizer.

A median package's list is 1,272 tokens; the mean is 3,812, the 90th percentile 8,361, the 99th 39,612. Remote servers look the same, median 1,328. 1,447 packages are over 10,000 tokens and 31 over 100,000; the largest is 304,780, more than a 200k context window. (The set has grown since the post: today the mean is 18.5 tools and the median 8.)

In money, per 1,000 requests at $2 per million input tokens, a Sonnet-class list price: $2.54 for a median server, $7.62 for the mean, $16.72 for the 90th percentile, paid on every request before the user has said anything. Prompt caching brings it to about a twentieth when the client keeps the list unchanged at the front of the prompt, and loading schemas only when a tool is about to be called avoids most of it.

Method, the full table and the caveats (one tokenizer; providers' own tool-use text not counted; every listed server weighs the same, used or not): github.com/alexar76/histor/blob/ma...

Thanks for the nudge.

[Arsen Apostolov]

running a few mcp servers on my homelab made me think about this a lot. the sandbox test answers "what does it do on first start", but the part that keeps me up at night is the runtime behavior — a server that passes initialize can still hit a network endpoint or read a file it never declared, and only when a tool is actually called. i ended up writing a small monitor that tracks which local services are alive, what they connect to, and how much vram they burn, so at least i can see the drift between what a tool says it does and what it does. github.com/SikamikanikoBG/homelab-monitor — not mcp-specific, but the "who is actually running what on my box" question is the same one. the "testing the tester" section is the right instinct. a log that lies is worse than no log at all.

[Alex]

The crawl records the first start. Inside the sandbox the observer speaks initialize and tools/list. That is the start, not what a tool does on a later call.

A sandbox that reports wrong things is worse than none.

[Richard Smith]

The placeholder key finding is wild - 1,039 servers told you their tools but didn't need a key. Makes you wonder how many are doing that on purpose vs cargo-culting the check.

[Alex]

1,039 started with a placeholder key. A key that works nowhere was enough for every one of them to tell us its tools. None needed a real key to do that; they just refused to start without one.

The run counts those servers. It cannot see whether that was on purpose or copied by habit.

[Paul Irolla]

The 41% cost startup time, not tokens: a server that never answers adds no tool list. The bill is the 59% that did, since every tool list they return rides along in each request.

[Alex]

5,259 of 12,915 never answered, so they returned no tool list. The 7,656 that answered did return one. Between them they listed 182,157 tool definitions. This run does not measure token cost.

[Kartik N V J K]

Seeing 292 PyPI servers die on import after mcp 2.0.0 is the result I'd send every maintainer, since 207 of the 269 you retried came back with mcp<2. I also appreciated the breakdown of the 295 decoy file opens, because the ~/.netrc reads from requests would have alarmed me too. As the crawl continues, will you track how many of those 292 recover once maintainers add an upper bound on mcp?

[Alex]

Following up with the fresh-install count you asked for: we rechecked the reconstructed 292-package cohort on 24 October, without adding our own mcp<2 constraint.
Fourteen answered initialize and tools/list: twelve had newer releases, while two worked at the same version after we installed their declared MCP extra. Eleven of the twelve newer-release successes have a matching fresh old-version control that still fails; the remaining old release is no longer available from PyPI.
Seven confirmed pairs add an upper bound, three move to SDK 2, and one changes an already bounded range with a Pydantic-related old failure. Another 276 still fail on imports, APIs or dependencies; two need real authentication or a prepared project index that our synthetic environment did not provide.
I also need to correct my earlier reply: not all 292 were proven MCP 2 failures. Our old classifier included missing extras and MCP 1/Pydantic errors because it matched the FastMCP path anywhere in a traceback.
The follow-up explains the controls and limitations: dev.to/alexar76/we-rechecked-292-f...
Success here means answering initialize and listing tools; safety and authenticated functionality were not established.

[Alex]

The 292, the 207 of 269, and the 295 are the 13 October snapshot. The 292 are PyPI servers that depend on mcp with no upper bound, install 2.x, and die on import. We started 269 of them again with pip held to mcp<2, and 207 answered. 295 opened a decoy credential file.

The crawl continues. Each pass installs the latest version and records whether that run answered initialize and tools/list, and what it printed as it exited. That exit text is what placed a server in the 292. A later pass records the new run. The snapshot does not carry a separate count of how many of those 292 recover once a maintainer adds an upper bound.