What I learned introspecting 922 npm MCP servers

TL;DR: We ran npx -y <package> against 922 npm-published MCP servers, sent them the JSON-RPC initialize and tools/list calls, and captured what they did. 359 responded. 563 failed in 15 distinct ways that say more about npm packaging than about MCP itself.

The stderr signature that broke 261 servers

[stderr] connecting to upstream...
[stderr] (no further output)
[timeout after 120s]

That is the single most common failure mode in the npm MCP ecosystem right now. The process spawns, the package loads, the constructor runs, and then the server tries to phone home to its upstream API before answering the protocol. The introspection runner waits 120 seconds and gives up. Two hundred sixty-one servers - 28% of everything published - never made it past their own startup.

If you maintain an MCP server and you connect to anything external during initialize, you are in this bucket. The fix is to defer the upstream connection until the first tool call.

What we actually ran

The protocol gives you a discovery method: tools/list. Send it after initialize and the server returns the full JSON Schema for every tool it exposes. No README scraping, no LLM interpretation, no guessing. It is the exact same thing your MCP client does when it connects.

The runner does this per package:

1. spawn:  npx -y <package> (over stdio)
2. write:  {"jsonrpc":"2.0","id":1,"method":"initialize",
            "params":{"protocolVersion":"2024-11-05",
                      "capabilities":{},
                      "clientInfo":{"name":"introspector","version":"1.0"}}}
3. read:   initialize response
4. write:  {"jsonrpc":"2.0","method":"notifications/initialized"}
5. write:  {"jsonrpc":"2.0","id":2,"method":"tools/list"}
6. read:   tool list
7. kill:   close stdin, SIGTERM

Concurrency was 8 in parallel, 120-second timeout per server, total wall time around 25 minutes for 922 packages. The output is one JSONL row per server with status, tool count, and (where available) the full input schema for every tool.

The 15 failure buckets

Every server that didn't return a clean tool array was classified by inspecting the exit code, stderr, and the JSON-RPC error if there was one. The breakdown:

Status	Count	Meaning
ok	359	clean tools/list response
init_timeout	261	spawned, never answered initialize
npm_install_generic	172	npx -y itself failed
needs_cli_args	54	exited with usage error
needs_env_var	42	generic missing env var
broken_install	11	malformed package, bad main or bin
error	8	unclassified crash with non-zero exit
needs_setup_step	3	required a CLI setup wizard run first
needs_slack_token	2	refused without SLACK_BOT_TOKEN
needs_azure_creds	2	refused without Azure auth
tools_list_timeout	2	answered initialize, hung on tools/list
needs_google_creds	2	refused without Google auth
needs_stripe_key	1	refused without STRIPE_API_KEY
needs_config_file	1	refused without a config path
needs_external_runtime	1	shelled out to a binary that was not installed
needs_openai_key	1	refused without OPENAI_API_KEY

The credential-wall buckets (everything needs_*) add up to 109 servers, almost 12% of the published set. If you are populating an agent's tool list by parsing READMEs, those servers register as 0-tool servers in your index. They are not 0-tool servers. They are 5-tool, 12-tool, 40-tool servers waiting for a key you didn't supply.

Windows-specific gotchas

The runner is on Windows. Two things bite hard.

Spawning npx. npx on Windows resolves to npx.cmd, a batch script. Node's child_process.spawn without a shell will not invoke a .cmd. The result is a flat ENOENT even though where npx shows it on PATH. The fix:

const child = spawn("cmd", ["/c", "npx", "-y", pkg], {
  stdio: ["pipe", "pipe", "pipe"],
});

You see this same pattern in every claude_desktop_config.json on Windows. If you are writing your own MCP client, you need it too.

UTF-8 stdio. The default code page on Windows is not UTF-8. If an MCP server writes a tool description that contains a non-ASCII character (German umlaut, em-dash, curly quote, anything), and you read its stdio without forcing UTF-8 decoding, you get a JSON parse error halfway through tools/list. Force it:

proc = subprocess.Popen(
    cmd,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    encoding="utf-8",
    errors="replace",
)

Three servers in our run had tool descriptions with non-ASCII characters. Without the encoding flag, all three would have been miscounted as error.

What this says to MCP server authors

A few patterns from the data, addressed to anyone shipping a server:

1. Don't connect upstream during initialize. If your server contacts an API on startup, it will fail introspection and it will fail any client that wants to enumerate tools without first burning a credential. Lazy-connect on the first tool call.

2. Don't require CLI args to print your tool list. 54 servers exit with a usage error before they will even tell you what they expose. If you need a config path, take it from an env var or accept tools/list without it.

3. Document the env vars in the README. The classifier successfully named the credential for the servers that wrote a clear "missing X" line to stderr. The ones that didn't ended up in the generic needs_env_var bucket. That bucket is your bug report queue.

4. Ship a real bin entry. 11 servers in broken_install had package.json that pointed at a file that doesn't exist, or a main field that crashed on require. None of them needed credentials. They needed a publish-time smoke test.

The full dataset

The 9,922 tool schemas and the 922 server status rows are on HuggingFace as automatelab/mcp-servers-tool-catalog under CC-BY-4.0. Pipeline source is at AutomateLab-tech/mcp-tool-catalog, and it re-runs on the 1st of every month via GitHub Actions.
Product page: https://automatelab.tech/products/datasets/mcp-tool-catalog/

from datasets import load_dataset

servers = load_dataset(
    "automatelab/mcp-servers-tool-catalog", "servers", split="train"
)

# Just the unreachable ones, grouped by failure mode
from collections import Counter
print(Counter(r["status"] for r in servers if r["status"] != "ok"))

If your server is in a failure bucket and shouldn't be, open a PR on the pipeline repo. The next monthly run picks it up.