Over the last couple weeks I built three small, independent CLIs that each check a different way an MCP (Model Context Protocol) server can be broken — not by reading marketing copy, but by pointing them at real, popular servers and reading what came back.
The three tools ask three different questions:
- mcp-doctor — is it documented? Static analysis of the source: missing tool descriptions, undocumented parameters, hardcoded secrets, no error handling, no tests.
- mcp-fuzz — does it fail safely? Actually launches the server over stdio and calls every read-only tool with schema-derived bad input (missing required fields, wrong types) to see if it returns a structured error or just crashes.
- mcp-reality-check — does it actually work? Calls tools with realistic input and checks whether a "successful" response is actually honest: not a disguised refusal, not empty content, not ignoring its own declared output schema.
None of them use an LLM to judge anything. All three are fully static/heuristic by design — deterministic, no API key, no per-call cost.
The pattern I didn't expect: each tool found a real bug in itself
I dogfooded all three against real servers as I built them, and the same thing happened three times: the first serious dogfood pass against a real-world server found a genuine bug in my own tool's logic, not the target.
- mcp-doctor, against homeassistant-ai/ha-mcp (4.5k stars): the secret scanner was flagging test fixtures and identifier-style constants as hardcoded credentials, and tool detection was counting mock functions inside test files as real tools. Both wrong — the server's actual score corrected from a false 55%/F to an accurate 89%/B. That fix led to a PR merged upstream: https://github.com/homeassistant-ai/ha-mcp/pull/2327
- mcp-fuzz, against mendableai/firecrawl-mcp-server: a server that validates input with zod and correctly returns a well-formed JSON-RPC -32602 INVALID_PARAMS error was being classified identically to an actual crash, because of a blanket except Exception. Root-caused the real distinction (-32602 = server validated and rejected properly; -32603 and everything else = still counts as broken) and fixed it — but not before nearly flipping an already-published, already-cited finding on a different repo to a false pass. Caught that by re-testing the cited repo before shipping, not after.
- mcp-reality-check, against modelcontextprotocol/server-time: no timezone hint existed in the realistic-input generator, so every timezone-shaped property got a bogus placeholder string and every call failed. Fixed by adding a real IANA timezone name as the hint.
I don't think this is a coincidence. It's what happens when you build a tool whose entire job is judging correctness, then finally point it at something popular enough to have edge cases you didn't think of. If your own tool never finds a bug in itself the first time it meets the real world, you probably haven't tested it against anything hard enough yet.
The strongest finding so far
Running mcp-fuzz against antvis/mcp-server-chart (4.3k stars, official Ant Design MCP server, 27 chart-generation tools) found that all 27 tools crash — raw JSON-RPC -32603 internal errors, not structured tool-level errors — on missing-required or wrong-type input. 133 of 214 test calls failed this way. Filed as issue #323: https://github.com/antvis/mcp-server-chart/issues/323
Traced it further: the fix already exists on main (an unreleased commit, 9fd0bb4), just never published to npm. Built main locally, replayed all four repro cases from the issue directly over stdio, confirmed the fix actually resolves what mcp-fuzz flags. Posted that as a comment instead of a "please fix this" ask, since there was nothing left to fix — just an unpublished release. The maintainer has since asked for the exact repro input, which I posted as literal JSON-RPC payloads.
Where things stand
- mcp-doctor: 33+ real-world dogfood passes, one PR merged upstream, a live public leaderboard scoring 17 real MCP servers on quality and security: https://vishalhabib99.github.io/mcp-doctor/
- mcp-fuzz: ~15 passes, the antvis finding above.
- mcp-reality-check: 13 passes, the newest of the three, still catching up on track record.
None of these have real traction yet — this is a few weeks old, built solo, with basically zero stars or followers behind it. I'm writing this up because the process itself — build a tool, immediately distrust it, verify it against something real before believing its output — is the part I think is actually worth sharing, independent of whether the tools themselves ever get popular.
Repos: mcp-doctor · mcp-fuzz · mcp-reality-check
Top comments (1)
A tool finding its own bug is a better advertisement for characterization tests than any benchmark. For agent infrastructure, I would add one adversarial pass where the auditor receives malformed tool metadata, oversized payloads, and conflicting schemas. The interesting failures often live in the contract between components, not in the component being tested.