DEV Community

kero168
kero168

Posted on

Your MCP server passes unit tests — and still breaks in Claude. Here's why.

#mcp #ai #testing #opensource

If you've ever written an MCP (Model Context Protocol) server, you've probably lived this: every unit test is green, but the moment you connect the server to Claude, ChatGPT, or Cursor — tools don't show up, calls fail, the connection dies.

The usual suspects:

  • stdout pollution — on stdio transport, stdout is protocol-only. One stray console.log corrupts the JSON-RPC stream.
  • Malformed tool schemas — inputSchema isn't object-typed, required is missing.
  • Undeclared capabilities — you implemented tools/resources/prompts but never declared them.
  • Ungraceful failures — a missing required argument crashes the server instead of returning a proper error.

None of these are catchable by language-level unit tests, because what's broken isn't your logic — it's the protocol boundary. There are 20,000+ public MCP servers, and until now no standard way to test that boundary.

mcp-testbench: test the protocol, not the language

mcp-testbench connects to your server as a real MCP client and verifies the protocol itself. Your server can be written in TypeScript, Python, Go, Rust — anything.

# Zero install — point it at any stdio MCP server
npx mcp-testbench run --server "npx -y @modelcontextprotocol/server-everything"
Enter fullscreen mode Exit fullscreen mode

That single command runs 10 built-in conformance checks (C000–C040): handshake completion, name/version exposure, capability declarations, tools/list validity, per-tool descriptions and object-typed JSON Schemas, graceful rejection of missing required args, and resources/prompts listing.

Your own cases are declarative YAML 

server:
  command: "node dist/index.js"     # or url: "http://localhost:3000/mcp"

suites:
  - name: search tool
    tool: search
    cases:
      - args: { query: "hello" }
        expect: { ok: true, result.contains: "hello" }
      - args: {}
        expect: { error: true }     # missing required arg must fail cleanly
Enter fullscreen mode Exit fullscreen mode

Supported expectations: ok, error, result.contains, result.matches (regex), result.equals, maxDurationMs. No test code — just YAML.

Four lines to put it in CI 

- uses: kero168/mcp-testbench@main
  with:
    server: "node dist/index.js"
Enter fullscreen mode Exit fullscreen mode

Failures show up as GitHub annotations on the PR, and a non-zero exit code fails the job. There's a json reporter for other CI systems.

What's next

The roadmap includes mcp-testbench audit (tool-description linting + declared-permissions vs. actual-behavior probes), snapshot testing, coverage reports, and a JUnit reporter. There are good first issues if you want in — the dev setup takes under 10 minutes.

Point it at your server and tell me how it breaks: https://github.com/kero168/mcp-testbench (MIT, npm)

Top comments (0)