DEV Community

Scott Jones
Scott Jones

Posted on

What broke when I tested MCP servers in four AI clients

#mcp #ai #claude #cursor

For the past week I've been running MCP servers through the same small tasks in Claude Code and Cursor, and some in Claude Desktop and ChatGPT (Codex). Each run checks install, connect and tool listing, then gives the agent one standard task and records pass, partial or fail. Here's what I learned.

1. "It's on the registry" doesn't mean it starts

Chroma MCP and the archived reference SQLite server both failed to start in Claude Code and Cursor on Oct 15. Their published uvx commands crash on current dependencies. Paper Search MCP and MCP Python Interpreter also failed to start in both clients. mcpdoc didn't start in either client.

2. Client config matters as much as the server

  • The Filesystem reference server passed in Claude Code only after adding the folder to permissions.additionalDirectories.
  • Open-WebSearch passed live search and page reading in ChatGPT (Codex), but Codex needed a 60-second server start timeout on a cold npx start.
  • Jupyter MCP passed in Claude Code and Cursor but failed in Codex. It also needs a running Jupyter server with a token.

3. Ties are common; tool-list size breaks them

Playwright MCP and Chrome DevTools MCP both clicked "Reveal code" on a test page and read the code back in 4 calls each. I gave Playwright the nod for its smaller tool list (25 vs 30). Same story for Terraform: OpenTofu, HashiCorp's Terraform MCP and an archived Terraform Registry server all returned the latest hashicorp/aws provider version in 2 calls each across both clients.

4. Keyless servers are easier to trust

Open-WebSearch was the only search server to pass both a live search and a page read in Claude Code and Cursor with no API key. Keyed servers (Brave, Exa, Firecrawl) I haven't tested yet.

Limits

One run per client, small fixtures, and a lot of servers still untested. Treat all of this as provisional. If you maintain one of these servers and something has changed, I'll retest.

I run ToolPassed, where these results are published per client with dates, including fails and "Not tested yet": https://toolpassed.com/?utm_source=devto&utm_medium=community&utm_campaign=launch

Top comments (1)

Collapse
 
autenai profile image
Auten •

Useful that you publish the fails and the "not tested yet" too, most comparisons quietly drop those.

From the Auten team (we ship a computer-use MCP server, so bias noted). If you ever add a screen-control server to the set, two things I'd score separately from "it starts":

  • The OS permission step. On macOS the server can start, list its tools and still do nothing until the terminal or client app gets Accessibility and Screen Recording, much like your additionalDirectories case for Filesystem. Worth recording as "passed after grant" rather than fail.
  • A task that proves the click landed, not that the tool returned OK: type a value into a native app field, then read it back from the screen.

Did the 60-second cold npx timeout in Codex also hit the npx-based servers in Cursor, or only Codex?