DEV Community

Scott Jones
Scott Jones

Posted on

What broke when I tested MCP servers in four AI clients

For the past week I've been running MCP servers through the same small tasks in Claude Code and Cursor, and some in Claude Desktop and ChatGPT (Codex). Each run checks install, connect and tool listing, then gives the agent one standard task and records pass, partial or fail. Here's what I learned.

1. "It's on the registry" doesn't mean it starts

Chroma MCP and the archived reference SQLite server both failed to start in Claude Code and Cursor on Oct 15. Their published uvx commands crash on current dependencies. Paper Search MCP and MCP Python Interpreter also failed to start in both clients. mcpdoc didn't start in either client.

2. Client config matters as much as the server

  • The Filesystem reference server passed in Claude Code only after adding the folder to permissions.additionalDirectories.
  • Open-WebSearch passed live search and page reading in ChatGPT (Codex), but Codex needed a 60-second server start timeout on a cold npx start.
  • Jupyter MCP passed in Claude Code and Cursor but failed in Codex. It also needs a running Jupyter server with a token.

3. Ties are common; tool-list size breaks them

Playwright MCP and Chrome DevTools MCP both clicked "Reveal code" on a test page and read the code back in 4 calls each. I gave Playwright the nod for its smaller tool list (25 vs 30). Same story for Terraform: OpenTofu, HashiCorp's Terraform MCP and an archived Terraform Registry server all returned the latest hashicorp/aws provider version in 2 calls each across both clients.

4. Keyless servers are easier to trust

Open-WebSearch was the only search server to pass both a live search and a page read in Claude Code and Cursor with no API key. Keyed servers (Brave, Exa, Firecrawl) I haven't tested yet.

Limits

One run per client, small fixtures, and a lot of servers still untested. Treat all of this as provisional. If you maintain one of these servers and something has changed, I'll retest.


I run ToolPassed, where these results are published per client with dates, including fails and "Not tested yet": https://toolpassed.com/?utm_source=devto&utm_medium=community&utm_campaign=launch

Top comments (3)

Collapse
 
autenai profile image
Auten •

Useful that you publish the fails and the "not tested yet" too, most comparisons quietly drop those.

From the Auten team (we ship a computer-use MCP server, so bias noted). If you ever add a screen-control server to the set, two things I'd score separately from "it starts":

  • The OS permission step. On macOS the server can start, list its tools and still do nothing until the terminal or client app gets Accessibility and Screen Recording, much like your additionalDirectories case for Filesystem. Worth recording as "passed after grant" rather than fail.
  • A task that proves the click landed, not that the tool returned OK: type a value into a native app field, then read it back from the screen.

Did the 60-second cold npx timeout in Codex also hit the npx-based servers in Cursor, or only Codex?

Collapse
 
scott_jones_e1104a4b87dc7 profile image
Scott Jones •

Only Codex. Open-WebSearch started with the same npx config in Cursor (Cursor CLI 2026.10.12) and Claude Code on Oct 13 and passed without any timeout change. Codex was the only client where we had to raise the server start timeout to 60s for a cold npx start. We didn't record connect times for those first runs, so I can't say how close Cursor came. New runs record connect time now. Good points on screen-control servers. If we add them, a permission grant would be recorded as a setup step next to the result, the same way we handled Filesystem's additionalDirectories, and the task would have to prove the click landed, not just that the tool returned OK.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Tokenade is what I build on this layer. Every client loads its own copy of the tool list, and that list goes back out with each request, so it's the part I measure.