DEV Community

Cover image for I Injected the Same Failures Into 5 MCP SDKs. Here’s What I Found.
L Anil Kumar Singha
L Anil Kumar Singha

Posted on

I Injected the Same Failures Into 5 MCP SDKs. Here’s What I Found.

#mcp #opensource #programming #testing

I've been building MCP Failure Lab, a deterministic failure-injection toolkit for testing Model Context Protocol clients and servers beyond the happy path.

Instead of asking whether an MCP client can successfully call a tool, I wanted to ask a different set of questions:

  • What happens when the connection disappears?
  • What happens when a response arrives twice?
  • What happens when the server sends malformed JSON-RPC?
  • Does the client recover?
  • Is the same behavior observable across SDKs?

Over the last few releases, I've been running the same failures against multiple official MCP SDKs.

Some of the differences turned out to be more interesting than I expected.

The test setup

The important part of these tests is determinism.

MCP Failure Lab deliberately produces a specific failure so that the same scenario can be replayed against different clients.

For the latest duplicate-response tests, I used:

mcp-failure-lab@0.10.0
Enter fullscreen mode Exit fullscreen mode

with both:

  • stdio
  • Streamable HTTP

The clients tested were:

Client Version
TypeScript SDK 1.30.0
Python SDK 2.2.0
Go SDK 1.7.0
Rust SDK (rmcp) 3.4.0
C# SDK 2.2.0

The runtime environment was macOS arm64 with Node.js 26.5.0, Python 3.12.14, Go 1.27.1, Rust 1.98.1 and .NET SDK 10.0.401.

The goal wasn't simply to see whether a request failed.

I also wanted to know whether the same session remained usable afterward.

Finding #1: HTTP disconnect recovery differed between TypeScript and Python

One of the earlier tests injected an HTTP disconnect and then attempted another request.

The TypeScript SDK recovered and the next request succeeded.

With Python SDK 2.2.0, the connection closed and the following request failed in the scenario I tested.

That was interesting because the injected failure was identical, but the observable recovery behavior wasn't.

I reported the reproduction upstream:

https://github.com/modelcontextprotocol/python-sdk/issues/3522

This was one of the first results that convinced me that failure testing across implementations was worth pursuing.

Happy-path interoperability doesn't necessarily imply recovery-path interoperability.

Finding #2: rmcp accepted a malformed JSON-RPC response

The next finding came from the Rust SDK.

MCP Failure Lab's malformed_message tool can intentionally return a JSON-RPC response containing both result and error.

Conceptually:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [],
    "resultType": "complete"
  },
  "error": {
    "code": -32603,
    "message": "injected error"
  }
}
Enter fullscreen mode Exit fullscreen mode

A JSON-RPC response should contain either result or error, not both.

I expected the client to reject it.

With rmcp 3.4.0, however, call_tool returned:

Ok(CallToolResult)
Enter fullscreen mode Exit fullscreen mode

The result was accepted and the error field was effectively ignored.

I reproduced this over both stdio and Streamable HTTP.

A subsequent normal request also succeeded, so this wasn't a connection-recovery problem. It was a message-validation difference.

The C# MCP SDK 2.2.0 rejected the same malformed wire response.

The likely Rust parsing path involves the untagged JsonRpcMessage representation: the response can deserialize into the JsonRpcResponse shape while Serde ignores the unexpected error field.

I reported that upstream as well:

https://github.com/modelcontextprotocol/rust-sdk/issues/1283

This is exactly the kind of issue that's difficult to notice when every server you're testing against sends valid messages.

Finding #3: Five SDKs survived a duplicate response, but exposed it differently

For MCP Failure Lab 0.10.0, I added a focused duplicate-response compatibility test.

The sequence was deliberately simple:

client
  |
  | tools/call
  v
server
  |
  | response #1
  | response #1 again
  v
client
  |
  | ping using the SAME session
  v
server
Enter fullscreen mode Exit fullscreen mode

A passing recovery check required the final ping to succeed.

Here were the results:

Client stdio Streamable HTTP Duplicate behavior Same-session ping
TypeScript Pass Pass Unknown-response-ID diagnostic Pass
Python Pass Pass No call-level duplicate error observed Pass
Go Pass Pass No call-level duplicate error observed Pass
Rust Pass Pass No call-level duplicate error observed Pass
C# Pass Pass No call-level duplicate error observed Pass

The good news is straightforward:

all five clients remained usable.

None of them allowed the duplicate response to corrupt the session.

But their observable behavior wasn't identical.

TypeScript made the duplicate visible

The TypeScript SDK accepted the first response normally.

When the duplicate arrived, that request ID was no longer pending, so the SDK reported the repeated response through its error callback as a response for an unknown ID.

The diagnostic did not close the session.

The next ping succeeded.

The other four didn't expose a call-level error

Python, Go, Rust and C# also returned the first result and successfully completed the following ping.

But my harness didn't observe an equivalent call-level duplicate diagnostic from those clients.

That does not mean they failed to detect the duplicate.

There are several possible implementations:

response arrives
       |
lookup pending request ID
       |
       +-- found --> resolve request
       |
       +-- missing --> report?
                      log?
                      discard?
Enter fullscreen mode Exit fullscreen mode

The current experiment only establishes what was observable through the public paths instrumented by the harness.

I did not instrument private SDK internals, so I can't claim that those four SDKs silently ignored the response internally.

That's a distinction worth preserving.

A "pass" isn't always the whole result

This has been one of the more useful lessons from these experiments.

If I reduced the duplicate-response test to:

TypeScript: PASS
Python:     PASS
Go:         PASS
Rust:       PASS
C#:         PASS
Enter fullscreen mode Exit fullscreen mode

I'd lose most of the interesting information.

All five passed the recovery invariant.

But one implementation exposed an unexpected response to the application while the others didn't expose an equivalent call-level diagnostic in my harness.

That matters when you're debugging a production MCP system.

Two clients can both recover successfully while giving developers very different visibility into what happened.

Why test the request after the failure?

I've started treating this as one of the most important parts of the test.

Triggering a fault only tells you what happened during the fault.

It doesn't tell you whether the protocol state survived it.

So a typical Failure Lab scenario now looks conceptually like:

normal operation
      ↓
inject deterministic failure
      ↓
observe client behavior
      ↓
send normal request
      ↓
verify recovery
Enter fullscreen mode Exit fullscreen mode

A timeout that returns an error but leaves the session healthy is very different from a timeout that poisons every request afterward.

Likewise, rejecting malformed JSON-RPC is different from accepting it while keeping the connection alive.

The post-failure request gives us that second dimension.

These aren't necessarily SDK bugs

Another thing I've tried to avoid is turning every behavioral difference into an upstream issue.

Different observable behavior isn't automatically incorrect behavior.

For example, the duplicate-response test currently shows an interoperability difference in diagnostics, but all five clients recover.

Before calling that a bug, the next step is to trace how each SDK handles a response whose request ID has already been completed:

duplicate response
       ↓
request ID no longer pending
       ↓
what does the SDK do?
       ↓
report / log / discard / reject
Enter fullscreen mode Exit fullscreen mode

If an implementation violates a protocol requirement, that's worth reporting.

If it's simply an intentional observability choice, it's better documented as a compatibility difference.

The malformed Rust response was different: there was a concrete validation behavior to reproduce and compare, so I opened an upstream issue.

What I'm testing next

The next step isn't simply adding more SDK names to a table.

I'm more interested in expanding the failure dimensions.

Some of the scenarios I'm looking at include:

  • late responses after client timeout
  • session loss
  • server restart and recovery
  • duplicate SSE delivery
  • malformed or partial transport messages
  • cancellation races
  • responses with unexpected IDs
  • transient HTTP failures followed by recovery

Then the same failure can be replayed against multiple implementations.

That should eventually produce something more useful than a basic compatibility matrix:

a map of how MCP implementations behave when the protocol stops being perfect.

Reproducing the tests

MCP Failure Lab is open source:

https://github.com/anilloutombam/mcp-failure-lab

The published package is available through npm:

npx -y mcp-failure-lab@0.10.0 serve
Enter fullscreen mode Exit fullscreen mode

The focused 0.10.0 duplicate-response compatibility report, including exact SDK and runtime versions, is here:

https://github.com/anilloutombam/mcp-failure-lab/blob/main/docs/compatibility/v0.10.0.md

The project documentation is also available at:

https://mcplab.dev/

If you're maintaining an MCP client, SDK or host and there is a failure mode you think would be useful to reproduce deterministically, I'd be interested in testing it.

The happy path tells us whether implementations can talk to each other.

I'm more interested in what happens after something goes wrong.

Top comments (4)

Collapse
 
raju_dandigam profile image
Raju Dandigam •

The two-axis result—protocol conformance and session survivability—is much more useful than one PASS. I’d add a third field for diagnostic visibility, because “recovered with an unknown-ID signal” and “recovered with no public signal” create very different production debugging experiences. Are you normalizing these runs into a shared event schema so the same injected fault can be diffed across SDKs and transports?

Collapse
 
anilloutombam profile image
L Anil Kumar Singha •

Agreed on diagnostic visibility. I saw exactly that all five recovered, but TypeScript surfaced the unknown-ID signal while the others didn’t expose one in the harness.

I’m not normalizing runs into a shared event schema yet. That’s a logical next step for clean cross-SDK/transport diffs.

Collapse
 
mihai_leanzero profile image
Mihai Perdum •

The duplicate-response finding and the rmcp finding read like two different categories to me, worth separating explicitly. Duplicate response is genuinely just an observability difference — all five recovered, TypeScript just happened to surface it through an error callback while the others didn't visibly. But rmcp accepting a response with both result and error set isn't an interop quirk, it's accepting a message the JSON-RPC 2.0 spec says shouldn't exist (a response has either result or error, never both) instead of rejecting it. One is "we disagree on what to expose to the caller," the other is "we accepted an invalid wire message." Does Failure Lab distinguish a spec-conformance tier from an interop/observability tier, or is that a split you're planning to add? Lumping them together would understate how serious the rmcp case actually is.

Collapse
 
anilloutombam profile image
L Anil Kumar Singha •

Yep, agreed.

Duplicate response is an interop/observability difference since all clients recovered. The rmcp result + error case is a real protocol-conformance bug; the upstream issue is now marked P1/spec violation.

I’m going to separate Failure Lab reporting into:

  1. protocol/spec conformance
  2. interop/resilience/observability