What Happens When an MCP Tool Fails Halfway? Turning Failures into Regression Tests

I maintain ResiliReplay, and I built it around a question that kept surviving ordinary MCP smoke tests: what happens after the server has been discovered and a tool fails at the boundary between “request accepted” and “useful result returned”?

Most first-pass checks are necessarily optimistic. Start the server, call tools/list, invoke one tool with valid arguments, and confirm a clean response. That proves the integration can work. It does not prove that a client recovers within a budget, avoids duplicate side effects, rejects hostile output, or preserves the same behavior after a later code change.

Reliability work begins where that clean call stops.

Inspector and ResiliReplay answer different questions

MCP Inspector is useful for interacting with a server: discover tools, inspect schemas, supply arguments, and see actual responses. ResiliReplay accepts an Inspector-shaped configuration, but it is not a replacement for that interactive workflow.

The split I use is simple:

• Inspector answers, “What does this server expose, and does a normal call work?”
• ResiliReplay asks, “What happens when a declared failure is inserted, does recovery stay inside its budget, and can I replay the result later?”

That second question needs more than random fault injection. A useful campaign must be bounded and reviewable: one seed, explicit scenarios, a small tool allowlist, fixed retry and time budgets, declared expectations, and local evidence. If a campaign will call a tool, ResiliReplay prints a canonical review plan and requires the exact campaign hash back before execution.

Start without contacting the server

With Node.js 22 or 24, the minimal entry point is a dry run:

npx --yes resilireplay@0.3.1 mcp audit \
  --inspector-config ./mcp.json \
  --server my-server \
  --dry-run

The output describes the executable, argument list, working directory, transport, timeouts, and environment-variable names. It does not start the server or call a tool. That makes it a useful place to catch an incorrect target or an unexpectedly broad configuration.

Then generate a campaign template:

npx --yes resilireplay@0.3.1 campaign init reliability.campaign.yml
npx --yes resilireplay@0.3.1 campaign validate reliability.campaign.yml

For an early field test, I keep concurrency at one, allow exactly one read-only idempotent tool, set retries to one, and use three scenarios.

1. Establish a clean control

The control uses fault: none and expects a pass with zero retries. This is not ceremonial. Without a clean control, an injected failure can be confused with a broken install, an invalid working directory, a missing browser binary, or an incompatible server version.

One of my Playwright MCP setup runs demonstrated the point: recovery failed because Chrome for Testing was not installed. That was an environment failure, not evidence about Playwright MCP recovery. After using the documented installer, the clean control and the bounded campaign could be interpreted honestly.

2. Inject a result-level failure and retry once

The next scenario uses mcp-tool-error with recovery: retry. ResiliReplay allows the real tool call to complete, replaces the observed result at the test boundary with a controlled error, and permits one retry. The assertions require successful recovery, no more than one retry, no duplicate side-effect attempt, and policy compliance.

That distinction matters. The server did not contain the injected error; the harness introduced it. A pass means only that the observed behavior matched this declared experiment.

3. Add a malicious-canary negative control

The third scenario injects mcp-malicious-canary-instruction and expects failure. Its purpose is to show that the campaign can detect a known bad condition instead of reporting green regardless of input.

This is a negative control, not a vulnerability claim. The canary is synthetic, no real secret is used, and the expected outcome is recorded before the run. When the expected failure occurs, ResiliReplay reduces the causal trace and writes a scenario, minimized fixture, executable Node test, and manifest. The campaign runner executes that generated regression immediately.

Baselines should fail closed

After a complete expectation-matching run, approve it explicitly:

npx --yes resilireplay@0.3.1 campaign approve runs/field-test \
  --output baselines/field-test.json

npx --yes resilireplay@0.3.1 campaign compare runs/field-test \
  --baseline baselines/field-test.json \
  --output runs/comparison

The comparison checks verified run identity and declared metrics such as score drop, retries, and duplicate attempts. Incomplete or hash-invalid evidence is not silently treated as a pass. This is the part that turns a one-off experiment into a CI boundary.

Three bounded field validations

I ran the public resilireplay@0.3.0 package against three pinned, independently maintained projects over their documented local stdio paths:

• MCP Everything Server: echo with harmless generated text.
• Playwright MCP: browser_snapshot on a blank isolated headless page, with no navigation.
• UI5 MCP Server: get_guidelines against bundled guidance, with no project analysis.

Each campaign ran the clean control, one injected tool-result error with one successful retry, and the canary negative control. Each negative control produced a verified executable regression. Each approved baseline comparison reported zero differences. The complete commands, versions, boundaries, and sanitized evidence are in the field results.

These are three narrow case studies, not universal benchmarks. They cover one reviewed operation per server. They do not rank the projects, imply upstream adoption, or certify security. Streamable HTTP is exercised by ResiliReplay’s local automated fixtures, but none of the three selected external packages used it as the documented local primary path, so I do not present them as third-party HTTP evidence.

Safety boundaries and limitations

ResiliReplay is defensive reliability software, not an OS sandbox. A tool you authorize can still mutate files, databases, browsers, or remote systems with the permissions of its server. Start locally; prefer read-only, idempotent calls; review every allowlisted tool; and never use production data merely to provoke a failure.

Reports are sanitized, but pattern redaction cannot prove that every application-specific secret is gone. Hashes link evidence and detect changes; they do not establish signer identity. The current scorer evaluates declared observable evidence, not open-ended semantic quality. Side-effecting campaigns are intentionally not resumed after interruption because replaying a partial operation may be unsafe.

The project, npm package, and five-minute field-test guide are available from the ResiliReplay site. If you maintain an MCP server, I would value a sanitized result from one real, reviewed tool: which boundary exposed the most useful behavior—transport failure, result-level failure, duplicate invocation, or recovery after partial completion?