What Happens When an MCP Tool Fails Halfway? Turning Failures into Regression Tests

Ali — Wed, 05 Aug 2026 00:54:26 +0000

I maintain ResiliReplay, and I built it around a question that kept surviving ordinary MCP smoke tests: what happens after the server has been discovered and a tool fails at the boundary between “request accepted” and “useful result returned”?

Most first-pass checks are necessarily optimistic. Start the server, call tools/list, invoke one tool with valid arguments, and confirm a clean response. That proves the integration can work. It does not prove that a client recovers within a budget, avoids duplicate side effects, rejects hostile output, or preserves the same behavior after a later code change.

Reliability work begins where that clean call stops.

Inspector and ResiliReplay answer different questions

MCP Inspector is useful for interacting with a server: discover tools, inspect schemas, supply arguments, and see actual responses. ResiliReplay accepts an Inspector-shaped configuration, but it is not a replacement for that interactive workflow.

The split I use is simple:

Inspector answers, “What does this server expose, and does a normal call work?”
ResiliReplay asks, “What happens when a declared failure is inserted, does recovery stay inside its budget, and can I replay the result later?”

That second question needs more than random fault injection. A useful campaign must be bounded and reviewable: one seed, explicit scenarios, a small tool allowlist, fixed retry and time budgets, declared expectations, and local evidence. If a campaign will call a tool, ResiliReplay prints a canonical review plan and requires the exact campaign hash back before execution.

Start without contacting the server

With Node.js 22 or 24, the minimal entry point is a dry run:

npx --yes resilireplay@0.3.1 mcp audit \
  --inspector-config ./mcp.json \
  --server my-server \
  --dry-run

The output describes the executable, argument list, working directory, transport, timeouts, and environment-variable names. It does not start the server or call a tool. That makes it a useful place to catch an incorrect target or an unexpectedly broad configuration.

Then generate a campaign template:

npx --yes resilireplay@0.3.1 campaign init reliability.campaign.yml
npx --yes resilireplay@0.3.1 campaign validate reliability.campaign.yml

For an early field test, I keep concurrency at one, allow exactly one read-only idempotent tool, set retries to one, and use three scenarios.

1. Establish a clean control

The control uses fault: none and expects a pass with zero retries. This is not ceremonial. Without a clean control, an injected failure can be confused with a broken install, an invalid working directory, a missing browser binary, or an incompatible server version.

One of my Playwright MCP setup runs demonstrated the point: recovery failed because Chrome for Testing was not installed. That was an environment failure, not evidence about Playwright MCP recovery. After using the documented installer, the clean control and the bounded campaign could be interpreted honestly.

2. Inject a result-level failure and retry once

The next scenario uses mcp-tool-error with recovery: retry. ResiliReplay allows the real tool call to complete, replaces the observed result at the test boundary with a controlled error, and permits one retry. The assertions require successful recovery, no more than one retry, no duplicate side-effect attempt, and policy compliance.

That distinction matters. The server did not contain the injected error; the harness introduced it. A pass means only that the observed behavior matched this declared experiment.

3. Add a malicious-canary negative control

The third scenario injects mcp-malicious-canary-instruction and expects failure. Its purpose is to show that the campaign can detect a known bad condition instead of reporting green regardless of input.

This is a negative control, not a vulnerability claim. The canary is synthetic, no real secret is used, and the expected outcome is recorded before the run. When the expected failure occurs, ResiliReplay reduces the causal trace and writes a scenario, minimized fixture, executable Node test, and manifest. The campaign runner executes that generated regression immediately.

Baselines should fail closed

After a complete expectation-matching run, approve it explicitly:

npx --yes resilireplay@0.3.1 campaign approve runs/field-test \
  --output baselines/field-test.json

npx --yes resilireplay@0.3.1 campaign compare runs/field-test \
  --baseline baselines/field-test.json \
  --output runs/comparison

The comparison checks verified run identity and declared metrics such as score drop, retries, and duplicate attempts. Incomplete or hash-invalid evidence is not silently treated as a pass. This is the part that turns a one-off experiment into a CI boundary.

Three bounded field validations

I ran the public resilireplay@0.3.0 package against three pinned, independently maintained projects over their documented local stdio paths:

MCP Everything Server: echo with harmless generated text.
Playwright MCP: browser_snapshot on a blank isolated headless page, with no navigation.
UI5 MCP Server: get_guidelines against bundled guidance, with no project analysis.

Each campaign ran the clean control, one injected tool-result error with one successful retry, and the canary negative control. Each negative control produced a verified executable regression. Each approved baseline comparison reported zero differences. The complete commands, versions, boundaries, and sanitized evidence are in the field results.

These are three narrow case studies, not universal benchmarks. They cover one reviewed operation per server. They do not rank the projects, imply upstream adoption, or certify security. Streamable HTTP is exercised by ResiliReplay’s local automated fixtures, but none of the three selected external packages used it as the documented local primary path, so I do not present them as third-party HTTP evidence.

Safety boundaries and limitations

ResiliReplay is defensive reliability software, not an OS sandbox. A tool you authorize can still mutate files, databases, browsers, or remote systems with the permissions of its server. Start locally; prefer read-only, idempotent calls; review every allowlisted tool; and never use production data merely to provoke a failure.

Reports are sanitized, but pattern redaction cannot prove that every application-specific secret is gone. Hashes link evidence and detect changes; they do not establish signer identity. The current scorer evaluates declared observable evidence, not open-ended semantic quality. Side-effecting campaigns are intentionally not resumed after interruption because replaying a partial operation may be unsafe.

The project, npm package, and five-minute field-test guide are available from the ResiliReplay site. If you maintain an MCP server, I would value a sanitized result from one real, reviewed tool: which boundary exposed the most useful behavior—transport failure, result-level failure, duplicate invocation, or recovery after partial completion?

Update — ResiliReplay v0.4.0 Adopt (August 5, 2026)

Since this article was published, v0.4.0 has shortened the same failure-to-regression workflow into two entry points:

npx --yes resilireplay@0.4.0 demo
npx --yes resilireplay@0.4.0 adopt --config ./mcp.json --dry-run
npx --yes resilireplay@0.4.0 adopt --config ./mcp.json

demo needs no configuration, account, API key, Docker, external server, or LLM. It runs a clean control, injects a tool-result failure, performs one bounded recovery, checks a malicious-canary negative control, generates a regression, and executes that regression.

adopt --dry-run parses an Inspector-compatible configuration and shows the exact sanitized target and generated-file plan while starting no process, opening no network connection, calling no tool, and writing no project file. Real adoption still requires explicit review of the target, exact tool and arguments, and one-duplicate retry suitability. Tool annotations remain untrusted hints.

Final verification against the public npm package measured a 239 ms demo and a 783 ms real local stdio adoption. Adoption generated 14 commit-ready artifacts; its executable regression and all four generated campaign scenarios passed. An external GitHub Action matrix passed on Ubuntu and Windows with Node.js 22 and 24 using the immutable aliengineering-byte/resilireplay@v0.4.0 tag.

The earlier three v0.3.0 field validations in this article remain historical bounded evidence; they are not upstream endorsements or v0.4 adopter claims.

Update — ResiliReplay v0.5.0 Everywhere (August 5, 2026)

v0.5.0 extends the failure-to-regression path from reviewed MCP campaigns into supported coding-agent sessions through one canonical, local-first integration core.

npx --yes resilireplay@0.5.0 demo
npx --yes resilireplay@0.5.0 connect --agent auto --dry-run
npx --yes resilireplay@0.5.0 mcp serve

Capture remains opt-in and off after installation. The Claude Code and Codex plugins pass disposable marketplace installation plus controlled fixture-capture regressions; Hermes passes isolated skill discovery and nine-tool stdio MCP registration. Those are deliberately narrow labels: Claude and Codex are installation + fixture verified, Hermes is installation verified, and none is presented here as a live model benchmark or vendor endorsement.

The canonical schemas bound captured agent, tool, outcome, timing, error-class, hash, and redacted-summary fields. Raw prompts, full transcripts, authorization headers, environment values, unrestricted tool bodies, and personal paths are not persisted by default. Hooks observe supported results only while capture is armed; they never inject a fault into an ordinary agent session or automatically retry a failed side-effecting call.

The public npm tarball is self-contained with no runtime dependencies and includes the portable Agent Skill. Official MCP Inspector 2.1.0 enumerated nine annotated tools. The immutable release gate passed 91 tests across 16 files, Windows and Ubuntu on Node 22/24, Studio Playwright E2E, packed-package smoke, secret/privacy scans, and the official Claude plugin and Agent Skills validators.

DEV Community: Ali