DEV Community

Cover image for Testing and Debugging MCP Applications: A Practical Production Guide
Sushyam Nagallapati
Sushyam Nagallapati

Posted on

Testing and Debugging MCP Applications: A Practical Production Guide

#ai #mcp #testing #devops

This article is part of my MCP series. In the previous article, I covered authentication, tool permissions, secrets management, input validation, and tenant isolation for production MCP servers.

Read the previous article: Securing MCP Servers: 7 Essential Controls for Production

An MCP application may work perfectly during a local demo and still fail in production.

A tool may return the wrong data. An external API may time out. A blocking function may freeze the event loop. One tenant’s expired credentials may create repeated failures. The model may also select the wrong tool or generate invalid arguments.

These problems are difficult to diagnose unless testing and observability are built into the application from the beginning.

In this article, we will look at practical ways to test and debug MCP applications before real users depend on them.

What Should Be Tested?

An MCP application usually contains several moving parts:

User
  ↓
AI Client
  ↓
MCP Server
  ↓
Tool
  ↓
External API, Database, or Service
Enter fullscreen mode Exit fullscreen mode

A failure can happen at any layer.

A complete testing strategy should cover:

  • Tool logic
  • Input validation
  • External integrations
  • Authentication and authorization
  • Timeouts and retries
  • Tool selection
  • Concurrent requests
  • Tenant isolation
  • Logs, metrics, and traces

Testing only the Python function is not enough. You also need to verify how the complete request behaves from the client to the external service.

1. Start with Unit Tests

Unit tests verify one small part of the application at a time.

Suppose an MCP server exposes a weather tool:

@mcp.tool()
def get_weather(city: str):
    if not city.strip():
        raise ValueError("City is required")

    return weather_client.get(city)
Enter fullscreen mode Exit fullscreen mode

A basic test could verify that empty input is rejected.

import pytest

def test_get_weather_rejects_empty_city():
    with pytest.raises(ValueError):
        get_weather("")
Enter fullscreen mode Exit fullscreen mode

Another test could verify the expected response:

def test_get_weather_returns_result(mocker):
    mocker.patch(
        "weather_client.get",
        return_value={"city": "Toronto", "temperature": 24}
    )

    result = get_weather("Toronto")

    assert result["city"] == "Toronto"
    assert result["temperature"] == 24
Enter fullscreen mode Exit fullscreen mode

Useful unit tests should cover:

  • Valid inputs
  • Missing inputs
  • Invalid values
  • Permission failures
  • Expected output structure
  • Error responses
  • Boundary conditions

Keep tools small and focused. Narrow tools are easier to test than tools that perform several unrelated actions.

2. Mock External Services

MCP tools often depend on APIs, databases, cloud platforms, and third-party services.

Calling real services in every test can make the test suite:

  • Slow
  • Expensive
  • Unreliable
  • Difficult to reproduce
  • Dependent on internet access

Instead, mock the external dependency.

def test_customer_lookup(mocker):
    mocker.patch(
        "customer_api.get_customer",
        return_value={
            "id": "cust-104",
            "status": "active"
        }
    )

    result = get_customer("cust-104")

    assert result["status"] == "active"
Enter fullscreen mode Exit fullscreen mode

You should also test failure responses.

def test_customer_api_timeout(mocker):
    mocker.patch(
        "customer_api.get_customer",
        side_effect=TimeoutError()
    )

    result = get_customer("cust-104")

    assert result["error"] == "service_unavailable"
Enter fullscreen mode Exit fullscreen mode

Do not test only successful responses.

Simulate:

  • Timeouts
  • Invalid credentials
  • Rate limits
  • Empty responses
  • Malformed JSON
  • Network failures
  • Server errors

Production systems fail in many ways. Your tests should reflect that.

3. Add Integration Tests

Unit tests confirm that individual functions work.

Integration tests confirm that multiple components work together.

For an MCP application, an integration test may verify that:

Client request
    ↓
MCP server receives request
    ↓
Tool is discovered
    ↓
Tool executes
    ↓
Structured response is returned
Enter fullscreen mode Exit fullscreen mode

A useful integration test should check:

  • Whether the server starts correctly
  • Whether expected tools are registered
  • Whether arguments are parsed correctly
  • Whether authentication is enforced
  • Whether the response follows the expected schema
  • Whether failures are returned in a controlled format

For example:

def test_weather_tool_integration(mcp_client):
    result = mcp_client.call_tool(
        "get_weather",
        {"city": "Toronto"}
    )

    assert result["city"] == "Toronto"
    assert "temperature" in result
Enter fullscreen mode Exit fullscreen mode

Run integration tests in an isolated environment with test credentials and test data.

Never point automated tests at production resources.

4. Test Tool Selection

A tool may work correctly but still be selected at the wrong time.

For example, a user may ask:

Explain how weather forecasts are created.

The model should answer conceptually rather than calling a live weather tool.

But when the user asks:

What is the weather in Toronto today?

The tool should be used.

Create a small set of evaluation prompts.

User request Expected behaviour
What is the weather in Toronto? Call get_weather
Explain weather forecasting No tool required
Show my open support tickets Call list_tickets
What is a support ticket? No tool required
Delete production Reject or require approval

Do not expect tool selection to be perfect in every case.

Instead, evaluate:

  • Whether the correct tool was selected
  • Whether unnecessary tools were avoided
  • Whether arguments were accurate
  • Whether dangerous actions were rejected
  • Whether the final answer reflected the tool result

These evaluations can be added to CI so changes to prompts, models, or tool descriptions do not silently reduce reliability.

5. Test Timeouts and Retries

External services will eventually become slow or unavailable.

Every external call should have a timeout.

response = api_client.get(
    "/orders",
    timeout=5
)
Enter fullscreen mode Exit fullscreen mode

Without a timeout, a request may wait indefinitely.

Retries can help with temporary failures, but they must be limited.

for attempt in range(3):
    try:
        return call_provider()
    except TimeoutError:
        if attempt == 2:
            raise
Enter fullscreen mode Exit fullscreen mode

Test that:

  • Requests stop after the configured timeout
  • Retries are limited
  • Backoff is applied
  • Permanent errors are not retried
  • Duplicate operations are avoided
  • A safe error is returned to the user

Be especially careful with non-idempotent operations.

Retrying a read request is usually safer than retrying:

create_order
send_payment
delete_resource
send_email
Enter fullscreen mode Exit fullscreen mode

A repeated write action may create duplicate or unintended results.

6. Detect Blocking Calls

One of the hardest production failures occurs when the process is still running but the application stops responding.

This can happen when synchronous work blocks an asynchronous event loop.

Examples include:

  • Synchronous API clients
  • Large file operations
  • CPU-heavy processing
  • Blocking database calls
  • Long-running third-party SDK functions

The container may still appear healthy at the process level, but health endpoints and user requests may stop responding.

Move blocking work away from the event loop.

import asyncio

result = await asyncio.to_thread(
    blocking_client.generate_embedding,
    text
)
Enter fullscreen mode Exit fullscreen mode

You can also monitor event-loop responsiveness.

import asyncio
import time

async def monitor_event_loop():
    while True:
        start = time.monotonic()
        await asyncio.sleep(1)
        delay = time.monotonic() - start - 1

        if delay > 10:
            logger.error(
                "Event loop delay detected",
                extra={"delay_seconds": delay}
            )
Enter fullscreen mode Exit fullscreen mode

For difficult hangs, a separate watchdog thread can capture thread stack traces when the event loop becomes unresponsive.

This turns an unexplained freeze into something the team can investigate.

Useful signals include:

  • Event-loop delay
  • Health-check response time
  • Active requests
  • Thread-pool saturation
  • Queue length
  • Tool execution duration

A process being alive does not always mean the application is healthy.

7. Test Concurrency and Tenant Isolation

An MCP server may work correctly for one user but fail under concurrent traffic.

Load tests should simulate multiple users calling tools at the same time.

Measure:

  • Response latency
  • Error rate
  • Active requests
  • Queue length
  • Database connections
  • External API limits
  • CPU and memory usage
  • Tool execution time

A basic concurrent test could look like this:

import asyncio

async def run_request(client, city):
    return await client.call_tool(
        "get_weather",
        {"city": city}
    )

async def test_concurrent_requests(client):
    results = await asyncio.gather(
        run_request(client, "Toronto"),
        run_request(client, "Vancouver"),
        run_request(client, "Calgary"),
    )

    assert len(results) == 3
Enter fullscreen mode Exit fullscreen mode

Multi-tenant systems also need failure isolation.

Suppose one tenant has an expired provider key and receives repeated 401 responses.

That failure should not reduce service capacity for every tenant.

Track errors using dimensions such as:

tenant_id
tool_name
provider
error_type
Enter fullscreen mode Exit fullscreen mode

Test that:

  • Tenant A cannot access Tenant B’s data
  • One tenant’s rate limit does not block others
  • One tenant’s invalid credentials remain isolated
  • Circuit breakers operate at the correct scope
  • Concurrency controls do not treat every error as global

8. Use Structured Logs and Traces

When a tool fails, a message such as this is not very helpful:

Something went wrong.
Enter fullscreen mode Exit fullscreen mode

Structured logs make failures easier to search and connect.

{
  "correlation_id": "req-72a91",
  "tenant_id": "tenant-18",
  "tool_name": "get_customer",
  "duration_ms": 842,
  "status": "failed",
  "error_type": "timeout"
}
Enter fullscreen mode Exit fullscreen mode

Useful fields include:

  • Correlation ID
  • Tenant ID
  • Tool name
  • External service
  • Execution duration
  • Retry count
  • Response status
  • Error category

Distributed tracing can show the complete request path:

User Request
    ↓
AI Client
    ↓
MCP Server
    ↓
Tool
    ↓
External API
Enter fullscreen mode Exit fullscreen mode

This helps answer questions such as:

  • Where did the request slow down?
  • Which service returned the error?
  • Was the tool called more than once?
  • Did a retry succeed?
  • Did the failure affect one tenant or everyone?

Do not log secrets, access tokens, private customer records, or full sensitive prompts.

9. Add Tests to CI/CD

Tests are most valuable when they run automatically.

A basic pipeline may include:

Code commit
    ↓
Static checks
    ↓
Unit tests
    ↓
Integration tests
    ↓
Security tests
    ↓
Container build
    ↓
Deployment to test environment
    ↓
Smoke tests
Enter fullscreen mode Exit fullscreen mode

A simple GitHub Actions job could look like this:

name: Test MCP Application

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest
Enter fullscreen mode Exit fullscreen mode

Production deployment should stop when critical tests fail.

After deployment, run smoke tests to verify that:

  • The server is reachable
  • Health checks respond
  • Tools are registered
  • Authentication works
  • A safe test tool can execute
  • Logs and metrics are being generated

Practical Testing Checklist

Before releasing an MCP application, confirm that:

  • Every tool has unit tests
  • Invalid inputs are rejected
  • External APIs are mocked in tests
  • Timeouts and retries are tested
  • Authentication and permissions are verified
  • Tool-selection prompts are evaluated
  • Dangerous operations require approval
  • Integration tests cover the complete request path
  • Concurrent requests have been tested
  • Tenant failures remain isolated
  • Event-loop responsiveness is monitored
  • Logs include correlation IDs
  • Sensitive information is not logged
  • Tests run automatically in CI/CD
  • Smoke tests run after deployment

Final Thoughts

Testing an MCP application is not only about checking whether a tool returns the expected result.

You also need to know how the application behaves when:

  • An API becomes slow
  • Credentials expire
  • The model selects the wrong tool
  • Several users send requests together
  • A blocking function freezes the event loop
  • One tenant begins generating repeated failures

The most useful tests focus on real failure scenarios, not only the happy path.

When tools are small, inputs are validated, dependencies are mocked, and failures are observable, debugging becomes much easier.

This completes my five-part MCP series, covering the journey from understanding MCP to building, deploying, securing, testing, and operating MCP-based applications.

Thanks for Reading

This article completes my MCP series:

  1. Model Context Protocol (MCP) Servers Explained: A Complete Beginner’s Guide
  2. Building Your First AI Agent with MCP: A Step-by-Step Guide
  3. Productionizing an MCP-Based AI Agent with Docker, Kubernetes, CI/CD, and Observability
  4. Securing MCP Servers: 7 Essential Controls for Production
  5. Testing and Debugging MCP Applications: A Practical Production Guide

I regularly share what I learn about AI engineering, MCP, DevOps, cloud infrastructure, Kubernetes, and Site Reliability Engineering.

LinkedIn: Connect with me on LinkedIn

What has been the most difficult MCP issue for you to test or debug?

Top comments (1)

Collapse
 
mads_hansen_27b33ebfee4c9 profile image
Mads Hansen •

One production failure class I’d add is abandoned work.

A client timeout or cancellation does not prove that the database query, API request, or background job stopped. The caller may see a failure while the server keeps consuming a worker slot—or completes a write after the user believes it was cancelled.

I’d add tests that cancel a tool call mid-flight and assert that the downstream operation is cancelled where possible, open transactions are rolled back, semaphore/connection-pool capacity is released, no automatic retry fires, and the trace ends with an explicit cancelled/indeterminate state. Repeat that under disconnect/reconnect and with the worker pool saturated; zombie calls become much easier to expose.

Correlating the client deadline, server cancellation event, and downstream request ID also makes “timed out” distinguishable from “finished after the client left.”